If you are using Fabric Mirroring or Delta Lake tables you will see an additional column in your data source: METADATA$ROW_ID. This column is added automatically by Fabric to manage versioning, change data capture, and transactional consistency. What exactly is this column, and you need to care.

The column is, for all intents and purposes, a SQL GUID, which is a 40-character unique identifier and because it is unique it is uncompressible, and this is the problem. Analysis Services models within Fabric are limited by your SKU, so space is at a premium. We are running an F64, so that limits our model size to 25GB.

When you create a model, it does a select * from <source>, which was never an issue until we started using a Mirrored source. That METADATA$ROW_ID snuck into our model and caused the size to explode. Our complete 5-year model normally takes up about 6GB of space, but this would blow out around 18 months. Digging into the models from both Azure Analysis Services and Fabric Analysis Services we found that sneaky ID.

The correction is easy, now that we found the issue, so I am passing it on here in hopes of preventing others the headaches that come from details buried in specific implementations.

Go into your semantic model through DAX Studio and manually remove the METADATA$ROW_ID column. Do this after you have created your model as it is simpler to remove the column than to modify the build process to exclude it.

I rely heavily on stored procedures for interaction both internally and externally. The improve performance and efficiency I’ve created many table types, most notably one for isbns. It looks like this:

create type dbo.ISBN as table(
    ISBN char(13) not null
    primary key clustered (ISBN asc)
) with (ignore_dupe_Key=off))

Let’s use this in a sample procedure

create procedure inventory.CheckISBNQty
(@ISBNs dbo.ISBN readonly)
with execute as owner
as
.....

We have a function app, [func-InventoryProcess-prod] that will be calling this procedure so naturally you would grant it the ability to execute the function.

grant exec on inventory.CheckISBNQty to [func-InventoryProcess-prod]

When the function app tries to run the stored procedure, you will get an error that you cannot execute dbo.ISBN. To solve this problem, you need to add execute permissions on the table object to the app.

grant execute on type::dbo.ISBN to [func-InventoryProcess-prod]
or
grant execute on type::[dbo].[ISBN] to [func-InventoryProcess-prod]

Microsoft SQL Server is very lenient when it comes to the format of a date it will be ingesting. It can be:

• ‘6/3/2025’

• ‘6/03/2025’

• ‘06/03/2025’

• ‘2025-6-3’

• ‘2025-06-3’

The list goes on. Snowflake however, is very specific: ‘YYYY-MM-DD’.

The pipeline I was building needed to add a FILEDATE column to the .csv file it was creating that would be ingested by Snowflake. I added the column into the source of the Copy data activity as follows:

The expected result would be: “2025-06-03T13:11:00” as per the documentation of the utcNow() function.

The result provided:

As I mentioned before, this would be an acceptable datetime in Microsoft SQL Server, so it was never an issue. On ingestion to Snowflake, this blew up. I tried the FormatDate function, I tried utcNow(‘yyyy-mm-ddTHH:mm:ss’) and numerous variations of that, all to no avail.

If however, you put the same function call in a variable:

You get the correct result:

Then simply use the variable vs the function when adding it to your source.

One might naturally simply give select permission to that specific table in the inventory schema. If you did this, you reduce the security aspect of hiding the underlying schema/objects. The correct method is simply to make sure both schemas are ‘owned’ by the same user. We have everything owned by dbo, so you need to run this on all schemas involved.

Instead run this command on both the inventory and customer schema:

alter authorization on schema::[inventory] to dbo;
alter authorization on schema::[customer] to dbo;

You preserve the integrity of your security model, while being able to properly isolate data via schemas.

At the beginning of your procedure (I usually put it as the first line of code after my comments) you need to put the following:

SET FMTONLY OFF

Now if you look in the Microsoft documents it tells you that the FMTONLY flag is deprecated and gives you a handful of alternatives. I asked CoPilot how I would use these specifically for Report Builder it suggests not using select into #table and either explicitly declaring that temp table or using table variables (which are explicitly declared), but when pressed it said that is not foolproof and the suggested method is the FMTONLY flag.

I will do some additional research myself into declaring # tables ahead of time and the use of table variables, but for now I am comfortable using this. I don’t foresee Microsoft scrapping Report Builder anytime soon.

I was building a data factory source the other day and I needed to skip the first 5 rows. This is intuitive and easy within a text (csv) source as there is a setting to skip the first N rows. In excel there is a 'Range' option which has as an example 'A1:B10'.

The example implies that you need to specify the entire area of the excel that you want to import. I simply put my starting point (5 lines down) and sure enough it took the entire excel, all columns, all rows, just skipping down 5 lines.

I know this is an incredibly small post and you might be wondering; "why post this?". I use this blog for 2 reasons. The first is I have an online, universally accessible set of notes. I routinely go to code something and say, "where did I put that formula, or code, or snippet, etc." and rather than trolling through sys.sql_modules or trying to remember the pipeline, I just come here.

The second reason is to help others find, what I believe are useful snippets to help others out.

1. Customer

2. Book

   1. Format

3. Sales Rep

Now in the fictional PrintItNow publishing company here are some base stats

• Number of Customers: 20,000

• Number of Titles: 100,000

  • Number of Formats: 5

• Number of Reps: 1,000

The rules are as follows:

• A rep can be assigned the following:

  • 1 or all customers

  • 1 or all titles

  • 1 or all formats

• Two reps cannot sell the same book+format to the same customer. That means I can sell the hardcover and you could sell the paperback to the same client.

• Reps can be realigned at any time across all formats, books, and customers. This realignment must be retroactively applied to all previous sales. -- GOTCHA!

To put that into perspective if I am allowed to sell all of the titles, but hardcover only, to all customers I have a 'territory' of 20,000*100,000=2 billion entries in my sales table. How often do reps get realigned? The two most common times are when a new rep gets hired and takes over a portion of territory from 1 or more other reps or if a rep leaves and that territory needs to be spread out over other reps. No matter what the case all the sales entries (which could be billions) need to be changed. Adding to that is every day you sell more books, so the universe of sales grows, and you publish new books, so the territory grows. I had tried to tackle this problem a few years ago with the use of bit masks, but the functions were not up to the task. With SQL Server 2022, they released two new functions; SET_BIT and GET_BIT.

Now that we have the background of our publishing company, let's get into what a bit mask is.

First, we need a table to manage the reps themselves.

Create table SalesRep
(RepName varchar(256) not null,
RepId int not null)

This gives me plenty of space for reps (just under 2 million for a 4-byte int)

Now you need to have a table that would manage the sales territory

Create table BookRep
(RepId int not null,
Book bigint not null)

The line is concise (8 bytes), but you could have tens of millions or more based upon the combination of what book a rep can sell. For this example, we have every rep able to sell every book and divide it up by customers. In that case it would be 1000*100000=100 million and it will grow for every rep you hire and every book you publish. Here's a better way.

Bit Masks

A bit mask works much like the lockers we had in school. Imagine a set of lockers numbered 0-7. Now let's take students in alphabetical order by first name: Alice, Bob, Jessica, and Joshua. It is important that you pick some ordering system for your data to insure a consistent mapping.

Now let's assign these students to lockers, starting at 0 for the first student, 1 for the second and so on: (MaskPosition = Locker #)

This would be represented in the code below

Create table SalesRep
(RepName varchar(256) not null,
EmployeeId char(4) not null,
MaskPosition int not null) -- new column to represent the assigned position

Now we will need a data type to represent the lockers. If you haven't already guessed, you need a binary data type and you will need 1 byte for every 8, or fraction of 8, sales reps (lockers) you will employ. In our example, we need a binary(1).

The table would look like this.

Create table BookRepMap
(Book bigint not null, --ISBN13 can be represented as a bigint 
 RepMap binary(1) not null)

This will only ever add a record for each Book, and you add 1 byte to the RepMap column for every 8 reps (starting with the first). Now before we end the first article in the series, let me explain just how powerful this is.

If we take 8000 bytes as the maximum row and subtract 8 bytes for the bigint, that leaves us with 7982 bytes for the RepMap column. You get 8 positions for each byte: 7982*8=63,856 total reps(positions) for single row!

In the next installment of this series, I will show the code used to fill this table.

Anatomy of a Stored Procedure

Similar to function calls a stored procedure has a name and 0 or more parameters. The parameters are actually placeholder variables, so they are required to begin with an ampersand (@). Like variables they are also required to have their datatype defined. Let's build a very simple stored procedure that takes an ISBN and returns the author of the book.

create procedure FindAuthorISBN13
(
@ISBN bigint
)
as
begin
--fill in with code
end

This is how you call it.

--call by parameter. This is the preferred method
exec FindAuthorISBN13 @ISBN=9781254872103
--call by position
exec FindAuthorISBN13 9781254872103
--implicit conversion can be called by position or parameter
exec FindAuthorISBN13 '9781254872103'

Now if you tried to call it with something other than a valid big integer you will get an error. That is the first line of defense.

Now for the guts of the procedure:

 create procedure FindAuthorISBN13
(
@ISBN bigint
)
with execute as owner
as
begin
 select AuthorName as [Author Name]
   from TitleMetadata
  where ISBN13=@ISBN
end

As you can see you are using the variable to test for a value against a column. You aren't passing in a SQL statement to be run.

Now from C# this is how you would make the call:

....
var cmd = new SQLCommand(Proc, conn)
cmd.Parameters.AddWithValue("@ISBN",_isbn);
cmd.CommandType = CommandType.StoredProcedure;
using var reader = cmd.ExecuteReader();
reader.Read();
result=reader.GetFieldValue<string>(0);
......

There is no way a malicious actor can gain access to anything other than the result programmed in the procedure.

/* The exported Power Query Formula Language (M Language ) can be used with Power Query in Excel and Power BI Desktop. For Power BI Desktop follow the instructions below:

1. Download Power BI Desktop from https://powerbi.microsoft.com/desktop/

2. In Power BI Desktop select: 'Get Data' -> 'Blank Query'->'Advanced Query Editor'

3. Paste the M Language script into the Advanced Query Editor and select 'Done' */

let AnalyticsQuery = let Source = Json.Document(Web.Contents("https://api.loganalytics.io/v1/subscriptions/your subscription here/query", [Query=[#"query"="AzureDiagnostics | where ResourceProvider == 'MICROSOFT.ANALYSISSERVICES' and Resource in ('server1','server2') and OperationName =='QueryEnd' and NTDomainName_s == 'AzureAD' | order by StartTime_t",#"x-ms-app"="AzureFirstPBI",#"timespan"="PT1H",#"scope"="hierarchy",#"prefer"="ai.response-thinning=true"],Timeout=#duration(0,0,4,0)])), TypeMap = #table( { "AnalyticsTypes", "Type" }, { { "string", Text.Type }, { "int", Int32.Type }, { "long", Int64.Type }, { "real", Double.Type }, { "timespan", Duration.Type }, { "datetime", DateTimeZone.Type }, { "bool", Logical.Type }, { "guid", Text.Type }, { "dynamic", Text.Type } }), DataTable = Source[tables]{0}, Columns = Table.FromRecords(DataTable[columns]), ColumnsWithType = Table.Join(Columns, {"type"}, TypeMap , {"AnalyticsTypes"}), Rows = Table.FromRows(DataTable[rows], Columns[name]), Table = Table.TransformColumnTypes(Rows, Table.ToList(ColumnsWithType, (c) => { c{0}, c{3}})) in Table in AnalyticsQuery

The highlighted section requires you to input your values. In the above example, this will monitor 2 servers. You can add more or split this into individual servers/query.

The application in question manages lists of books. The workflow is as follows:

1. Select 1 or more ISBNs from the list of books we publish (over 1m)

2. Assign this collection a name and a catalog (collection of lists)

3. Save the collection

4. Go back to the main menu with the names and metadata of all your catalogs

The issue happens between 3 and 4. This system keeps metadata about each list and metadata about the folder (number of total lists, number of total items, last modification). It needs to make a series of calls. It can't just save the list and return you to the main screen without first making a call to the system to populate that screen with the latest information (which includes audit information). This means a call to SaveList and a call to GetCatalogs. Developers noted that 70% of the time the main menu was missing the newly saved list, but if they did a refresh it appeared. We could not reproduce this by hand (though Postman).

We are running the database on an Azure Hyperscale with RCSI (Read Committed Snapshot Isolation), so the first thought was there are independent threads making the calls and the GetCatalogs is returning the previously committed set. However, the development team confirmed that it was a single chain of events through a single thread with an average time between calls of 300ms.

All calls in this application are controlled via stored procedures, so we can guarantee the order of operations and within the SaveList procedure is a commit before the result set is returned guaranteeing the data is in the database before the thread gets the go-ahead to move to the next call. At this point I realized it was the replica.

300ms is not enough time for the data to be moved to the replica, so I simply removed the ApplicationIntent=ReadOnly and the bug disappeared.

A word of caution when designing an application. The flow of the application is important to understand while making architectural decisions. In this specific example, the need to pull back the data vs having the application carry it means the timing of when the data is where (primary vs replica) is very important. If the application did a few other things and the avg time between the save and when it made the call to GetCatalog was say 750ms or 1s, the replica might have been just fine.

Using this connection string: (Some portions were left out for brevity)

"Server=tcp:<servername,port>;Initial Catalog=<database name>;ApplicationIntent=ReadOnly;"

The option ApplicationIntent=ReadOnly tells the system to first look for a replica when asking for a connection, but if there is no replica, then connect to the existing primary(read/write) server.

I have 2 connection strings in my APIs, one that includes that option and one that doesn't. This helps balance the calls between all the available backend resources without doing any special work within the calls themselves.

This is the query I am working with:

select s.SalesRepName,
       m.material as ISBN
  from MaterialSalesRepMap m
  join SalesHierarchy s on (get_bit(EligibilityKey,Position)=1
       and SalesRepName in ('Smith,Joe','Doe,John','Doe,Jane','Lowry,Amanda'))

The optimizer chose to parallelize this and we can see it here

One thing to note is that when you make the collection parallel (distribute streams) you then need to funnel those threads back into a controlling thread to output (gather streams).

Statistics (pay attention to the highlighted portion):

It turns out that 2nd worktable grows/shrinks based on the number of reps.

select s.SalesRepName,
       m.material as ISBN
  from MaterialSalesRepMap m
  join SalesHierarchy s on (get_bit(EligibilityKey,Position)=1
       and SalesRepName in ('Smith,Joe','Doe,John','Doe,Jane'))

MaterialSalesRepMap

• Scan count: NumReps+1

• logical reads: 301586*NumReps

The worktable grows by millions for each additional rep. We have 3,998 'rep' entries! It was a this point that I remembered SQL Server (from 2018+) can be 'aggressive' in its choice of how many workers to use when it chooses parallelism.

There is an optimizer hint (maxdop N) which stands for MaxDegreeOfParallelism. This directs the optimizer to use N number of threads when determining how many to use. Rather than guess what would be an optimal number for a query with a varying number of reps in the request, I wanted to see how efficient the system would be without it. So I turned off parallelism by saying (maxdop 1).

select s.SalesRepName,
       m.material as ISBN
  from MaterialSalesRepMap m
  join SalesHierarchy s on (get_bit(EligibilityKey,Position)=1
       and SalesRepName in ('Smith,Joe','Doe,John','Doe,Jane','Lowry,Amanda'))
option (maxdop 1)

As you can see this forced the optimizer to ignore any type of parallel processing.

The results were fantastic (and consistent)

The only thing that changes is the underlined blue portion. The Scan Count is equal to the number of reps and the logical reads fluctuates slightly up or down accordingly.

If you are looking to squeeze out some additional performance or look for consistent results this would be one of those specialized tuning approaches that you can take. I have used this approach successfully in selects, inserts, and deletes.

CREATE function [dbo].[ValidateISBN]

(@ISBN as bigint)

returns char(1)

as

begin

 declare @is978 tinyint=(978-(@ISBN /10000000000))*-1,

                 @checkdigit tinyint

 set @checkdigit =10-cast((cast((@ISBN %10000000000000/1000000000000) as tinyint)

  + cast((@ISBN %1000000000000/100000000000) as tinyint)*3

  + cast((@ISBN %100000000000/10000000000) as tinyint)

  + cast((@ISBN %10000000000/1000000000) as tinyint)*3

  + cast((@ISBN %1000000000/100000000) as tinyint)

  + cast((@ISBN %100000000/10000000) as tinyint)*3

  + cast((@ISBN %10000000/1000000) as tinyint)

  + cast((@ISBN %1000000/100000) as tinyint)*3

  + cast((@ISBN %100000/10000) as tinyint)

  + cast((@ISBN %10000/1000) as tinyint)*3

  + cast((@ISBN %1000/100) as tinyint)

  + cast((@ISBN %100/10) as tinyint)*3)as tinyint)%10



if ((@checkdigit=10 and @ISBN %10=0) or @checkdigit=@ISBN %10)

        return 'Y'

 return 'N'

end

GO

The developers normally build within dev, however, there are situations where 1-off reports are being developed or a new report is being developed and there is an impact on production.

This method of reporting is near real-time vs real-time for 2 reasons. First, SSAS doesn't send information during the run itself like say SQL Server does. It tells you when something starts and then when it finishes. The good news about that is the 'QueryEnd' event has all the calculated metrics (start-time,end-time, total time, etc) so you don't need to capture both 'QueryStart' and 'QueryEnd'. Second, the system behind the Log Analytics Workspaces is an ADX(Azure Data Explorer) which itself requires time to ingest. We see about 5 min delay.

To interact with the ADX cluster you need to write a language called KQL (Kusto Query Language), which looks like a cross between SQL and Unix scripting. Below is the query.

AzureDiagnostics
| where ResourceProvider == 'MICROSOFT.ANALYSISSERVICES'
    AND Resource in ('<myserver1>','<myserver2>')
    AND OperationName == 'QueryEnd'
    AND NTDomainName_s == 'AzureAD'
| project ServerName = Resource, CubeName = DatabaseName_s,QueryId = RootActivityId_g, QueryStart = StartTime_t, QueryEnd = EndTime_t, Duration = Duration_s, CPUTime = CPUTime_s, Success = Success_s, Error = Error_s,Query = TextData_s

You start with the table you want to query AzureDiagnostics (you can do joins, but they are beyond the scope of this article) and then you put a | (pipe) to take that output and move it to the next filter. In this case a where clause. After that filter you push those results to a project which is a select statement, but unlike SQL is not required. I am using the project because I want to not only create user-friendly names but reduce the number of columns (for efficiency on both ends).

The next step is to convert this into a Power BI (M query). This is under the Export option. Take that query and run it from within PowerBI and you now have a dataset to build your reports off of. The downside is you have to manually refresh the data, however, there is a mechanism to make this streaming. If there is interest in the streaming or any question, please let me know in the comments.

As an example all daily data acquired between 10/21/2018 and 10/27/2018 must have a WeekEndingDate of 10/27/2018. This can be done with a case statement, but a there is an inline formula that will come in handy when doing projections (YTD, previous 6 months, etc). I’ll present the formula and then show modifications to handle variations.

select dateadd(dd,7-datepart(dw,'10/21/2018'),'10/21/2018')

The above formula determines how many days away from 7 the current date is and simply adds that many days. We use 7, because Saturday is the 7th day of the week. I’ll break it down.

datepart(dw,'10/21/2018')

The above snippet returns a value of 2, because it is a Monday. 7 if it is a Saturday. You subtract 7 from that to determine how many days you have to move forward. In the example, you need to move 5 days from Monday to make it to Saturday. This gives you the number of days to add. Reducing the example:

select dateadd(dd,5,'10/21/2018')

This comes in very handy when you are looking to move backward in time a certain number of days, weeks, months, or years but align to a Saturday. An example is below, which goes back 1 month.

select dateadd(dd,(datepart(dw,7-dateadd(mm,-1,'10/21/2018')),dateadd(mm,-1,'10/21/2018'))

Here is a simple way to remember how to use the formula. Remember you have to move your date first, then push it to Saturday.

declare @var date

select @var=dateadd(mm,-1,'10/2/2018')

select dateadd(dd,7-datepart(dw,@var),@var)

We use Azure Data Factory as our ETL system, which has native drivers for Snowflake, but they only work under very limited situations, none of which suited this specific workflow. This meant we needed to use their odbc driver and to do that, we had to create a managed VM to contain the drivers and an Integration Runtime. This enables us to connect Snowflake directly to our Synapse system. Viola!

This setup, however, is not without its own quirks. You see we also use Azure Analysis Services, which requires ODBC, and some of its data sets also reside on Snowflake. This starts to put a significant amount of pressure on the VMs that act as a gateway, so we began looking for alternatives to moving data, to free up the VMs for cube builds only.

Snowflake allows you to export any table as a series of .gzipped files with the copy into command. You either copy into Snowflake or copy into a set of files. To do this required a script that would normally be run on a machine with the ability to connect to Snowflake. There was no easy way to coordinate this. Originally, we decided to have the job run as the end of a nightly process on Snowflake. We extended the job to push the table to a set of files in an Azure connected storage account and then call our event hub to signal the job is complete. That event would trigger a logic app, which would then call a Data Factory Pipeline that would ingest those files using a feature of Synapse called Polybase.

Polybase is the feature that Synapse uses to push/pull files (like the copy into command). Woot! We have bypassed the VM, but all was not good. Polybase gives incredible performance gains but is very finicky and expects the file to be formatted in some special way before it can ingest it. Even a simple '|' delimited file was copied to a set of staging files and then ingested. This was a dead-end, that is until the new Script Object became available in the Azure Data Factory.

Now we could run that exact command in our pipeline, which gave us more fine-grained control. Part 1. Part 2 was sneaky.

I wanted to see just what Polybase was doing with these files and it turns out to be something incredibly simple.

Polybase is simply replacing whatever delimiter you use in the source file with: \u2bd1

That seemed simple enough, but it turns out for Snowflake you need to tell it to use: \xE2\xaf\x91 as the delimiter.

This looks script object looks like this:

The source of the copy object looks like this:

What type of performance do you get?

461MB/s that is 100 times faster than we were getting with an odbc connection through a managed VM. Done!

You can use this trick to improve the ingest speed of any csv file.spee

A quick update: If you are interested in coming to the presentation or anything else DeveloperWeek has to offer, here is a link to a free pass. There is a limited number of them, so first come, first serve: Free Pass