How to Remove a Data Disk from an Azure VM (How not to blow your leg off)

I was working with a client recently, were we had to reconfigure storage within a VM (which is always a messy proposition). In doing so, we were adding and removing disks from the VM. this all happened mostly during a downtime window, so it wasn’t a big deal to down a VM, which is how you can remove a disk from a VM via the portal. However, upon further research, I learned that through the portal you can remove a disk from a running VM.

For the purposes of this demo, I’ve built a a SQL Server VM with two data disks and a single disk for transaction log files. The SQL VMs use Storage Spaces in Windows, which is a recommended best practice–but even if you are not using Storage Spaces, most of this will apply.

How To Identify Your Disk

This is the really important part of this post–how to identify what your disk is in the portal and with your VM. When you define a data disk in the portal, either you or the system will define a LUN number for the individual disk. You can see it on the portal in the below screenshot.

This number is mostly meaningless, except that within Windows, it lets you identify the disk. If you open up Server Manager and navigate to Storage Pools > Physical Disk, you can see where this LUN number shows up.

That number maps back to the values you see in the Azure portal, and unless you size each of your disks differently (which you shouldn’t do for performance reasons). If you aren’t using Storage Spaces, you can also see the LUN number in Disk Management in Windows as shown below.

You can also get this information using PowerShell using the Windows cmdlet Get-PhyiscalDisk.

It is very important to ensure that you have identified the correct disk before you remove it.

Removing the Disk

Azure will let you delete a disk from a running VM. Even if that disk is mounted in the VM and has data on it. Yes, I just did this on my test VM.

If you click one of those highlighted Xs and then click Save, your disk will be removed from your VM. There’s also a series of PowerShell commands you can use to do this. It is also important to note, that at this point your disk is still an Azure resource. Even though you have removed it from the VM, the disk still exists and has all the data it had the moment you detached it from the VM.

If you chose the correct disk to remove from your VM, and you have confirmed that your VM is healthy, you can navigate into the resource group for your VM where you will see your disks.

The important thing to note is that the state of the disk is “Unattached”, which means it’s not connected to a VM. So it can be deleted from Azure–I don’t recommend doing so until you have validated your VMs are running as expected.

You may ask how you can prevent disks from being removed from running VMs. I’ll write a post about this next week, but while you are waiting read up on resource locks in Azure.

Azure SQL Managed Instance versus Amazon RDS for SQL Server—Which Should You Choose? (Or why Managed Instance is faster)

Microsoft, in conjunction with Principle Technologies recently produced a benchmark, comparing the performance of Azure SQL Managed Instance, and Amazon RDS SQL Server. I normally really dislike these benchmarks—it can be really hard to build proper comparisons and the services frequently don’t have perfect equivalent service tiers, making them really hard to ultimately compare their performance. In fact, when I was reading this benchmark, I saw something when comparing the two services that made my eyes light up. And then I realized it was a limitation of RDS.

I immediately saw that Azure MI had 320,000 IOPs while AWS only had 64,000. Obviously Azure is going to crush any database benchmark with that difference. And then I did a bit more research. I the visited the AWS docs.

You’ll note that while Oracle RDS does get up to 256,000 IOPs (I guess those customers have more money), SQL Server has a max number of IOPs of 64,000. Needless to say, in this benchmark comparing the price/performance ratio, Managed Instance crushes RDS before you even add in the hybrid licensing benefits that Microsoft supports for Azure SQL services.

But Wait There’s More

While Managed Instance is by no means a perfect service, there are a number of reasons why I strongly recommend against running your database on RDS. Here are the main ones:

  • You can’t migrate a TDE encrypted database using backup and restore—you have to extract a BACPAC and import into a database in the service
  • The native backup solution doesn’t support restoring a database to point in time.
  • You can’t deploy cross-region, meaning, there is no near real-time option for disaster recovery
  • There is no instant file initialization which can make some restore and file growth operations extra painful

These are the major concerns, with an additional licensing concern of not being able to use Developer edition for your workloads, means your overall costs to run an environment are going to be a lot higher of you use RDS.

When Not to Choose PaaS?

While RDS has a lot of costs associated with it, and performance is limited, there is a price/performance/data volume curve that I feel I applies to both Azure and AWS platforms. If you need high end storage performance (which means using the Business Critical service tier), on Managed Instance, and your data volume, is more than a terabyte, you have to scale your Managed Instance to 24 cores. If your volume is more than 2 terabytes, you need to scale to 32 cores, and if you need more than 8-16 TB, you will need to scale to 80 cores, which will cost close to $30,000/month. I perfectly understand why this is the cost model—the storage is stored locally on the VM itself rather than remote—so Microsoft can’t put other VMs on that piece of physical hardware.

What Should You Do for SQL Server on AWS?

If you need to run SQL Server an AWS, what should you do? The answer is to use EC2 VMs. Sure you lose the minor benefits of having your servers patched and limited benefits of the backup feature, but you have more granular control over your IO performance and overall configuration.

Tl;dr Azure SQL Managed Instance delivers a lot more throughput than Amazon RDS for SQL Server, so your workloads will run a lot faster on Azure.

Taking Your Azure Active Directory Security to the Next Level

What if I told you just using multi-factor authentication (MFA) wasn’t enough anymore? The Lapsus$ hacking group, who were at least partially made up of a group of teenagers in the UK, took a very targeted hacking approach. They used password stuffing to try to breach the password credentials of power users within organizations they were targeting.

Once they identified the passwords (through a variety of tactics, but mostly password reuse—use a god damned password manager) , they sent 100s of MFA requests to multiple users. While a single user may have the discipline to ignore a series of MFA approvals that they didn’t prompt themselves, the odds are that if you send them to several people many times. Once that happens, and the attackers have an admin token, they can then move laterally, secure command and control, and do all manner of other bad things around credentials.

If this sounds scary, and it does to me, who is by far not an expert in all things security, but knows a little bit, you may ask, what are some alternative solutions? The answer to that question is Fido2, a different protocol for MFA and auth. Remember all of that stuff Microsoft talks about with passwordless login? That’s all based around Fido2. I configured this for DCAC’s Azure Active Directory yesterday, and I wanted to walk you through the steps.

Step 0 is to acquire a FIDO2 key for you and/or your team—I have a Yubico 5c, but there are others you can consider.

After that, the first step was to go to the Azure Portal and navigate to Azure Active Directory authentication methods.

Click on FIDO2 security key–even though it shows as enabled here, it is not enabled by default. When you click on the text you will see the next screen.

In this case, I enabled for All Users–this doesn’t mean they have to authenticate using this method, just that they have the option to. You also have some advanced options that go above my pay grade and are not happening in DCAC’s AAD.

Following this–I configured my Macbook to use my Yubi key as a authentication method. I followed the guidance on their site here. After doing that configuration, I was ready to make the change to my account. Navigate to myaccount.microsoft.com and select Update Info, under security info.

Once there, you can add a method, which is called “security key” here. I think this can be done globally in your org, but for this basic trial, I just enabled it for myself.

So that’s all of the prework you need to do. Now–let’s show logging into the Azure portal. You have to change the options in the portal as shown below:

Once you have selected Sign in with security key, you will prompted to choose the key. I was also promoted to touch the key (not captured in this screenshot) and then to enter the PIN you created when configuring your key.

Once you have entered your PIN, you are now authenticated to the Azure portal, without using a password. What I would love to see, but haven’t been able to configure is the ability to set a conditional access policy for AAD where untrusted location logins were required to use a harder level of authentication like a security.

Passing AZ-104–Azure Administrator

This Monday, I took and passed the Azure Administrator exam (Az-104) exam. It was a little bit unusual for me to take this exam, as I’m already an Azure Solutions Architect, but as part of the new Microsoft partner requirements, I had to take this exam, even though it’s a subset of what’s on the architect exams. Full disclosure: I didn’t study at all for this exam–I’m not saying that to brag, but if you are very experienced with Azure, particularly IaaS and Azure Active Directory, you can probably pass this exam cold.

clear light bulb
Photo by Pixabay on Pexels.com

Obviously due to NDA, I can’t disclose any questions on the exam, but I can review some of the high level topics you need to know. Some of the topics covered on this exam included:

  • Azure Virtual Machines
  • Azure Storage
  • Azure Networking (understand the various load balancer services)
  • Azure Active Directory user security
  • Azure Monitoring
  • Azure Policy

I didn’t feel like there was significant depth or advanced questions on any of these topics. Networking comes up a lot in any of these exams (and in my day to day to work with Azure, it’s exceedingly important). This is a good exam for you to take if you are just learning Azure, and want to validate your skills. If you are more advanced, I would focus on the architecture exams, unless you have to take this for Microsoft partner reasons.

PREEMPTIVE_OS_FILEOPS Waits and Filestream Restores

We had a case over the weekend where our automated restore process at client got hung up on this wait type, for a single database. What was the unique characteristic about this relatively medium (2-300 GB) database? It had a lot of filestream data–it seemed like the file count wasn’t that high, but my guess is the filestream data was the majority of the data in that database. When the job hung up, the restore had been waiting on PREEMPTIVE_OS_FILEOPS for over a day and still had a null value for percentage complete.

One interesting thing that happened was after I attempted to kill the restore process, it remained in place. My restore task was running in SQLCMD, so I went a step further, and killed the SQLCMD process on the server. The SPID in the database stayed alive, and since it was a non-production environment, and a weekend, I restarted the SQL service.

Per SQLSkills this wait type means “This wait type is a generic wait for when a thread is calling one of several Windows functions related to the file system”, and more commonly you see it on the end of a backup, when SQL Server is growing a log file, which does not benefit from instant file initialization (IFI). In our case the server did not have IFI enabled, and I suspect this was one of the contributors to the problem. After we enabled IFI, the restore complete in just under three hours.

EXEC sp_configure filestream_access_level, 2
RECONFIGURE
GO

CREATE DATABASE Filestream
ON
PRIMARY ( NAME = Arch1,
FILENAME = 'c:\data\archdat1.mdf'),
FILEGROUP FileStreamGroup1 CONTAINS FILESTREAM( NAME = FSData,
FILENAME = 'c:\filestream')

LOG ON ( NAME = Archlog1,
FILENAME = 'c:\data\archlog1.ldf')
GO

create table fs_table
(id INT IDENTITY (1,1),
UI UNIQUEIDENTIFIER ROWGUIDCOL NOT NULL UNIQUE,
FS_Data varbinary(max) filestream NULL)

Those are the database objects–I then used PowerShell to create some files, and generate an insert script.

$i=1
while ($i -le 10001)
{new-item -ItemType file -Path "C:\fstemp\" -Name Fs$i.txt -Value "Text file $i"; $i++}

$i=1
while ($i -lt 10001)
{add-content C:\temp\fsinsert.sql "`nINSERT INTO [dbo].[FS_Table] (UI, FS_Data) VALUES (NEWID(),(SELECT * FROM OPENROWSET(BULK N'C:\fstemp\FS$i.txt', SINGLE_BLOB) AS Image$i))"; $i++}

Those are just a couple of loops that create 10000 files and then insert them into the database. The files are very small (13-14 bytes), but it would be a very representative test. I kicked off a restore and ran procmon to see what the SQL Server was doing. SQL Server first queried the filestream directory for each file.

SQL Server is doing four operations for each file:

  • Create File
  • Query Standard Information
  • Write File
  • Close File

Since my files are very small, this happens very quickly, but it has to happen 5000x, which gives me enough time to prove this behavior. You should note that I was able to control for IFI being enabled–there was a performance improvement that I think is related to the number of operations. Instead of doing the four operations per file SQL Server appears to only creating and closing each file. Performance on my test instance was inconsistent, but I was working in a constrained VM (my CPU fan has been running all morning).

I suspect this restore process would be impacted by either a large number of files or a large volume of data. This can be confusing, as even though the restore process is running, this isn’t reflected in percent complete for the restore.

I hope you learned something in this post, I know I did. also, don’t #$%^ing store files in your database, unless you like hitting yourself with a hammer.

Why You Shouldn’t Use Amazon RDS for your SQL Server Databases

Disclaimer: I’ve a Microsoft MVP and shareholder, but neither of these things affected my opinions in this post.

Cloud vendors have built a rich array of platform as a service (PaaS) solutions on their platforms. They market these heavily, because they have higher degrees of stickyness compared to IaaS platforms (and in many cases they likely have higher profit margins), but they also have key benefits to the users of the platform. Because a PaaS solution is fully managed by the cloud provider, they tend have a set of common features:

  • Easy to deploy–you are never running setup.exe
  • Built-in high availability–no need to configure a cluster
  • Easy disaster recovery/geo-replication–usually in most services it’s just a few clicks away
  • Automated backups
  • Automated and possibly zero downtime patching

While some services include other really useful features (for example the query data collected by the Azure SQL Database and Managed Instance platforms), I wanted to focus on the common value adds to PaaS systems across providers. I made the last two of these bold, because I feel like they are are the most important, especially in scenarios where the vendor doesn’t own the source to the applications. Like Amazon RDS for SQL Server.

Amazon RDS Backups

I’m writing this post this week, mainly because of what I just learned this week about the backups for SQL Server on RDS. I was in some client calls this week when I learned the default backup approach is volume snapshots (which I knew), but what I didn’t know what that you couldn’t restore an individual database with these default backups.

I feel like their docs are deliberately vague about this–it’s not clearly obvious that this is the case, but a few Stack Overflow threads and discussions with fellow MVPs confirmed that I was told on the call. Amazon does support you taking your own backups on RDS, and you can then restore individual databases to individual points in time, but where’s that fun (and more importantly the value proposition) in that. To me, this really eliminates one of the biggest benefits of using a PaaS service. AWS refers to normal backup/restore as “Native Backup/Restore” as you are reading docs.

Patching

Azure SQL Database and Managed Instance, both seamlessly patch your databases without your knowledge. For the most part those services utilize hot patching, which means in many cases there isn’t any downtime to install the Azure equivalent of a CU. (Owning the source code has it’s benefits) Amazon RDS can automatically install selected CUs, but you should be aware that is an option when you deploy your instance.


"ValidUpgradeTarget": [
    {
        "Engine": "sqlserver-se",
        "EngineVersion": "14.00.3192.2.v1",
        "Description": "SQL Server 2017 14.00.3192.2.v1",
        "AutoUpgrade": true,
        "IsMajorVersionUpgrade": false
    }

The other thing you should note, is that you may not always have the most current CU available to you. Currently, AWS supports CU12, which is three CUs behind current. However, our customer was only on CU8–so patching doesn’t seem to be as automatic or easy as it is on the Azure side.

Licensing

This one really isn’t AWS’ fault (it’s Microsoft’s fault), but there are a couple of issues with licensing RDS. The first is that you don’t have the option of running Developer Edition for non-production workloads. Which, especially if you are running Enterprise Edition represents an expensive choice–you either need to license Dev/Test for Enterprise, or run Standard Edition in dev to save some money, but not have adequate features in place (though the code platform is mostly the same, performance characteristics can be dramatically different). Additionally, you cannot bring your own SQL Server licenses to RDS, you have to lease them though AWS. Neither of these problems are the fault of AWS, but they still suck.

AWS is an excellent cloud platform, even thought I actively hate their console. For most part a lot of components are very similar or even better than Azure. However, when I comes to a service where Microsoft owns the source code and AWS doesn’t, you can see the clear superiority of Azure. So what is an AWS shop that runs SQL Server to do? IMO, the backup/restore thing is a deal breaker–I would just recommend running in an EC2 VM.

Fixing SQL Server Database Corruption (when you get lucky)

First things first–if you are reading this, and not regularly running consistency checks on your SQL Server databases, you should drop everything you are doing and go do that. What do I mean by regularly? In my opinion, based on years of experience, you should run DBCC CHECKDB at least as frequently as you take a full backup. Unless you have a clean consistency check, you don’t know if that last backup you took is valid. SQL Server will happily back up a corrupted database.

Screenshot of a failed checkdb, followed by a successful backup command

I cheated a little bit here and used an undocumented command called DBCC WRITEPAGE to corrupt a single 8kb page within an non-clustered index on a table I created. You should basically never use this command, unless you are trying to corrupt something for a demo like this, but as you can see, after we’ve corrupted the page, SQL Server fails CHECKDB, and then happily takes a backup of our now corrupted database.

What Causes Database Corruption?

Other than doing something terrible, like editing a page, database corruption is mostly caused by storage failures. An example of this could be your local SAN, where the SAN’s operating system acknowledges that a write operation is complete back to the host operating system, but the write doesn’t actually complete. SQL Server recieved the write acknowledgment and thinks that the data was correctly written to the page, however for whatever reason it didn’t happen. I had this happen a couple of times in a past job when the SAN in our “data center” (it wasn’t) crashed hard when the building lost power (yeah we didn’t have a generator, hence the quotes. Be careful who you buy your medical devices from). What was actually happening is that the SAN was acknowledging a write when the data hit memory on the SAN, which is a performance enhancement, that assumes you have a proper power infrastructure that will prevent the SAN from “crashing hard”. You know what happens what you assume, right?

Anyway, this is far less common than it used to be for a number of reasons, one of which is the use of cloud based storage, which is very robust in terms of data protection. Also, modern enterprise class SANs are more efficient, and less likely to have failures like this. However, it’s still very possible–I had a minor corruption event in an Azure VM a couple of years ago, and we had a customer who filled up their very non-enterprise class SAN, with terrible results (it was all the corruption). So the moral of the story, is wherever you are running SQL Server you need to run checkdb. (Except Azure SQL DB, and possibly Managed Instance).

Fixing Corruption

There is a lot of stuff and tools that people will try to sell you on the internet to fix your database corruption. Almost all of them are crap–if you have corruption in a table or clustered index, or worse in one of the system pages that determines allocation, you are screwed and need to restore your last good backup. (see why backup retention matters here?)

However, in some cases you can get lucky. If your corruption is limited to a nonclustered index, you don’t need to restore the database, and you can just rebuild your index.

However, in my case that just threw the dreaded SQL Server 824 error. I suspect this had something to do with how I corrupted the page, but that investigation is not complete. I was able to disable the index, and then rebuild and we had a sucessful CHECKDB.

Getting Started with Log Replay Service for Azure SQL Managed Instance

Recently, I’ve started on a project where we are migrating a customer to Azure SQL Managed Instance, which now supports a few different migration paths. You can simply backup and restore from a database backup, but you can’t apply a log or differential backup to that database. You can also use the Database Migration Service, but that requires a lot of infrastructure and Azure configuration. The log replay service, or LRS, is the functional equivalent of log shipping to your new managed instance database. While log shipping is a well known methodology for both database migrations or disaster recovery. However, the implementation is a little different–let’s talk about how it works.

flock of birds flying
Photo by Shakhawat Shaon on Pexels.com

First, You Must backup to URL

In order to use the log replay service, you need to take a backup of your database to Azure Blob Storage. Alternatively, you can push on-premises backups using AzCopy or uploading files into the portal or Storage Explorer. But let’s focus on backup to URL. The first thing you need to do is create a credential in SQL Server. While there are other methods of creating a credential, I have had absolutely the best luck, using this PowerShell code which produces a CREATE CREDENTIAL statement to execute on your SQL instance.

CREATE CREDENTIAL [https://storageaccount.blob.core.windows.net/sqlbackups] WITH IDENTITY='Shared Access Signature', SECRET='sv=2020-08-04&si=sql1&sr=c&sig=Dw0uE13l2347%2FCNJgQbm5sHhM9ZtDts9F8SMi5Re1ASUw%3D'

The SAS token you create is created on the container level–you will also need to create a policy (that code from MS will create everything for you including the storage account). Next execute the backup.

EXECUTE dbo.DatabaseBackup@Databases = 'DB1',@URL = 'https://storageaccount.blob.core.windows.net/sqlbackups',@BackupType = 'LOG',@Compress = 'Y',@Checksum = 'Y',@Verify = 'Y',@DirectoryStructure = '{DatabaseName}',@NumberOfFiles = 8,@MinBackupSizeForMultipleFiles = 10240,@logtotable=y

I’m using Ola Hallengren’s code to do my backups, but you can also do this manually.

BACKUP LOG [db1] TO URL = N'https://storageaccount.blob.core.windows.net/sqlbackups/db1/687593983fe8_db1_LOG_20211216_162916.trn' WITH CHECKSUM, COMPRESSION

I highly recommend using Ola’s code to perform these backups and all of your backups. A couple of things you should note–you have to use the checksum option, and with Ola’s backups I’m using the Directory Structure of {DatabaseName}, The reason for this is that LRS does not support nested directories, like Ola’s code would default to, nor does it support back ups stored in the root of the container. (You should note the backup example on the LRS doc page does just this).

Docs also say you need a full, differential, and log backup to do a complete migration, however in my testing I’ve been successful with just a log backup in addition to a full. To complete your restore process, you will need to have the name of your last log backup. You can use this code to generate the powershell code to initiate the migration process.

DECLARE @DBName SYSNAME;

DECLARE @RGName VARCHAR(256) = 'ResourceGroupName'

DECLARE @MIName VARCHAR(256) = 'ManagedInstanceName'
DECLARE database_cursor CURSOR FOR

SELECT name FROM MASTER.sys.sysdatabases where name not in ('master','msdb','model','tempdb')

create table #migration (command nvarchar(max))

OPEN database_cursor FETCH NEXT FROM database_cursor INTO @DBName
WHILE @@FETCH_STATUS = 0 BEGIN

SELECT @lastlogbackup = REVERSE(SUBSTRING(REVERSE(physical_device_name ),1,CHARINDEX('/',REVERSE(physical_device_name )) - 1))FROM msdb.dbo.backupset bs INNER JOIN msdb.dbo.backupmediafamily bmf ON [bs].[media_set_id] = [bmf].[media_set_id]WHERE (bs.database_name = @DBName OR @DBName IS NULL) AND bs.type = 'L'ORDER BY bs.backup_start_date DESC;

insert into #migration select ‘Start-AzSqlInstanceDatabaseLogReplay -ResourceGroupName “'+@RgName+'" -InstanceName "'+@MiName'" ` -Name "'+@dbName+'" `-LastBackupName "'+@lastlogbackup+'" -Collation "SQL_Latin1_General_CP1_CI_AS" -StorageContainerUri "https://storageaccount.blob.core.windows.net/sqlbackups/'+@dbname+'/" -AutoCompleteRestore -StorageContainerSasToken "sp=rl&st=2021-12-14T21:25:07Z&se=2021-12-25T05:25:07Z&spr=https&sv=2020-08-04&sr=c&sig=h8%2FimcTh%2BnlCQ3WodgTShWERT7yA38HfBQFOLwvEoiqw%3D"'
FETCH NEXT FROM database_cursor INTO @DBName

END

CLOSE database_cursor DEALLOCATE database_cursor

select * from #migration

For a small database, this only takes a few minutes. If your PowerShell execution goes out to lunch, and your database is stuck in restoring, there is probably something wrong with the way your formatted your backup string. This also manifests itself in the form of http 204 error in the log of your managed instance.

Message

Sending the notification action: NotifyRestoreProgress

Date                       12/9/2021 8:09:42 AM

Log                         SQL Server (Current – 12/9/2021 4:27:00 AM)

Source                  spid101

Message

Http code after sending the notification for action NotifyRestoreProgress: 204

What’s annoying is that the PowerShell doesn’t attempt to do any validation of requirements. Hoepfully, this will change in the future because this was a challenging problem to troubleshoot.

Message

A Post-mortem for PASS, and What That NDA* Issue Was

Editor’s Note: I wrote this last December, and went back and forth on publishing. As I see some of the poor leadership patterns from PASS, creeping into new offshoots of the SQL Community, I decided to publish it today.

I am not writing this to dance on the grave of PASS or throw pitchforks at anyone. Just like the rest of the community, I am disappointed at the end of PASS, as the community worked really hard to make the organization great. However, a lot of mistakes were made over the course of many years, and I think it’s important to talk about those mistakes, and how we as a community can avoid them in future endeavours. The biggest failure was not protecting community assets like SQL Saturday and the Virtual Chapters that were built by the sweat of the community. But first I’d like to talk about the “NDA” violation the came up while I was running for the board.

The reason why I’m including this story into my post-mortem is that the board NDA was a major organizational problem and was abused by senior leadership at C&C, in order to in my opinion hide a lot of bad practices that took place within the organization. Nearly every interaction with PASS required an NDA—for a community organization this notion is ridiculous. There are small set of activities that actually require an NDA—typically harassment cases and personnel matters like salaries. Other than that, all activities and budgets should be fully transparent. One of the biggest issues that happened in PASS’s failure was that members at large on the board of directors did not have a clear picture of PASS’s finances until it was much too late. This stonewalling was not for a lack of effort—even to members of the board PASS did not operate transparently. This behavior continued until the very end of the organization.

About that NDA Violation

I’d also like the clarify the difference between whistleblowing activity and NDA violations. Whistleblowers report things when they see their leadership acting with maleficence, and whistleblower activity nearly always violates NDAs. Whistleblowers have exposed government, and corporate negligence and saved lives and costs.

Many people reached out to me after the posts I wrote last year. There was rightful and genuine concern for PASS—despite frustrations there was a lot of emotion for PASS, and more important our community has been largely structured around a centralized organization. As people reached to me, I learned fairly early about the plan to ask Microsoft for a very large bailout (on the order of millions of dollars), in an effort to save PASS. Part of this plan involved selling Microsoft a large number of PASS Pro memberships.

Given that PASS had no plan for a revenue stream in 2021, this funding project seemed absurd, and at PASS’ current burn rate they would require another bail out next year. I had numerous sources for this information, and while I won’t reveal the sources, they included community members and C&C employees. I did share this information with selected individuals at Microsoft as well, but I never revealed it publicly. I also shared information in confidence to a few selected individuals One of those people decided that was a violation and reported it to PASS exec. I have a couple of things to say about this—first of which is that in the application that prospective board members sign, there is no reference to an NDA, and I was not under contract with PASS. I’m not a lawyer, but I took a few semesters of business law, and if there is not an NDA in-place, there is no expectation of privacy. In the end, all of that didn’t matter, as PASS is now dead.

A Board Operating in the Dark

In my discussions with former board members, one topic that constantly came up was the complete lack of transparency C&C showed, particularly when budgets and spending were discussed. Multiple Board and even exec members reported to me that they gave explicit directives to C&C only to have those directives be ignored.

Much of the blame for the end of PASS will be laid onto the Covid-19 pandemic taking away the primary revenue source, in-person Summit. While this is the direct cause for PASS’ failure, the truth is that PASS has been on a downward slide for several years. In preparing to write this post, I spoke to board members across several generations of the board and deeply involved community members to gain insights into what happened. There are a number of things that have always concerned me about PASS—the lack of transparency the organization has always, always been the core problem.

C&C’s Conflicts of Interest

Why do I think there was such a culture of secrecy in the organization? I think it stems from the management organization, and its leadership. C&C always acted in the best interests of C&C, and not the best interests of PASS or the Microsoft Data community down to the final days of the organization. C&C was paid by PASS as a percentage of PASS’ spending—while this is somewhat common in event management circles, in organizational management circles it can lead to perverse outcomes, and likely what was ultimately responsible for PASS’ bankruptcy.

C&C had so many conflicts of interest I don’t even know where to begin talking about them. The CEO’s sister was responsible for the finances of both PASS and C&C, and C&C held a non-voting seat on the executive board. While many members of the board would say this was helpful as PASS and C&C operated in partnership, it is probably the most explicit and visible conflict of interest. The fact that the organization providing all of your services knows what your bank balance and income are problematic at best and can lead to poor outcomes. It also means your partner has no real interest in saving the organization money. C&C had no other clients other than PASS, which is likely a violation of IRS and Revenue Canada employee/contractor regulations.

I’ll share some specific examples, but what this practice led to was a singular focus on PASS Summit, because that’s where the spending (and revenue) is for the organization, and therefore C&C. This came at the expense of community events like user groups and SQL Saturday events. This also led to C&C influencing heavily decisions like spending a ton of money on a virtual Summit that was unlikely to be successful, or spending a lot of money on PASS Pro, which effectively had no hope of ever being profitable. While PASS’ noble mission was “Connect, Share, and Learn” in practice it became “whatever it takes to drive C&C’s revenue growth”.

Lost Opportunities

One of the challenges PASS faced well before Covid was lack of growth (and even a downturn) in the number of people attending PASS Summit. Throughout my time with PASS the board acknowledged this single source of revenue as a risk and made a few attempts at growing the community or diversifying revenue. One of these was the Business Analytics conference, which was initially driven by Microsoft, but could have evolved into something much bigger. As opposed to a technology only conference, business analytics can be marketed to wide swath of data professionals and not just technologists. The marketing for this conference was handled poorly, as C&C tried to market just as though as it was PASS Summit.

This led to PASS turning down the opportunity to run Power BI user groups, which in turn led to the successful Microsoft Business Analytics (MBAS) conference, which could have both increased revenues, and more importantly grown our data community.

Another story shared with me, was that Microsoft wanted to make PASS Summit a free event, in order to dramatically grow the size of the conference. While this is a bit of one-upmanship between software vendors as to who has the largest conference, this had to potential to exponentially grow the PASS community. Microsoft was willing to fund the revenue difference, but this was shot down by C&C, presumably because the initiative would have cost them money. This goes against PASS’s stated mission of Connect, Share, Learn, and is just one of myriad of bad acts perpetrated by C&C through the years that ultimately doomed PASS.

What Happened in 2020

There are countless examples of C&C’s actions that were detrimental to the broader community to try and drive Summit attendance and revenue, but I’d really like to focus on the actions that took place this year, which lead to the death of PASS. When the pandemic hit in February and March, it became very obvious to everyone that having large conferences was an unlikely proposition until a vaccine was in place, and even then, probably a year after that.

Assuming people are mostly vaccinated by the middle of 2021, that means the best hope for an organization who’s revenue was dependent on conference revenue would have been to move to an austerity budget which limited services to completely essential services, until a time, probably in 2022 when a large scale conference was a good bet. This was bad news for PASS, because its major revenue source was compromised until at least 2022. (Editor’s note: I wrote this last year, and it has mostly held true)

Even in lieu of an austerity budget, what PASS could have done in April 2020, was to move to protect the assets that the community built like SQL Saturday and the Virtual Chapters, and now were at risk due to multiple organizational failures. The executive board and C&C executed none of these options and kept the board members at large as well as the general membership in the dark as to how perilous the financial situation was.

There were additional poor decisions made in 2020. The Exec and C&C decided to go along with PASS Pro, a paid membership offering, that was built on a learning management system. While a noble concept, it was competing against mature services like Pluralsight and LinkedIn Learning. There was some Microsoft funding to this project (my understanding is around $300k USD), and several community members were paid (around 3-5K USD) to record videos for the service. While a lot of money was spent to build and launch the service, there was never hope of it being a significant revenue source for PASS.

Despite what C&C has always said about PASS having 300,000 members, based on better estimates, PASS likely had < 5000 “active” members and a maximum of 20-30,000 “passive” members who unwittingly signed up by attending a SQL Saturday or PASS Summit. As of November, PASS Pro had only sold 300 memberships (note some of these may have been corporate memberships which included up to six subscriptions). Even if we assume all 300 of those were corporate memberships (which I know not to be the case), that provides PASS $180,000 of revenue. While that is not nothing, for an organization with PASS’s expenses, it is a completely insignificant amount of revenue, and since PASS/C&C had no real plans for a content pipeline, it was unlikely to grow over time, instead being more likely to fade away so like many other PASS initiatives.

Which brings us to Virtual Summit. I don’t have a final accounting to know what this event cost, but after PASS’s bankruptcy we know it was responsible for the death of the organization. PASS was very slow to cancel in-person Summit, with the virtual event was not announced until June. My understanding is that part of this was because it was a challenge to get out of the contract with the Houston Convention Center, however it put PASS in place where lots of other (sponsor funded) conferences had gone virtual and more importantly to this discussion, FREE.

I’m of the opinion that C&C leadership took this opportunity to apply the death blow to PASS. Instead of moving into an austerity budget, and taking assistance for revenue loss from the Canadian government, PASS decided to go “all in” on the virtual conference using a very expensive virtual conference platform (that was ultimately derided by both speakers and attendees), that promised a “unique, interactive” conference, that ended up just being a really fancy wrapper for a Zoom meeting. This was defended in the December board meeting minutes, that in order to protect PASS’s cashflow, executing a virtual conference was a necessity.  

The pricing of the virtual conference was lower than in-person Summit, though my understanding was that C&C was pushing for significantly more expensive pricing than the final number. C&C sold the board on the notion that since people didn’t have to travel to an in-person event, they would have a potentially larger audience than at an in-person event. That assumption was flawed from the start, as it didn’t account for the pandemic recession, virtual meeting fatigue, and the fact that most of the other conferences PASS usually competes against, were FREE.

PASS weakly attempted to highlight how the conference was going to be different and more interactive than other conferences, but they never had a free trial event to build hype, and C&C was ineffective at marketing the virtual conference. During the entire summer and into early fall there were few emails or tweets about the conference, other than short video describing the platform.

I lay the responsibility for death of PASS on C&C largely, but secondarily on PASS’ executive leadership. In good times, some of the decisions made vis a vis C&C were poor, but they didn’t matter given the revenue streams. A microcosm of this is that PASS sent speaker shirts to 2020 Virtual Summit speakers. Anyone who’s ever run an event knows that the first two things you cut when your budget is not looking good, are speaker shirts and speaker’s dinner.

The decisions made by PASS in 2020, from launching PASS Pro, to betting the organization on a virtual conference, and finally the decision to ask Microsoft for a major bailout, with no significant plan forward, show a complete lack of leadership from anyone on exec, instead deferring all decisions to the deeply conflicted C&C organization, which had shown repeatedly they only cared about their own interests and not that of the broader data platform community. If you are waiting on a refund from PASS Summit, you will have to stand in line behind C&C, because they are getting paid first. Like always, Judy got her money before the community did. And that just fucking sucks.

Like much of the community, I am lamenting the loss of PASS, but I feel the to call out executive leadership, not just for their leadership failings (which are many), but for their passive aggressiveness and general rudeness to community members who cared about the organization. These actions not only took place against community members, the abusive behavior extended to other members at large of the board of directors.  The exec did everything they could to keep the rest of the board in the dark on the financial situation all the way to the bitter end of the organization. Board members were chastised for asking hard questions and kept in the dark before votes. There were never open and honest communications between the board members at large and the exec and C&C. While in normal times this meant C&C got resources that should have gone to the community, in this pandemic year, it meant the complete death of the organization without protecting any of its assets. This is not a recipe for a strong organization. That secretive behavior did nothing to help, and actively alienated many active and senior members of the community, who otherwise could have been strong allies to the PASS cause.

I am thankful for the existence of PASS, and the community won’t go away. I don’t want to necessarily focus on what’s next—yet. I think the best thing community members can do is keep running their user groups, and hopefully in the second half of 2021, we can start having data events on Saturday, whatever they are called. I think the best approach to a larger organization is to have a federation of larger events, so they can coordinate on things like scheduling, and have a unified code of conduct and diversity and inclusion policy. Having a centralized organization comes with a lot of expense, so starting small and building is the right approach. Andy Mallon has a good vision for this in his post here.

Passing the AZ-500 Exam–How I Prepared

A few months ago, I was asked to do a webinar on security for Microsoft Teams. In preparing for that webcast, I noticed a number of security features that required either Azure Active Directory P2, or Office 365 M5. In order to get either of those two features for free, DCAC needed to gain the silver security competency from Microsoft. In order to do this one of us, had to pass wither MS (the Office/Microsoft 365 version) or the AZ-500 security exam. Since, I was the one who wanted the features, I volunteered to take the exam.

The first time I took the exam, I didn’t do much preparation. I was pretty busy, and I recently taught some Azure Architecture training and I was hoping that would be enough to get me through the exam. That almost worked–I came pretty close to passing, and likely didn’t pass because I was frustrated with some questions and kind of blew through them. Pro-tip: don’t schedule an exam for Monday at 4 PM.

The second time I took the exam, I prepared a little more. Here were the key topics I focused on:

  • Azure Active Directory
  • Azure Sentinel
  • Azure Role Based Access Control (RBAC)
  • Azure Networking
  • Azure Key Vault

You can see the full exam outline here. I spend a lot of time working on most of these technologies in my daily job, so this was more refreshing my memory on the specifics of how each of them work. I also paid for the practice exam, which was helpful but likely not necessary, but it did get me more into the exam mindset.

The second time I took the exam, I didn’t find the questions as friendly to my knowledge, and generally thought the exam was harder. However, this time I passed the exam, fairly easily. Yay, a week later we had P2 AAD so I can now do cooler demos. My general recommendation to folks who want to take cert exams, is to just dive in and take them as it’s really the best way to study, and know the areas where you are weak.