An Unusual R Services Problem

I have had the good luck of having a customer that was onboard with SQL Server 2016 very early—like we started testing in March of 2015, and went live in August of 2015. In fact, their home directory refers to vNext instead of 2016. This customer also adopted what felt like most of the new features list. Temporal tables, columnstore, PolyBase, and R Services amongst other features. You can read more about this great use of SQL Server 2016 here. Anyway, we had R up and running, and it ran for a while.

Recently, and unfortunately I don’t have an exact date on when this started failing (though it was around service pack 1 install time) with the following error:

Error
Msg 39012, Level 16, State 1, Line 10
Unable to communicate with the runtime for ‘R’ script. Please check the requirements of ‘R’ runtime.
STDERR message(s) from external script:

DLL ‘C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER1601\MSSQL\Binn\sqlsatellite.dll’ cannot be loaded.
Error in eval(expr, envir, enclos) :
DLL ‘C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER1601\MSSQL\Binn\sqlsatellite.dll’ cannot be loaded.
Calls: source -> withVisible -> eval -> eval -> .Call
Execution halted
STDOUT message(s) from external script:

Failed to load dll ‘C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER1601\MSSQL\Binn\sqlsatellite.dll’ with 182 error.

I troubleshot this with some colleagues at Microsoft and we weren’t able to resolve. We tried a couple of different approaches including reinstalling CU1, but all to no avail. Yesterday, I got on a call with a couple of folks on the product team to try an isolate the problem. We looked at binaries and timestamps and it looked like everything matched up. Then, my friend Arvind Shyamsundar (b|t) suggested we run procmon on the server.

image

There Arvind noticed these odd calls to sqlos.dll in the shared directory. We then looked at add/remove programs and found the following item installed:

Screen Shot 2017-08-11 at 8.19.38 AM

The T-SQL compiler service which was a legacy of CTP 2.3 was there, and as soon as we uninstalled our problems went away. So, if you happen to be running on a server that you’ve upgraded since very early versions of SQL Server 2016, you may see this issue.

Just to give credit to Microsoft and the current SQL Server install process, this server has had nearly every release of SQL Server 2016 on it (we’re behind a couple of CUs), and this is the only issue we’ve had. Thanks again to Arvind and UC for solving this tough issue.

Analyzing Your Dump Files

I’m blogging about this, because A) It’s something really awesome that the SQL Server team built and B) it seems to have terrible SEO, because it took me like three google searches to find the page. With the introduction of SQL Server Management Studio 17, the Tiger team at Microsoft built a plugin that allows you to debug and resolve memory dumps you may have encountered during otherwise normal operations. This is really awesome, as it is something that usually requires a support case with CSS.

For those of you wearing aluminum hats, this does require you do upload the dump file to Azure, where it is analyzed for free (as in beer) on Microsoft’s software. You can even choose your region if you have data provenance concerns. And according to this blog post the memory dumps are stored in accorded with Microsoft’s Privacy Policy.

You will need SSMS 17 for this, as well as to install the plug in, which you can get here.

image

After that you can quickly get feedback on your dumps. Microsoft have even built an API, so if you want to built something automated to upload your dump files using Python or PowerShell you can.

%d bloggers like this: