Consuming a C++ DLL in dotnet core

For a client, I need to open a drawing in a specialised program. When the user closes the program it should upload the drawing to a server. What looks like a straightforward start of a process, turned into a mindbogglingly number of failures. Journey with me through all the steps I took to get this seemingly simple task to work.

After a quick Process.Start('program', 'path/to/drawing') didn’t work, I knew something was wrong. So I contacted the developers of the software and asked for clarification. They sent over very helpful documentation and I was on my merry way again.

Apparently, they have a .dll that is included with their program and that is how an external application can communicate with it. Phew, not so hard after all. So I modelled the external calls to methods and decorated them with the [DllImport] attribute.

private delegate int Callback(int code, string message);
[DllImport("path/to/dll", CharSet = CharSet.Unicode, SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
private external static int RegisterCallback(Callback method);

[DllImport(/*see previous*/)]
private external static int StartApplication();

[DllImport(/*see previous*/)]
private external static int DoWork(string command);

In this flow, which is apparently more common in C++ than dotnet, I first register a callback method, start the application and then send work to it. The registered callback method handles status updates and other information that gets sent back. This way, a single marshalling needs to be done and multiple DoWork methods can be called, even different external functions, and they’ll all report back to the registered callback method.

The code parameter in the callback method determines what will be in the message and how to interpret it. For example, the code 0 can refer to application initialisation and the message then is “SUCCESS” or “FAILURE: with a specific reason”. More complex information can be sent back as well. For example, code 2 could be that the application has saved the drawing, the message then contains XML with the location of the drawing and some meta information about the changes that were made.

Although this is quite an interesting approach, as most examples of interop use a specific callback in each function. This approach however, does pose a few interesting challenges. I found that the callback was never invoked. To get more error information, I had to enable SetLastError = true. This then allows me to call Marshal.GetLastWin32Error(). That function returns a number that I can then look up in the official Microsoft docs.

Unfortunately, I forgot the error so I can’t reference it here. It did lead me to an article to set the CallingConvention to Cdecl. This is to indicate that I am responsible to clean the stack and not the code that I’m calling. This has to do with defaults in both runtimes, C++ uses __cdecl convention while the default for C# is __stdcall. The conventions need to align to allow the processes to talk to each other.

Huzzah, done… No wait, too soon, still no callback. The application that I am calling is starting, but it also gets cleaned up too fast. I’m not sure what C++ techniques are used, but either Windows or the GC are cleaning up the application before it could properly start.

What I did notice is that the StartApplication() method returns almost immediately and I have to build a way to wait for the callback that says that the full initialisation of the application is done. I assume that the StartApplication method sets a process in motion and reports that the process kicked off correctly. Dotnet on the other hand figures that the invoked method was done and doesn’t need all those resources anymore, including the application that is halfway through its startup process.

If anybody is interested in the wait mechanic that I tried here: I used a TaskCompletionSource<bool> to return whether the initialisation would have completed succesfully based on the message the callback would receive. This way, I could use await to wait for the external program to finish.

Here’s where the developers of that program gave me a helping hand: they told me dotnet needs to keep a reference to the method that is being called and the way to do that is to use functions from the Kernel32 library to load the functions into memory and only release them when they are done.

[DllImport("kernel32.dll", SetLastError = true)]
private static extern IntPtr LoadLibrary(string pathToLibrary);

[DllImport("kernel32.dll", SetLastError = true)]
private static extern IntPtr GetProcAddress(IntPtr libraryReference, string procedureName);

[DllImport("kernel32.dll")]
private static extern bool FreeLibrary(IntPtr libraryReference);

So instead of using a DllImport, I have to do this by hand.

The LoadLibrary function is used to load the library into the running application. The IntPtr that is returned is a pointer to a memory address where the library is loaded. The GetProcAddress function takes the pointer returned by LoadLibrary and the name of the exposed method. When the library has fulfilled it’s purpose, so after the DoWork function is done and the callback method has received the done signal, I need to clean up the references to the library by feeding the pointer to the library to the FreeLibrary function.

It’s comparable to how DllImport works behind the scenes. Except that DllImport is applied to a method that can be called. The GetProcAddress only returns another pointer to where the function I want to call is loaded into memory. To convert the pointer to an callable function, I need to marshal it. In full dotnet framework, there is a static function Marshal.GetDelegateForFunctionPointer which takes the IntPtr and the Type of a delegate. It then returns an object that can be cast to the delegate. If you are using dotnet core, there is a generic function Marshal.GetDelegateForFunctionPointer<T>.

internal class CustomApplicationIntegration : IDisposable
{
  // kernel32 imports here
  private readonly IntPtr _dllPointer;
  public CustomApplicationIntegration(string pathToLibrary)
  {
    _dllPointer = LoadLibrary(pathToLibrary);
  }

  public void Dispose()
  {
    if (_dllPointer != IntPtr.Zero)
      FreeLibrary(_dllPointer);
  }

  private delegate void Callback(int code, string message);
  private Callback _keepAlive;
  private bool _applicationDone = false;
  private delegate int RegisterCallback(Callback method);
  private delegate int StartApplicationDelegate();
  public void Interact()
  {
    _keepAlive = CallbackMethod;
    var registerPointer = GetProcAddress(_dllPointer, "RegisterCallback");
    if (registerPointer == IntPtr.Zero)
      throw new Exception("Something went wrong!");
    // dotnet 4.X aka full framework
    var registerCallback = (RegisterCallback) Marshal.GetDelegateForFunctionPointer(registerPointer, typeof(RegisterCallback));
    var registerResult = registerCallback(_keepAlive);
    // check that registerResult returns ok

    var startPointer = GetProcAddresss(_dllPointer, "StartApplication");
    // dotnet core
    var startApplication = Marshal.GetDelegateForFunctionPointer<StartApplicationDelegate>(startPointer);

    var startResult = startApplication();
    // check that startResult returns ok
    while (!_applicationDone)
      Thread.Sleep(100);
  }

  private static void CallbackMethod(int code, string message)
  {
    // handle callback
    _applicationDone = code == StopCode;
  }
}

If something goes wrong in either the LoadLibrary or GetProcAddress, they return the value IntPtr.Zero. That’s how I can check for success, everything that is not the zero value is a valid pointer. Because the SetLastError is set on the DllImports of these functions, I can get the id of the error with the Marshal.GetLastWin32Error function, just like I did earlier.

At the end of the Interact method, there is a loop to keep waiting on the external application to send the stop code. Only when that code is received, can I continue with my application. There are functions in the library to do specific tasks and to stop the external application, but I’m omitting them here for brevity.

In the actual app, there are several loops waiting for the external application to finish some work or signal some status changes. For example, I have to wait for the application to start, before I can assign any work to it.

Now lets run this thing and see… still… no… callbacks… What I can see, is that the application is still exiting, but now it takes longer. The startup logs indicate that the library is doing it’s job correctly, but that the process that is started by the library is being terminated. Windows, stop killing my processes!

The Task.Run is to blame here! I run the Interact function in a Task and it runs a while longer, but it doesn’t protect the process the library is starting. When I changed this to a Thread, it solved the matter. To be honest, I have no idea why the Task terminates early, while the Thread runs as intended. If anybody knows why this is, do get in touch.

// bad way
await Thread.Run(() =>
{
  var integration = new CustomApplicationIntegration();
  integration.Interact();
}

// the good way
var workThread = new Thread(() =>
{
  var integration = new CustomApplicationIntegration();
  integration.Interact();
});
workThread.Start();

I hope this runs, I hope this runs and… I wait forever on the callback function. Don’t give up hope, this is the last problem to solve, but what a problem it is. The issue is that I created a deadlock: the external application invokes the callback and waits for it to return (eventually it would crash after a looong time) and my code waits for the callback but is too busy waiting to notice the callback.

Because it’s a console application, there is no main event loop. The console expects there to be only one way through the application. It can wait on async methods, but it does not expect any application events. What this means is that it does not see the event of the callback, because it does not originate from within the application.

Simple callbacks that happen within the execution of the called C++ code expect the callback method to be invoked. Because I registered the callback and then went on to other functions that do not have that callback explicitly referenced, the console is not ready for these invocations. Instead, the Windows event loop is used. Since the console app does not have a main event loop, they simple do not get processed. This is the same reason why Timers and Sockets do not work as expected in console applications.

The most simple solution to this problem is to convert the console app to a WinForms app and call Application.DoEvents() in the while loop in the CustomApplicationIntegration class.

while (!_applicationDone)
{
  Thread.Sleep(100);
  Application.DoEvents();
}

To me, it feels a bit hacky. That is because in a normal WinForms app, the Application.DoEvents() method gets handled by the framework. It has a number of guards to handle concurrency when multiple events fire in quick succession. The official Microsoft documentation seems to discourage manually waiting on Application.DoEvents. Since I’m only expecting events from one external source, there should be no problems. Should…

Wow, that was a lot to take in. From DllImports, over marshalling functions and waiting for an asynchronous response to finally transforming the console app to a WinForms app so Windows events could be handled. Somebody has earned their favourite beverage, so go on and treat yourself to a nice beer, wine or two fingers of whiskey. I know I’m going to enjoy all myself now.

Specflow GenerateFeatureFileCodeBehindTask error

In the past few months I have been contacted by a number of readers who notified me that the code for my A simple SpecFlow 3 setup in Rider blog post was broken. I took me a while, but I’ve finally taken the time to fix it.

Some time ago I already looked at it and I saw that with the upgrade to dotnet core 3 there was a breaking change with SpecFlow .feature.cs file generation. Due to a change in the MSBuild pipeline, the files were not generating correctly. The following error was shown during the build of the solution.

The “GenerateFeatureFileCodeBehindTask” task failed unexpectedly.

MSBuild

Fortunately, upgrading to the latest version of all the packages fixed all the errors. Thank you SpecFlow team!

The SpecFlowSetup project on GitHub has been updated and should work again.

Set up MTA-STS on a GSuite hosted GitHub pages

To further protect my email communication, I have enabled MTA-STS on my GSuite domain. My site is hosted on GitHub pages, so I’ll walk you through my setup.

It starts with creating a new GitHub repository that will hold the files for the MTA-STS subdomain. For some reason, the config for the MTA-STS is read from an mta-sts.txt file, located in the .well-known folder, but it has to be loaded from the mta-sts subdomain. Why it can’t be done from the main domain is beyond me, but here we are.

Now that I have a repository, I create the .well-known folder and I place the mta-sts.txt file inside that folder. The content of the file can be found in my GSuite Admin section. It is the middle value: MTA-STS Policy Diagnostic. I’ll come back to the other values shortly.

Unfortunately, this is where I bumped into the problem with hosting on GitHub pages. By default, it does not expose folders starting with a . (dot). Probably because the servers are Linux based and any Linux folder starting with a dot, is automatically a hidden folder. So Stack Overflow to the rescue!

The fix is as easy as adding a _config.yml file to the base repository with the single line:

include: [".well-known"]

Important detail: do not end with an empty line! Just add that single line to the file to expose the .well-known folder.

The last step in GitHub is to set up the custom domain for this repository. It’s pretty easy to set up a GitHub pages domain, just be sure to include the subdomain before your domain.

Don’t worry if GitHub displays an error, I have not set up the subdomain DNS yet, so it can’t find the setup for the domain just yet.

I’ll fix that right now. I let Cloudflare handle my DNS settings. In the DNS settings of the dashboard, I add 4 A records with the name of mta-sts, one for each IP-address that GitHub pages can handle. For more information about the specific setup of GitHub pages, I refer to their good documentation. Now that the IP redirects are set up, the subdomain should be ready and available.

Two more steps and I’m done. Luckily for me, they are both in my DNS setup. I add a TXT record with the name _mta-sts and the value found in my GSuite Dashboard after “MTA-STS TXT Record Diagnostic”. I add another TXT record with the name _smtp._tls and the value found in my GSuite Dashboard after “Reporting Policy Diagnostic”.

Do not forget to change the rua=mailto: value of the “Reporting Policy Diagnostic” text to an email address which you can receive. That is where reports will be sent to. In the near future, Report URI should get support to process the values.

Now I enjoy more secure email communication. If anybody wants to learn more about SMTP MTA Strict Transport Security, I recommend reading Scott Helme’s very good blog post or URI Ports expanded blog post. That’s where I learned about it.

Edit: Thanks Faisal from emailsecuritygeek.com for pointing out a typo. Cheers mate!

Scammers used my email as a spam address

On the 7th of November 2019, I received an email from AliExpress that told me that I created an account with them. Seeing as I didn’t do this, at first I thought it was a scam. My email address contains a dot between my first and last name and that was missing. So I did what I do with all spam, I ignored it.

A few weeks later, on November 25th, I received a notification that I had a shopping cart with items in it. I decided to go to the AliExpress website and do a password reset on “my” account. Surprisingly, I had not received spam and a few moments later, I was the proud owner of an AliExpress account.

The first thing I did was check out my shopping cart. I did not take a precise inventory at the time, I just deleted the few items that were in it. It did prompt me to look into my already purchased items. There was a range of strange choices from plastic apples for table decoration to knockoff Disney dolls. The one thing they all had in common was that they cost under 20 euros, thus skipping most customs controls. So the buyers evade sales tax, limit checks on the knockoff goods and get a higher chance the goods will get delivered.

When I looked at the account details, I saw a fake name with Bonny as the first name and a bogus shipping address in France. It was entered half a dozen times, so I concluded I was dealing with a master criminal that knew how to efficiently navigate the site.

I looked the address up on Google Maps and it turned out to be a corn field. I’ve always wondered how they deliver to such places. The delivery guy shows up in a truck with the stuff in the back and then what? Is there a shady guy with a nondescript white van ready to take the goods? I guess I’ll never know.

Back to the order history. All in all, there were 28 items bought on “my” account. When I saw that, I blamed AliExpress for not verifying the account before accepting orders. I received a welcome mail, but I never had to verify that my account is controlled by me. So there are probably countless unverified accounts that are used by scammers to buy counterfeit goods. That means that AliExpress is profiting from, what are in my opinion, fraudsters.

Until I checked the orders more closely. Apparently 20 out of the 28 orders haven’t been paid yet. That means that over 70% of the orders haven’t been paid 18 days after they were shipped. Somehow, I doubt that they will ever be paid, even if I did not take back the account. Which means that both AliExpress and the third party sellers are missing out on revenue.

All this scammer needs to do is create another fake account and buy as much goods as he can before the account is suspended. They can keep doing this as long as accounts are not verified as there is a treasure trove of emails out there for anybody who knows where to look. And it’s not exactly hard to find even if you don’t know where to look.

So I don’t know why AliExpress is not verifying accounts. It’s costing them money. It’s costing their subcontractors money. It’s costing European countries taxes. They are basically enabling scammers. The only thing they’d need to do to stop these thieves, is verify an account before that account can be used to buy goods.

At no point was my email compromised. They just used my email address to sign up. Thanks to a combination of a password manager (shameless plug for 1Password) and a strong second factor (shameless plug for YubiKey security keys), scammers will be hard pressed to get into my most valuable accounts. For full transparancy, I’m not sponsored by either vendor, I bought these products myself. I’m a big fan of them.

And as a last item, just to be thorough: I did not report this to the police. I do not feel that the information I have to share will make a compelling case against anybody. So instead of adding more white noise to the pile of noise the police already has to deal with, I’m going to ignore this.

What I do want to shine a light on, is that we cannot let scammers just use our emails for their fake accounts. So if I receive an email that I created an account somewhere, especially online shops, will get a closer look to see if it’s an actual welcome mail or a scam in itself.

A Simple SpecFlow 3 setup in Rider

Recently, I’ve gotten some mails on how to use SpecFlow 3 with Jetbrains Rider 2019.3.1. So I thought I’d update my how-to posts with this addition. If you are just here for the code, you can find it on my GitHub account.

I’m keeping this specifically simple and small. As soon as the project does what it needs to, I’ll stop, because my previous articles on this explain the rest (which still works) in more detail.

I’m just creating a new solution from Rider and adding a Unit Test Project with the xUnit framework. If you’re unaware, I’m a big fan of the xUnit framework. Then I added 3 Nuget packages:

  1. SpecFlow
  2. SpecFlow.Tools.MsBuild.Generation
  3. SpecFlow.xUnit

At the time of writing this article, all these packages have version 3.1.74.

When I want to install the SpecFlow.xUnit package, Nuget complains that xUnit is not the latest version and I have to update from 2.4.0 to 2.4.1. I took the opportunity to use Riders awesome Upgrade All Packages In Solution. Quick and easy.

Now that everything is installed, I just insert a File and name it SomeTest.feature. I type away and create a simple SpecFlow file. I could now build and run the test, but out of habbit, I create a SomeTest.steps.cs file and put in the basic Binding and Scope attributes above the class.

Then I run the tests and check the output.

As you can see on the right part of the output pane, it still works as in my earlier article and I can just copy out the relevant parts for my test setup. After I flesh out the test a bit, I run it again and get a green check.

I did exclude the generated *.feature.cs as it contains an unknown element, but does compile. The error does show the annoying red line below the file hierarchy it influences. They are not needed in the project (or the git repo, I excluded them via the .gitignore) to make the tests run. They are being included during the build step anyway.

SpecFlow 3 works very nicely with DotNet Core 3.x and the Rider IDE. It’s a great time to be a (DotNet) coder!

TensorFlow mask definition

A little while ago, I needed to parse a TensorFlow result in a dotnet application. Most of the interpretation of the result was fairly easy as there is a ton of documentation about TensorFlow. Until I had to apply the mask that TensorFlow returns.

Warning: math and complex code ahead!

I needed to block out a part of an image, a face, to make people in photo’s unrecognisable. The easy way would be to block out the rectangle that is supplied via the detection_boxes. It would be nice if I could block out just the face instead of a huge square block. So enter detection_masks output.

When TensorFlow returns a result, it’s a dictionary of multi-dimensional arrays. One of the dictionary items, detection_masks, is a three dimensional array of floats that specify which part of the detection_boxes contains the actual face. Before I get into this, let me back up a moment and explain what this service returns.

For me, it was quite confusing because the different dictionary items correlate to one another. Let’s say that I have a picture that I process and TensorFlow returns 10 recognised objects. The first dictionary item that I need are the detection_scores. The first score is the object with the highest confidence, so TensorFlow is fairly certain that it identified this object. The value will indicate how high the score is, 1 is 100% certain and 0 is 0% certain. In my results, the first few scores were 0.8 or higher and then it suddenly dropped off to less than 0.3. For this example, let’s say that the first score is 0.92, this means that TensorFlow is 92% certain about what it found.

If I want to know what the result indicates, I need to check the dictionary item detection_classes for the corresponding item in the list. So the first item in the detection_classes will tell me what the first item in the detection_score identifies. The second item in the classes will tell me what the second item in the scores identifies. And so forth. In the detection_classes array will be numbers that correspond to what the model found. The numbers are specific to the model. So class 1 from this model may mean something completely different from class 1 from somebody else’s model. For this example, let’s say that the first class is 1, which in this model is a face.

Now that I know that TensorFlow is 92% certain the first result is a face, let’s find the box it is located in. The coordinates are located in an array of floats in the dictionary item detection_boxes. The boxes are a little strange as they are grouped together in blocks of four, meaning that positions 0, 1, 2 and 3 contain the coordinates of the box around the first score and class. Array position 4, 5, 6 and 7 contain the box coordinates for the second score and class. Let me clarify further with an example.

Let’s focus on the positions of the first result (positions 0 through 3). Each value is a percentage of the width or height of the image. The first two positions (0 and 1) are the minimum positions of y and x. Be careful as the normal positions are reversed! The first value is the minimum y position and the second value is the minimum x position. Then come the maximum y and x positions. Again, watch which value you use, I ended up drawing some weird boxes before I figured it out.

How do I get to the actual x and y coordinates of the image? I multiply the value with the width (for an x point) and the height (for a y point). For this example, let’s say that the first 4 values are 0.2, 0.3, 0.5, 0.8 and the image dimensions are 10px wide and 20px high. This would give us two points: (y: 4 = 0.2 x 20 | x: 3 = 0.3 x 10) and (y: 10 =0.5 x 20 | x: 8 = 0.8 x 10).

Whew, that was confusing and I still have the most funky scenario coming up. Let’s just pause for a moment, catch our breath and marvel at the wonder that the box around a face can be drawn. Did you get something to drink, a cool glass of water, maybe a 15 year old single malt? Good, then let’s continue through the example with the last step: blurring the face.

To find the face, I need the box indicated by TensorFlow. TensorFlow tells me where in this box the face is located. Again, the next part depends on how the model is trained. So your results may vary from mine.

What TensorFlow does, is divide the box up in mini rectangles. For example, if TensorFlow divides the box up in rectangles 16×16 and the box is 32px high and 64px wide then each box will be 2 by 4 pixels in dimensions. Each box will receive it’s own score how likely it is that the object, in this case a face, is in the box.

Let’s take the above picture as an example: it is the box that TensorFlow identified as my face from a larger picture. In this example, the mask has a granularity of 5×5. Each box will get its own score. For this example, I want to focus on 3 boxes. Box number 1 will receive a score very close to 0, something along the lines of 0.001132… This will tell me TensorFlow does not think my face is in this box. Box number 2 will probably receive a score around 50% (think 0.540887…), which tells me that my face may be in this part of the box. Box number 3 will receive a high score, probably over 90% (think 0.938492…). This means this box will surely contain my face.

The scores can be used to guess the outline of my face and black out my face. Depending on how rigorous I want to be, I can block out just my nose, mouth and eyes or go for my hair and chin too if I set the confidence level lower. But where do I find these numbers?

TensorFlow returns a three-dimensional array of floats called detection_masks. The first dimension of the array refers to the detection_boxes, the second dimension refers to the number of rows the box is divided in and the third dimension contains the actual confidence levels.

Let’s apply that to the example. I would have an array with the first dimension length equal to the number of found items in the total picture. The first item in the first dimension refers to the box with my face that has that 92% certainty. The two following dimensions tell me in how many parts the box is divided. With the picture above, this will be a 5 by 5 array. That means that the first array of the second dimension contains an array with the confidence values of the top row of boxes in the picture. This is also how you can calculate how big the boxes are in my 16×16 example earlier. The third dimension then contains all the numbers with the confidence levels (the values of the three boxes I talked about earlier).

Finally, I present you my code to blur the face when I cut out the detection_box from the bigger picture where I pass in the specific detection_mask for that box.

public Image BlurFace(Image image, float[][] mask)
{
    var destImage = new Bitmap(image.Width, image.Height, PixelFormat.Format24bppRgb);
    using (var graphics = Graphics.FromImage(destImage))
    {
        graphics.DrawImage(image, 0, 0, image.Width, image.Height);
        var yRatio = (float)image.Height / mask.Length;
        var xRatio = (float)image.Width / mask[0].Length;
        var maskBlockSize = new Size((int)Math.Ceiling(xRatio), (int)Math.Ceiling(yRatio));
        var maskBlocks = new List<Rectangle>();
        for (var y = 0; y < mask.Length; y++)
        {
            for (var x = 0; x < mask[0].Length; x++)
            {
                var shouldMask = mask[y][x] < MaskMaxConfidence;
                if (shouldMask)
                {
                    var maskBlock = new Rectangle(new Point((int)Math.Ceiling(x * xRatio), (int)Math.Ceiling(y * yRatio)), maskBlockSize);
                    maskBlocks.Add(maskBlock);
                }
            }
        }
        var brush = new SolidBrush(Color.Black);
        graphics.FillRectangles(brush, maskBlocks.ToArray());
    }
    return destImage;
}

Phew. That was a difficult last part to get my head around. I hope this information helps somebody to better understand how TensorFlow returns results, because I didn’t really find any help on what was in the detection_masks arrays. This is very specific and I hope I shed some light in the darkness that is machine learning.

JetBrains giveaway

During a very fun NDC Oslo, I got to talk to the awesome guys at JetBrains. Because I’m writing nice things about their newest IDE Rider, I get to give away a code for a 3 month JetBrains All Product Pack subscription to one of my readers!

Because JetBrains makes good tools, I think I’m going to receive a lot of tweets about this. To make it easier to make a decision, I hope to receive funny, inspirational or heartwarming messages on why these tools will improve your quality of life and code.

Writing Azure Functions with Rider

With the release of Rider 2019.1, there’s now support for Azure Functions in the form of a plugin. Let’s find out how easy it is to run a Function locally.

The first step is to install the plugin that will enable support for Azure Functions. Go to the Settings (ctrl+alt+s) > Plugins tab and search for “Azure Toolkit for Rider” and install it. I think it’s a very popular plugin, or Jetbrains seriously wants to promote it, because it was the first plugin even when I hadn’t installed it yet. Could be alphabetical too, not sure.

The settings page for Plugins

After a quick Rider restart, there is a new option in the Settings > Tools tab: “Azure”. Select the Functions subsection, and install the latest version of the Azure Functions Core Tools. There is a link that will install the latest version automatically. Rider then downloads and installs or updates the Azure Functions Core Tools via NPM.

The settings page for Azure Functions

After another Rider restart (because restarting is what IT people do best), there is a new project template and several new class templates.


New Azure class templates

For this example, I create a Timer Trigger (because that will force me to set up the Azure Storage Emulator). This adds a class with the correct attributes and method signature for a timed Function. I looked up that the default CRON expression (0 */5 * * * *) runs every 5 minutes. Code Hollow has a convenient cheat sheet, be sure to check that out if you are creating a custom schedule.

Now that the code seems OK, I want to run this little “hello world” program. The build stops me dead in my tracks, however.

The fix is not obvious, but fortunately, it’s easy. In the projects .csproj file, the target framework is set to netcoreapp2.1 by default. This should be changed to netstandard2.0 (or whatever the latest and greatest version my dear reader is using). I know that at the time of writing dotnet core 3 just came out, but the template from Rider defaults to netcoreapp2.1, so I’m using the closest netstandard. There is either a little bug in the Rider template or there is something wrong with my setup.

<PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
</PropertyGroup>

Aah, a building Functions project. Now let’s run it…

The build warning

The next problem surfaces quite quickly. One of the debug outputs already warned me for it, but it becomes painfully clear that there is no AzureWebJobsStorage setup for local development.

The run fail

The AzureWebJobStorage needs a local instance of the Azure Storage Emulator or an actual web storage endpoint. What was not immediately clear for me, is that the Azure Storage Emulator needs a SQL LocalDB instance. To get the LocalDB installer, download the SQL Express database. Select Download Media and in the next screen, select the LocalDB option.

Download Media
Select LocalDB

The download location will open automatically. There will be an .msi installer to easily install the LocalDB database. I have accidentally installed SQL Server 2017 as well, which gave me problems while initialising the LocalDB. To circumvent those problems, install the latest Cumulative Update for SQL Server 2017. The problem was that the Azure Storage Explorer didn’t work properly because it could not connect properly to the LocalDB. That prevented the Azure Storage explorer form creating a database with all the tables it needs. By the way, some articles told me to manually create the AzureStorageEmulatorDb## database. That won’t solve the problem, it will just mask the problem as the database won’t have the necessary tables.

Once that’s all done, verify that the LocalDB installed correctly by running this command SQLLocalDB.exe i. It should print out MSSQLLocalDB. Don’t forget to start the database with the command SqlLocalDB.exe s MSSQLLocalDB. Now, the Storage Emulator should work, right? Wrong!

There is no database with the specific name AzureStorageEmulatorDb59. I found this out after I tried starting the Storage Emulator and seeing that the initialisation of the emulator crashed. So, I tried running the command:

> AzureStorageEmulator.exe init
Windows Azure Storage Emulator 5.9.0.0 command line tool
Found SQL Instance (localdb)\MSSQLLocalDB.
Creating database AzureStorageEmulatorDb59 on SQL instance '(localdb)\MSSQLLocalDB'.
Cannot create database 'AzureStorageEmulatorDb59' : The database 'AzureStorageEmulatorDb59' does not exist. Supply a valid database name. To see available databases, use sys.databases..
One or more initialization actions have failed. Resolve these errors before attempting to run the storage emulator again.
Error: Cannot create database 'AzureStorageEmulatorDb59' : The database 'AzureStorageEmulatorDb59' does not exist. Supply a valid database name. To see available databases, use sys.databases..

To remedy this, open SQL Server Management Studio, connect to the LocalDB instance and create a database with the name AzureStorageEmulatorDb59. I tried doing this in Rider with the built in DataGrip tools, but I got an error. I’ve reported this, so it will get fixed in the future. See earlier remark about the Cumulative Update! That should prevent this error from happening. I’m keeping it in as I think others will run into the same problem.

Now that the database is set up, I can finally start the storage emulator. All I have to do is fill in the AzureWebJobsStorage with UseDevelopmentStorage=true. All set, lets run the function. Unfortunately, the output still complains that the AzureWebJobsStorage still isn’t filled in. After some checking in the config and the place where the function runs, it appears that the local.settings.json file is not being copied. So, I change the Copy to output directory setting in the file properties to Copy Always. Now the Azure function starts up and a minute later, the breakpoint in my function is hit.

To make my life easier, I should add some “Before launch” external tools arguments in the Run/Debug Configuration so the LocalDB and Azure Storage Emulator start before each run. I think I’ll set that up later.

Now everything is set up correctly. I can run and debug Azure Functions with Rider. It’s not as transparent process as I’d hoped it would be, but it was a good learning experience for me.

Using SpecFlow 3.0 with Rider

A while ago, I wrote about using SpecFlow with JetBrains IDE Rider. Recently, SpecFlow updated their version to 3.0 and it brings some different behaviour with it. After using it for a while, I really like the new flow.

The first change is that I need to install three packages, instead of two. SpecFlow and a unit testing framework (MSTest, NUnit or xUnit) still need to be installed. Additionally, SpecFlow.Tools.MsBuild.Generation needs to be installed as well.

The biggest benefit of the SpecFlow.Tools.MsBuild.Generation nuget is that I don’t need to set up a file writer anymore. I do need to update the .csproj file to include some configuration. The whole process is nicely documented on the SpecFlow site, but I’ll give the basics here.

First, right click on the test project and choose Edit > Edit MyProject.csproj option. Then just paste in the next bit of XML:

<Target Name="AfterUpdateFeatureFilesInProject">
    <!-- include any generated SpecFlow files in the compilation of the project if not included yet -->
    <ItemGroup>
        <Compile Include="**\*.feature.cs" Exclude="@(Compile)" />
    </ItemGroup>
</Target>

The old SpecFlow version can add configuration tags in the .csproj file: <Generator>SpecFlowSingleFileGenerator</Generator>. They can easily be replaced by doing a search and replace (ctrl+h in rider) and using this regex to find all of them: \n[ ]+SpecFlowSingleFileGenerator.

Save the .csproj file and just build the project. The .feature.cs files will be generated next to the .feature files. You can include them in the project, but the build server will update the generated files when it builds the project anyway. So you don’t need to include the .feature.cs files in the project anymore if you don’t want to.

Happy coding and keep those tests green!

Update: I got some feedback that it isn’t intuitive to find the Given/When/Then declarations for the .feature.cs file. All I need to do is run the tests. The method definitions appear in the test output window. If I add a new line, I just rerun the test. The test will be marked incomplete with the new method signature in the test output window. All that’s left to do is to copy and paste the signatures into the right file and flesh out the new method. Oh, and don’t forget to add Binding and Scope attribute on the class. I’ve forgotten that more times than I dare to admit.