Splitting IMediator interface

For a past project, I used the awesome MediatR package to send messages throughout the system. Because I used the pipeline for some compute heavy checks, it was not wise to send a request from a request handler. That is why I split the functionality for sending requests and notifications or events.

When a message entered the system, mostly via a REST endpoint, it got dispatched through MediatR to the corresponding handler. The message travels through the pipeline where some logging was done, some validity checks were performed and sometimes, there were even some security checks (e.g. “can this user access this data”). In this application, the pipeline is not the most lightweight part.

For some actions, I wanted to reuse logic from other handlers. Each handler has a single responsibility and I’d be reusing code, good decision in my opinion. Unfortunately, this triggered the pipeline each time which was not necessary at that point. I quickly saw that this significantly slowed the application. That is why the team and I decided to never send requests from within request handlers.

All said and done, we refactored this pattern of sending requests within handlers to simple service calls. That way, the reused request handler was nothing more than a facade in front of the service being called.

Notifications were also being used throughout the system to notify other parts when certain events happened. This would mean that the IMediator interface was passed into a significant number of handlers so they could publish these notifications (or events if you like that term better).

This also meant, that the team has easy access to the send request functionality. Now being the diligent programmers that we all are, I (and other team members, especially the newer ones) never succumbed to the temptation of cutting corners. So we always refactored the second handler into a service and called the functionality via the service. Or maybe not always…

Because that send request is just so easy to (mis)use, it still happened more than I would’ve liked. We all knew that not refactoring would just come to bite us later. From time to time, for whatever reason (pressure, tired, deadlines, new team member,…), it happened again.

That’s when I created a specific interface for sending events through the system. I created an implementation that used the MediatR library. This allowed us to use the MediatR publishing mechanism, without exposing the send request functionality.

public interface IPublisher
{
  Task Publish<TNotification>(TNotification notification, CancellationToken cancellationToken = default)
    where TNotification : INotification;
}

public class MediatrPublisher : IPublisher
{
  private IMediator _mediator;
  public MediatrPublisher(IMediator mediator) => _mediator = mediator;
  public Task Publish<TNotification>(TNotification notification, CancellationToken token = default) => _mediator.Publish(notification, token);
}

Because I really like what Jimmy Bogard (@jbogard, the creator of the MediatR package) does, I’ve recently submitted a PR to get this into the MediatR package. We all know it’s much better to rely on somebody else’s interface than to create our own (who noticed the sarcasm dripping from this sentence?).

In all seriousness, I think it will benefit the MediatR package to separate these concerns. That is why I’ve created two new interfaces: IPublisher and ISender. These contain the Send and Publish methods that resided in the IMediator interface. Because not everybody wants to switch to these specialised interfaces, I left the IMediator interface in place and have that inherit from the new ones.

public interface IPublisher
{
  Task Publish(object notification, CancellationToken cancellationToken = default);
  Task Publish<TNotification>(TNotification notification, CancellationToken cancellationToken = default)
    where TNotification : INotification;
}

public interface ISender
{
  Task<TResponse> Send<TResponse>(IRequest<TResponse> request, CancellationToken cancellationToken = default);
  Task<object?> Send(object request, CancellationToken cancellationToken = default);
}

public interface IMediator : ISender, IPublisher { }

I’m a big fan of Jimmy’s work and I hope that with this change, I’ve helped improve the quality of life for a number of programmers, including mine. I’m not sure when this will be available in the MediatR package, but I hope soon.

Consuming a C++ DLL in dotnet core

For a client, I need to open a drawing in a specialised program. When the user closes the program it should upload the drawing to a server. What looks like a straightforward start of a process, turned into a mindbogglingly number of failures. Journey with me through all the steps I took to get this seemingly simple task to work.

After a quick Process.Start('program', 'path/to/drawing') didn’t work, I knew something was wrong. So I contacted the developers of the software and asked for clarification. They sent over very helpful documentation and I was on my merry way again.

Apparently, they have a .dll that is included with their program and that is how an external application can communicate with it. Phew, not so hard after all. So I modelled the external calls to methods and decorated them with the [DllImport] attribute.

private delegate int Callback(int code, string message);
[DllImport("path/to/dll", CharSet = CharSet.Unicode, SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
private external static int RegisterCallback(Callback method);

[DllImport(/*see previous*/)]
private external static int StartApplication();

[DllImport(/*see previous*/)]
private external static int DoWork(string command);

In this flow, which is apparently more common in C++ than dotnet, I first register a callback method, start the application and then send work to it. The registered callback method handles status updates and other information that gets sent back. This way, a single marshalling needs to be done and multiple DoWork methods can be called, even different external functions, and they’ll all report back to the registered callback method.

The code parameter in the callback method determines what will be in the message and how to interpret it. For example, the code 0 can refer to application initialisation and the message then is “SUCCESS” or “FAILURE: with a specific reason”. More complex information can be sent back as well. For example, code 2 could be that the application has saved the drawing, the message then contains XML with the location of the drawing and some meta information about the changes that were made.

Although this is quite an interesting approach, as most examples of interop use a specific callback in each function. This approach however, does pose a few interesting challenges. I found that the callback was never invoked. To get more error information, I had to enable SetLastError = true. This then allows me to call Marshal.GetLastWin32Error(). That function returns a number that I can then look up in the official Microsoft docs.

Unfortunately, I forgot the error so I can’t reference it here. It did lead me to an article to set the CallingConvention to Cdecl. This is to indicate that I am responsible to clean the stack and not the code that I’m calling. This has to do with defaults in both runtimes, C++ uses __cdecl convention while the default for C# is __stdcall. The conventions need to align to allow the processes to talk to each other.

Huzzah, done… No wait, too soon, still no callback. The application that I am calling is starting, but it also gets cleaned up too fast. I’m not sure what C++ techniques are used, but either Windows or the GC are cleaning up the application before it could properly start.

What I did notice is that the StartApplication() method returns almost immediately and I have to build a way to wait for the callback that says that the full initialisation of the application is done. I assume that the StartApplication method sets a process in motion and reports that the process kicked off correctly. Dotnet on the other hand figures that the invoked method was done and doesn’t need all those resources anymore, including the application that is halfway through its startup process.

If anybody is interested in the wait mechanic that I tried here: I used a TaskCompletionSource<bool> to return whether the initialisation would have completed succesfully based on the message the callback would receive. This way, I could use await to wait for the external program to finish.

Here’s where the developers of that program gave me a helping hand: they told me dotnet needs to keep a reference to the method that is being called and the way to do that is to use functions from the Kernel32 library to load the functions into memory and only release them when they are done.

[DllImport("kernel32.dll", SetLastError = true)]
private static extern IntPtr LoadLibrary(string pathToLibrary);

[DllImport("kernel32.dll", SetLastError = true)]
private static extern IntPtr GetProcAddress(IntPtr libraryReference, string procedureName);

[DllImport("kernel32.dll")]
private static extern bool FreeLibrary(IntPtr libraryReference);

So instead of using a DllImport, I have to do this by hand.

The LoadLibrary function is used to load the library into the running application. The IntPtr that is returned is a pointer to a memory address where the library is loaded. The GetProcAddress function takes the pointer returned by LoadLibrary and the name of the exposed method. When the library has fulfilled it’s purpose, so after the DoWork function is done and the callback method has received the done signal, I need to clean up the references to the library by feeding the pointer to the library to the FreeLibrary function.

It’s comparable to how DllImport works behind the scenes. Except that DllImport is applied to a method that can be called. The GetProcAddress only returns another pointer to where the function I want to call is loaded into memory. To convert the pointer to an callable function, I need to marshal it. In full dotnet framework, there is a static function Marshal.GetDelegateForFunctionPointer which takes the IntPtr and the Type of a delegate. It then returns an object that can be cast to the delegate. If you are using dotnet core, there is a generic function Marshal.GetDelegateForFunctionPointer<T>.

internal class CustomApplicationIntegration : IDisposable
{
  // kernel32 imports here
  private readonly IntPtr _dllPointer;
  public CustomApplicationIntegration(string pathToLibrary)
  {
    _dllPointer = LoadLibrary(pathToLibrary);
  }

  public void Dispose()
  {
    if (_dllPointer != IntPtr.Zero)
      FreeLibrary(_dllPointer);
  }

  private delegate void Callback(int code, string message);
  private Callback _keepAlive;
  private bool _applicationDone = false;
  private delegate int RegisterCallback(Callback method);
  private delegate int StartApplicationDelegate();
  public void Interact()
  {
    _keepAlive = CallbackMethod;
    var registerPointer = GetProcAddress(_dllPointer, "RegisterCallback");
    if (registerPointer == IntPtr.Zero)
      throw new Exception("Something went wrong!");
    // dotnet 4.X aka full framework
    var registerCallback = (RegisterCallback) Marshal.GetDelegateForFunctionPointer(registerPointer, typeof(RegisterCallback));
    var registerResult = registerCallback(_keepAlive);
    // check that registerResult returns ok

    var startPointer = GetProcAddresss(_dllPointer, "StartApplication");
    // dotnet core
    var startApplication = Marshal.GetDelegateForFunctionPointer<StartApplicationDelegate>(startPointer);

    var startResult = startApplication();
    // check that startResult returns ok
    while (!_applicationDone)
      Thread.Sleep(100);
  }

  private static void CallbackMethod(int code, string message)
  {
    // handle callback
    _applicationDone = code == StopCode;
  }
}

If something goes wrong in either the LoadLibrary or GetProcAddress, they return the value IntPtr.Zero. That’s how I can check for success, everything that is not the zero value is a valid pointer. Because the SetLastError is set on the DllImports of these functions, I can get the id of the error with the Marshal.GetLastWin32Error function, just like I did earlier.

At the end of the Interact method, there is a loop to keep waiting on the external application to send the stop code. Only when that code is received, can I continue with my application. There are functions in the library to do specific tasks and to stop the external application, but I’m omitting them here for brevity.

In the actual app, there are several loops waiting for the external application to finish some work or signal some status changes. For example, I have to wait for the application to start, before I can assign any work to it.

Now lets run this thing and see… still… no… callbacks… What I can see, is that the application is still exiting, but now it takes longer. The startup logs indicate that the library is doing it’s job correctly, but that the process that is started by the library is being terminated. Windows, stop killing my processes!

The Task.Run is to blame here! I run the Interact function in a Task and it runs a while longer, but it doesn’t protect the process the library is starting. When I changed this to a Thread, it solved the matter. To be honest, I have no idea why the Task terminates early, while the Thread runs as intended. If anybody knows why this is, do get in touch.

// bad way
await Thread.Run(() =>
{
  var integration = new CustomApplicationIntegration();
  integration.Interact();
}

// the good way
var workThread = new Thread(() =>
{
  var integration = new CustomApplicationIntegration();
  integration.Interact();
});
workThread.Start();

I hope this runs, I hope this runs and… I wait forever on the callback function. Don’t give up hope, this is the last problem to solve, but what a problem it is. The issue is that I created a deadlock: the external application invokes the callback and waits for it to return (eventually it would crash after a looong time) and my code waits for the callback but is too busy waiting to notice the callback.

Because it’s a console application, there is no main event loop. The console expects there to be only one way through the application. It can wait on async methods, but it does not expect any application events. What this means is that it does not see the event of the callback, because it does not originate from within the application.

Simple callbacks that happen within the execution of the called C++ code expect the callback method to be invoked. Because I registered the callback and then went on to other functions that do not have that callback explicitly referenced, the console is not ready for these invocations. Instead, the Windows event loop is used. Since the console app does not have a main event loop, they simple do not get processed. This is the same reason why Timers and Sockets do not work as expected in console applications.

The most simple solution to this problem is to convert the console app to a WinForms app and call Application.DoEvents() in the while loop in the CustomApplicationIntegration class.

while (!_applicationDone)
{
  Thread.Sleep(100);
  Application.DoEvents();
}

To me, it feels a bit hacky. That is because in a normal WinForms app, the Application.DoEvents() method gets handled by the framework. It has a number of guards to handle concurrency when multiple events fire in quick succession. The official Microsoft documentation seems to discourage manually waiting on Application.DoEvents. Since I’m only expecting events from one external source, there should be no problems. Should…

Wow, that was a lot to take in. From DllImports, over marshalling functions and waiting for an asynchronous response to finally transforming the console app to a WinForms app so Windows events could be handled. Somebody has earned their favourite beverage, so go on and treat yourself to a nice beer, wine or two fingers of whiskey. I know I’m going to enjoy all myself now.

Specflow GenerateFeatureFileCodeBehindTask error

In the past few months I have been contacted by a number of readers who notified me that the code for my A simple SpecFlow 3 setup in Rider blog post was broken. I took me a while, but I’ve finally taken the time to fix it.

Some time ago I already looked at it and I saw that with the upgrade to dotnet core 3 there was a breaking change with SpecFlow .feature.cs file generation. Due to a change in the MSBuild pipeline, the files were not generating correctly. The following error was shown during the build of the solution.

The “GenerateFeatureFileCodeBehindTask” task failed unexpectedly.

MSBuild

Fortunately, upgrading to the latest version of all the packages fixed all the errors. Thank you SpecFlow team!

The SpecFlowSetup project on GitHub has been updated and should work again.

A Simple SpecFlow 3 setup in Rider

Recently, I’ve gotten some mails on how to use SpecFlow 3 with Jetbrains Rider 2019.3.1. So I thought I’d update my how-to posts with this addition. If you are just here for the code, you can find it on my GitHub account.

I’m keeping this specifically simple and small. As soon as the project does what it needs to, I’ll stop, because my previous articles on this explain the rest (which still works) in more detail.

I’m just creating a new solution from Rider and adding a Unit Test Project with the xUnit framework. If you’re unaware, I’m a big fan of the xUnit framework. Then I added 3 Nuget packages:

  1. SpecFlow
  2. SpecFlow.Tools.MsBuild.Generation
  3. SpecFlow.xUnit

At the time of writing this article, all these packages have version 3.1.74.

When I want to install the SpecFlow.xUnit package, Nuget complains that xUnit is not the latest version and I have to update from 2.4.0 to 2.4.1. I took the opportunity to use Riders awesome Upgrade All Packages In Solution. Quick and easy.

Now that everything is installed, I just insert a File and name it SomeTest.feature. I type away and create a simple SpecFlow file. I could now build and run the test, but out of habbit, I create a SomeTest.steps.cs file and put in the basic Binding and Scope attributes above the class.

Then I run the tests and check the output.

As you can see on the right part of the output pane, it still works as in my earlier article and I can just copy out the relevant parts for my test setup. After I flesh out the test a bit, I run it again and get a green check.

I did exclude the generated *.feature.cs as it contains an unknown element, but does compile. The error does show the annoying red line below the file hierarchy it influences. They are not needed in the project (or the git repo, I excluded them via the .gitignore) to make the tests run. They are being included during the build step anyway.

SpecFlow 3 works very nicely with DotNet Core 3.x and the Rider IDE. It’s a great time to be a (DotNet) coder!

Using SpecFlow 3.0 with Rider

A while ago, I wrote about using SpecFlow with JetBrains IDE Rider. Recently, SpecFlow updated their version to 3.0 and it brings some different behaviour with it. After using it for a while, I really like the new flow.

The first change is that I need to install three packages, instead of two. SpecFlow and a unit testing framework (MSTest, NUnit or xUnit) still need to be installed. Additionally, SpecFlow.Tools.MsBuild.Generation needs to be installed as well.

The biggest benefit of the SpecFlow.Tools.MsBuild.Generation nuget is that I don’t need to set up a file writer anymore. I do need to update the .csproj file to include some configuration. The whole process is nicely documented on the SpecFlow site, but I’ll give the basics here.

First, right click on the test project and choose Edit > Edit MyProject.csproj option. Then just paste in the next bit of XML:

<Target Name="AfterUpdateFeatureFilesInProject">
    <!-- include any generated SpecFlow files in the compilation of the project if not included yet -->
    <ItemGroup>
        <Compile Include="**\*.feature.cs" Exclude="@(Compile)" />
    </ItemGroup>
</Target>

The old SpecFlow version can add configuration tags in the .csproj file: <Generator>SpecFlowSingleFileGenerator</Generator>. They can easily be replaced by doing a search and replace (ctrl+h in rider) and using this regex to find all of them: \n[ ]+SpecFlowSingleFileGenerator.

Save the .csproj file and just build the project. The .feature.cs files will be generated next to the .feature files. You can include them in the project, but the build server will update the generated files when it builds the project anyway. So you don’t need to include the .feature.cs files in the project anymore if you don’t want to.

Happy coding and keep those tests green!

Update: I got some feedback that it isn’t intuitive to find the Given/When/Then declarations for the .feature.cs file. All I need to do is run the tests. The method definitions appear in the test output window. If I add a new line, I just rerun the test. The test will be marked incomplete with the new method signature in the test output window. All that’s left to do is to copy and paste the signatures into the right file and flesh out the new method. Oh, and don’t forget to add Binding and Scope attribute on the class. I’ve forgotten that more times than I dare to admit.