Moving to trunk based development

The project I’m working on is a smaller one. The current release pipeline has a number of specially named branches for each environment. This looks like it was based on GitFlow, but with some oddly specific named branches. So I started thinking about a better way to organise the releases.

Because the project is small, my first instinct was to consolidate on one branch. This looks a lot like trunk based development and that’s because it is. I now only have a master branch from which I deploy to every test environment. With only me as developer and a few business people who test on it, it felt overkill to keep the development and quality assurance environment.

A problem arises when I talk with the business people about release management. They want to test the version in the test environment and when everything looks ok, I can deploy outside of business hours to production. At the moment, I do not have a way to enable new features in the test environment and disable them for the production environment. So I would not be able to continue development without also pushing those changes to production. The same issue arises with hotfixes, I’d need a way to push changes to production, without pushing every experimental feature on the master branch. The best approach would be to use feature toggles, but this is not a change I can do overnight.

Queue the only other long lived branch besides the master branch: the production branch. I use feature branches taken from the master branch. When the business gives the green light that everything looks good, I can create a release branch from the master branch. When I have the release branch in place, I can wait until outside working hours before merging the release to production. Basically I create a snapshot from a well tested environment and push it to production. Should bugs in production appear, I can create a hotfix branch from the production branch and merge that back to the test environment for verification.

With this in structure in place, I have only 2 branches: production for the production environment and master for everything else. I can work trunk based in the testing environment and have a dedicated branch for production releases and issues. There is a temporary (and optional) release branch so I have a snapshot of a production release, but that is only for practical reasons.

In the next part, I describe the changes and important remarks I noticed while working towards the end result. This will start from the old implementation with different branches and releases and work towards the solution described above.

So now I just have to change the release pipeline. The first thing I noticed, was that there are a lot of branches that can kick off the single pipeline that is in control of the continuous integration (CI) build, the release build and the deploys to the various environments.

trigger:
  - master
  - develop
  - feature/*
  - bugfix/*
  - hotfix/*
  - release/*

Before a pull request (PR) can be merged into the branches master or develop (both kick off a deploy to a specific environment), it needs to pass a build pipeline. I think the person who set this up, thought that there should be triggers for the CI pipeline to kick in. In Azure DevOps, I can specify a policy on a branch that needs to run before the PR can complete. This kicks off a separate build process for the policy.

What this in practice does, is when I push to a feature branch (or any other described above), it kicks off a build (described in the release pipeline). Then I start a PR, which also kicks off a build (for the branch policy). I see one build too many here. This does not only cost time as it basically doubles the time for a PR build, it also costs money as the build process is a paid service (after the first 1800 free minutes). It becomes even more wasteful if I need to add changes to the PR. Remember that each push to a branch starts a build and each PR change kicks off a policy build.

Seeing as I do not want multiple branches for each environment (right now, the develop branch goes to the develop environment and the master branch goes to the test and production environments), I can simplify the trigger.

trigger:
  - master
  - production

With the simplified triggers, I’m now left with a whole section that is only used for a CI build. As this is not needed for a deployment, I moved this to it’s own pipeline. Azure DevOps supports multiple pipelines. I simply create one in the DevOps portal (or maybe through the Azure CLI, I’m not familiar with that) and point it at a yaml file.

trigger:
  - none

variables:
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  solution: src/SolutionFile.sln
  buildPlatform: Any CPU
  buildConfiguration: Release

stages:
  - stage: CI_Build
    displayName: CI build on PR
    jobs:
      - job:
        displayName: CI Build
        pool:
          vmImage: windows-latest
        steps:
          - task: [email protected]
            displayName: Install NuGet Tooling
          - task: [email protected]
            displayName: Restore NuGet Packages
            inputs:
              restoreSolution: $(solution)
              feedsToUse: select
              vstsFeed: feed-identifier
          - task: [email protected]
            displayName: Build Solution
            inputs:
              solution: $(solution)
              platform: $(buildPlatform)
              configuration: $(buildConfiguration)
          - task: [email protected]
            displayName: Run Test Suite
            inputs:
              platform: $(buildPlatform)
              configuration: $(buildConfiguration)

The content of the ci-pipeline.yml contains a simplified build process. I have added two variables to opt out of providing Microsoft with telemetry data about my builds. The DOTNET_CLI_TELEMETRY_OPTOUT and DOTNET_SKIP_FIRST_TIME_EXPERIENCE work together to not send information to Microsoft and speed the build up a little bit. More information can be found in the Microsoft documentation.

Now that I placed my CI build in a separate pipeline, my original pipeline is already looking cleaner. I’ll save you all the refactoring steps, cursing and head scratching (of which there was a lot) and present the finished product. I redacted some parts as they are client specific details, it should not be difficult to figure out what goes where. Let’s start with the general release-pipeline.yml file.

trigger:
  - master
  - production

variables:
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  solution: src/SolutionFile.sln
  prodBranch: refs/heads/production

stages:
  - template: pipeline/release-build.yml

  - template: pipeline/deploy.yml
    parameters:
      environment: TEST
      condition: and(succeeded(), not(eq(variables.build.sourceBranch, variables.prodBranch)))

  - template: pipeline/deploy.yml
      parameters:
        environment: PROD
        condition: and(succeeded(), eq(variables.build.sourceBranch, variables.prodBranch))

The trigger and variables sections are pretty self explanatory, so lets look at the different stages. I won’t display the release-build.yml, but safe to say it looks a lot like the CI pipeline yaml from earlier. I’ve added a few different steps as I want to build the website and processing services separately. This is necessary to deploy them separately later.

There are two deploys, one to the test environment and one for production. The test deployment runs after a successful build when it’s not the production branch and the production deploy only runs after a successful build when it is the production branch. This is the only difference between the deploys.

The deploy.yml in the pipeline folder, contains a stage that calls two job templates: one to deploy the website and one for the processing services. The rest of this template should be pretty straightforward.

parameters:
  environment: ""
  condition: ""

stages:
 - stage: ${{ parameters.environment }}_Deploy
    displayName: Deploy To ${{ parameters.environment }}
    dependsOn: Release_Build
    condition: ${{ parameters.condition }}
    pool:
      vmImage: windows-latest
    jobs:
      - template: deploy-website.yml
        parameters:
          environment: ${{ parameters.environment }}
      - template: deploy-processing-services.yml
        parameters:
          environment: ${{ parameters.environment }}

The website template is just a deployment job that first cleans the target folder, then extracts the new build and finally transforms the web.config.

parameters:
  environment: ""
  serviceName: "Client.Website"
  artifactPrefix: "Client.Website.Build"

jobs:
  - deployment: Public_Website
    displayName: Install public website
    environment:
      name: ${{ parameters.environment }}
      resourceType: VirtualMachine
      tags: website
    strategy:
      runOnce:
        deploy:
          steps:
            - task: [email protected]
              displayName: "Delete ${{ parameters.serviceName }}"
              inputs:
                SourceFolder: "C:\\WebSites\\${{ parameters.serviceName }}"
                Contents: "**/*"
            - task: [email protected]
              displayName: "Extract ${{ parameters.serviceName }}"
              inputs:
                archiveFilePatterns: "$(Pipeline.Workspace)/Mooose/${{ parameters.artifactPrefix }}.$(Build.BuildId).zip"
                cleanDestinationFolder: true
                destinationFolder: "C:\\WebSites\\${{ parameters.serviceName }}"
            - task: [email protected]
              displayName: "Transforming $(Environment.Name) web.config"
              inputs:
                folderPath: 'C:\\WebSites\\${{ parameters.serviceName }}'
                xmlTransformationRules: "-transform web.$(Environment.Name).config -xml web.config"

The deploy-processing-services.yml is quite easy. It installs two agents that process service bus messages. One is for general processing, the other is for pdf generation. This gets its own service as this needs to go fast and can’t wait for other general messages. There is an install-service.yml script that contains the powershell to stop, install and start the service. So that is easily reusable as well.

parameters:
  environment: ""

jobs:
  - deployment: Processing_Services
    displayName: Install processing agents
    environment:
      name: ${{ parameters.environment }}
      resourceType: VirtualMachine
      tags: processing
    strategy:
      runOnce:
        deploy:
          steps:
            - template: install-service.yml
              parameters:
                serviceName: Backend.Agent
                artifactPrefix: client.backend.agent
            - template: install-service.yml
              parameters:
                serviceName: Pdf.Agent
                artifactPrefix: client.pdf.agent

I don’t do a lot of special deployment things, but there is one area where I want to focus on: the environment.tags. I set the deployment to a specific environment name (QA or PROD, from the release-pipeline.yml) on a virtual machine (because they use machines in a local server park) and I use tags to specify which servers I deploy to.

The use of tags allow me to tag one server in the Azure DevOps Environments with the “website” and “processing” tags and everything gets deployed on one server. In the production environment on the other hand, those are several different servers (multiple for the website and one for the services). Yet I can use the same pipeline to deploy to different combinations. Should I ever need more processing servers, I can just add the servers to the Azure Environment, apply the correct tags and the deployment process would know what to do.

With a simplified pipeline, I can spend less time worrying about getting code into the right environment and focus more on getting the features right.

Writing Azure Functions with Rider

With the release of Rider 2019.1, there’s now support for Azure Functions in the form of a plugin. Let’s find out how easy it is to run a Function locally.

The first step is to install the plugin that will enable support for Azure Functions. Go to the Settings (ctrl+alt+s) > Plugins tab and search for “Azure Toolkit for Rider” and install it. I think it’s a very popular plugin, or Jetbrains seriously wants to promote it, because it was the first plugin even when I hadn’t installed it yet. Could be alphabetical too, not sure.

The settings page for Plugins

After a quick Rider restart, there is a new option in the Settings > Tools tab: “Azure”. Select the Functions subsection, and install the latest version of the Azure Functions Core Tools. There is a link that will install the latest version automatically. Rider then downloads and installs or updates the Azure Functions Core Tools via NPM.

The settings page for Azure Functions

After another Rider restart (because restarting is what IT people do best), there is a new project template and several new class templates.


New Azure class templates

For this example, I create a Timer Trigger (because that will force me to set up the Azure Storage Emulator). This adds a class with the correct attributes and method signature for a timed Function. I looked up that the default CRON expression (0 */5 * * * *) runs every 5 minutes. Code Hollow has a convenient cheat sheet, be sure to check that out if you are creating a custom schedule.

Now that the code seems OK, I want to run this little “hello world” program. The build stops me dead in my tracks, however.

The fix is not obvious, but fortunately, it’s easy. In the projects .csproj file, the target framework is set to netcoreapp2.1 by default. This should be changed to netstandard2.0 (or whatever the latest and greatest version my dear reader is using). I know that at the time of writing dotnet core 3 just came out, but the template from Rider defaults to netcoreapp2.1, so I’m using the closest netstandard. There is either a little bug in the Rider template or there is something wrong with my setup.

<PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
</PropertyGroup>

Aah, a building Functions project. Now let’s run it…

The build warning

The next problem surfaces quite quickly. One of the debug outputs already warned me for it, but it becomes painfully clear that there is no AzureWebJobsStorage setup for local development.

The run fail

The AzureWebJobStorage needs a local instance of the Azure Storage Emulator or an actual web storage endpoint. What was not immediately clear for me, is that the Azure Storage Emulator needs a SQL LocalDB instance. To get the LocalDB installer, download the SQL Express database. Select Download Media and in the next screen, select the LocalDB option.

Download Media
Select LocalDB

The download location will open automatically. There will be an .msi installer to easily install the LocalDB database. I have accidentally installed SQL Server 2017 as well, which gave me problems while initialising the LocalDB. To circumvent those problems, install the latest Cumulative Update for SQL Server 2017. The problem was that the Azure Storage Explorer didn’t work properly because it could not connect properly to the LocalDB. That prevented the Azure Storage explorer form creating a database with all the tables it needs. By the way, some articles told me to manually create the AzureStorageEmulatorDb## database. That won’t solve the problem, it will just mask the problem as the database won’t have the necessary tables.

Once that’s all done, verify that the LocalDB installed correctly by running this command SQLLocalDB.exe i. It should print out MSSQLLocalDB. Don’t forget to start the database with the command SqlLocalDB.exe s MSSQLLocalDB. Now, the Storage Emulator should work, right? Wrong!

There is no database with the specific name AzureStorageEmulatorDb59. I found this out after I tried starting the Storage Emulator and seeing that the initialisation of the emulator crashed. So, I tried running the command:

> AzureStorageEmulator.exe init
Windows Azure Storage Emulator 5.9.0.0 command line tool
Found SQL Instance (localdb)\MSSQLLocalDB.
Creating database AzureStorageEmulatorDb59 on SQL instance '(localdb)\MSSQLLocalDB'.
Cannot create database 'AzureStorageEmulatorDb59' : The database 'AzureStorageEmulatorDb59' does not exist. Supply a valid database name. To see available databases, use sys.databases..
One or more initialization actions have failed. Resolve these errors before attempting to run the storage emulator again.
Error: Cannot create database 'AzureStorageEmulatorDb59' : The database 'AzureStorageEmulatorDb59' does not exist. Supply a valid database name. To see available databases, use sys.databases..

To remedy this, open SQL Server Management Studio, connect to the LocalDB instance and create a database with the name AzureStorageEmulatorDb59. I tried doing this in Rider with the built in DataGrip tools, but I got an error. I’ve reported this, so it will get fixed in the future. See earlier remark about the Cumulative Update! That should prevent this error from happening. I’m keeping it in as I think others will run into the same problem.

Now that the database is set up, I can finally start the storage emulator. All I have to do is fill in the AzureWebJobsStorage with UseDevelopmentStorage=true. All set, lets run the function. Unfortunately, the output still complains that the AzureWebJobsStorage still isn’t filled in. After some checking in the config and the place where the function runs, it appears that the local.settings.json file is not being copied. So, I change the Copy to output directory setting in the file properties to Copy Always. Now the Azure function starts up and a minute later, the breakpoint in my function is hit.

To make my life easier, I should add some “Before launch” external tools arguments in the Run/Debug Configuration so the LocalDB and Azure Storage Emulator start before each run. I think I’ll set that up later.

Now everything is set up correctly. I can run and debug Azure Functions with Rider. It’s not as transparent process as I’d hoped it would be, but it was a good learning experience for me.