GitLab CI/CD Series: Putting it all together and making adjustments

This is the fifth post in the GitLab CI/CD Series (oh, how the time flies), where we'll be setting up a build & deploy pipeline using GitLab CI/CD. In the previous posts, we built all the necessary pieces, and now it's time to finally put them all together!

If you've never worked with GitLab CI/CD before, the first post is a great thing to have a look through, as it covers most of the basics.

Recap & summary

Here are the assumptions about the project:

  • The project is hosted on GitLab
  • One repository contains both .NET and Angular applications
  • The Docker GitLab Runner executor will be used
  • The CI/CD pipeline will consist of the following stages: build, staging, and production, with the last two being all about deployment to different environments
  • The project structure is:
client/
└── webapp/
    ├── src/
    ├── angular.json
    ├── package.json
    └── tsconfig.json
server/
└── MyApp/
    ├── src/
    │   ├── Api/
    │   │   ├── Infrastructure
    │   │   ├── Controllers
    │   │   ├── Api.csproj
    │   │   ├── appsettings.json
    │   │   ├── Program.cs
    │   │   └── Startup.cs
    │   ├── Logic/
    │   │   ├── Services
    │   │   └── Logic.csproj
    │   └── Data/
    │       ├── Models
    │       ├── DbContext
    │       └── Data.csproj
    └── MyApp.sln
Text

And this the the current look of the gitlab-ci.yml file:

stages:
  - build
  - deploy
    
variables:
  RUNTIME_ID: 'win-x86'
  MIGRATIONS_RUNTIME_ID: 'alpine-x64'
  AZURE_STORAGE_CONTAINER: '$$web'

build-frontend:
  stage: build
  image: node:16.9.0
  rules:
    - changes:
      - 'client/**/*'
  cache:
    key:
      prefix: 'frontend'
      files:
        - 'client/webapp/package-lock.json'
    paths:
      - 'client/webapp/node_modules'
  before_script:
    - 'cd client/webapp/'
    - 'npm ci'
  script:
    - 'npm run build'
  artifacts:
    name: 'frontend-${CI_COMMIT_SHORT_SHA}'
    paths:
      - 'client/webapp/dist'
      
build-backend:
  stage: build
  image: mcr.microsoft.com/dotnet/sdk:6.0
  rules:
    - changes:
      - 'server/**/*'
  cache:
    key: 'backend'
    paths:
      - '.nuget'
  before_script:
    - 'dotnet restore server/MyApp/src/Api/Api.csproj --packages .nuget/ --runtime ${RUNTIME_ID}'
    - 'dotnet tool install --global dotnet-ef --version 6.0.5'
    - 'mkdir -p ./tmp/build-artifact'
    - 'mkdir -p ./tmp/migrations-artifact'
  script:
    - 'dotnet publish server/MyApp/src/Api/Api.csproj -c Release --no-restore --runtime ${RUNTIME_ID} --no-self-contained -o ./tmp/build-artifact'
    - 'cd server/MyApp && dotnet ef migrations bundle --target-runtime ${MIGRATIONS_RUNTIME_ID} --self-contained -o ../../tmp/migrations-artifact/efbundle'
  artifacts:
    name: "backend-$CI_COMMIT_REF_NAME"
    paths:
      - 'tmp/build-artifact'
      - 'tmp/migrations-artifact'

deploy-frontend:
  image: mcr.microsoft.com/azure-cli
  rules:
    - changes:
      - 'client/**/*'
  dependencies:
    - build-frontend
  before_script:
    - 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
    - 'STORAGE_KEY=`az storage account keys list --account-name ${AZURE_STORAGE_NAME} --output json --query "[0].value"`'
    - 'az storage container create --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --name ${AZURE_STORAGE_CONTAINER} --public-access blob'
  script:
    - 'az storage blob upload-batch --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --overwrite true --source ./client/myapp/dist/ --destination ${AZURE_STORAGE_CONTAINER}'

deploy-backend:
  image: mcr.microsoft.com/azure-cli
  rules:
    - changes:
      - 'server/**/*'
  dependencies:
    - build-backend
  before_script:
    - 'pushd ./tmp/build-artifact'
    - 'zip -r ../../publish.zip .'
    - 'popd'
    - 'apk add gcompat'
    - 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
    - 'DB_STRING=`az sql db show-connection-string --client ado.net --server ${AZURE_DB_SERVER} --name ${AZURE_DB_NAME}`'
    - 'DB_STRING=`echo ${DB_STRING:1:-1}`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/<username>/${AZURE_DB_USER}/"`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/<password>/${AZURE_DB_PASSWORD}/"`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/$/TrustServerCertificate=true;/"`'
    - 'IP=`curl -s https://api.ipify.org/`'
    - 'az sql server firewall-rule create --resource-group ${AZURE_RG} --server ${AZURE_DB_SERVER} --name GitLab-Runner --start-ip-address ${IP} --end-ip-address ${IP}'
  script:
    - 'az webapp deploy --clean true --resource-group ${AZURE_RG} --name ${AZURE_APPSERVICE} --src-path publish.zip --type zip'
    - './tmp/migrations-artifact/efbundle --connection "${DB_STRING}"'
YAML

To sum up: the build jobs are created when there are any changes in the client/ or server/ directories. The dependencies of each build are downloaded from the caching service before each build is started and uploaded back to it once the build succeeds. The build-frontend job produces an artifact with the Angular application. The build-backend job's artifact contains the deployable .NET application and an EF Core Migration Bundle that is capable of applying database schema and data updates.

Then the Angular application is deployed to Azure Storage. The .NET Application is deployed to Azure Web Service, and the Entity Framework migrations are applied to the database.

Environments

So all the steps are pretty much done, but the script only handles one environment, which is not enough in most cases. All the deployment targets in the gitlab-ci.yml file are taken from variables that were set in the GitLab UI. That's good because those support scoping them to different environments. But how to define that? It's just a matter of adding a environment: <environment-name> section to a job definition. Seems easy? Good, because it is.

Let's update the CI/CD configuration (only changes are shown here):

stages:
  - build
  - deploy-staging
  - deploy-production
  
deploy-frontend-staging:
  stage: deploy-staging
  environment: staging
  image: mcr.microsoft.com/azure-cli
  ...

deploy-backend-staging:
  stage: deploy-staging
  environment: staging
  image: mcr.microsoft.com/azure-cli
  ...

deploy-frontend-production:
  stage: deploy-production
  environment: production
  image: mcr.microsoft.com/azure-cli
  ...

deploy-backend-production:
  stage: deploy-production
  environment: production
  image: mcr.microsoft.com/azure-cli
  ...

Now the variables in the GitLab UI can be scoped to each environment:

This looks okay...ish, the deployment can now be done to more than one environment. But the gitlab-ci.yml file has gotten really long because the deployment steps are repeated for each environment. The readability of the file dropped enormously and not to mention how difficult it will be to maintain in the future - if there are more environments, then it will be even more difficult.

YAML has a feature called anchors, which can be used to reuse content across the file. It's a great feature, but for someone not very familiar with the syntax it might look a little quirky.

.job_template: &job_configuration
  image: mcr.microsoft.com/azure-cli
  ...

job-1:
  <<: *job_configuration
  script:
    - 'echo TEST'

It seems that the GitLab folks thought the same as they introduced a nicer way of reusing configurations:

.job_template
  image: mcr.microsoft.com/azure-cli
  ...
  
job-1:
  extends: .job-template
  script:
    - 'echo TEST'

More documentation about the feature is available at Use extends to reuse configuration sections (there's even a possibility to split the configuration into several files!). Let's apply this to the gitlab-ci.yaml file:

.deploy-frontend:
  image: mcr.microsoft.com/azure-cli
  ...

.deploy-backend:
  image: mcr.microsoft.com/azure-cli
  ...

deploy-frontend-staging:
  extends: .deploy-frontend
  stage: deploy-staging
  environment: staging

deploy-backend-staging:
  extends: .deploy-backend
  stage: deploy-staging
  environment: staging

deploy-frontend-production:
  extends: .deploy-frontend
  stage: deploy-production
  environment: production

deploy-backend-production:
  extends: .deploy-backend
  stage: deploy-production
  environment: production

That's so much neater.

Side note: Any job whose name starts with the dot (.) character will not be executed on its own.

All might seem well, but after a few deployments, you might notice that one of the deployment jobs fails with the following message:

That's because GitLab assumes that the deploy-backend-* and deploy-frontend-* jobs deploy to the same place and will prevent deploying outdated artifacts. It can be easily fixed - let's change the environment names a bit:

deploy-frontend-staging:
  extends: .deploy-frontend
  stage: deploy-staging
  environment: staging/frontend

deploy-backend-staging:
  extends: .deploy-backend
  stage: deploy-staging
  environment: staging/backend

deploy-frontend-production:
  extends: .deploy-frontend
  stage: deploy-production
  environment: production/frontend

deploy-backend-production:
  extends: .deploy-backend
  stage: deploy-production
  environment: production/backend

Great, but does that mean that all the variables will need to be declared multiple times? No. Have a look at Scope environments with specs - the environment definition in the variable configuration supports wildcards (*).

Now GitLab will also nicely group the environments in the Deployments tab under the Environments option. It's a great place to quickly see what's currently deployed to each environment.

Limiting the number of deployments

So one thing that might not be perfect is that the deployments are triggered automatically for any code change, and deploying everything to production is not always desired. To limit the number of deployments two things can be introduced.

Rules

There's a variety of rules that can be specified that define when the job will be created. Full documentation can be found at rules. There's a myriad of options available, but as an example, we will add two rules:

  • jobs should be created only for commits to the master branch
  • or for commits in a merge request

The rules may be applied to each job definition separately, but they can also be defined for all of them in one place, using the workflow section at the root of the YAML file:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == "master"'

When

Sometimes you might want jobs to be created but not run immediately. That's where the when keyword comes to play. There are a couple of options that you can look up in the documentation, but for this case, the manual option is perfect.

deploy-frontend-staging:
  extends: .deploy-frontend
  stage: deploy-staging
  environment: staging/frontend
  when: manual

deploy-backend-staging:
  extends: .deploy-backend
  stage: deploy-staging
  environment: staging/backend
  when: manual

deploy-frontend-production:
  extends: .deploy-frontend
  stage: deploy-production
  environment: production/frontend
  when: manual

deploy-backend-production:
  extends: .deploy-backend
  stage: deploy-production
  environment: production/backend
  when: manual

After these changes, the build will still be triggered automatically, but the deployments will require manual action through the GitLab UI. Unfortunately, restricting access to deployments is only available in the GitLab Premium tier.

One more thing

This is the last improvement that will be made - I promise. Last week there was a post on this blog about running the EF Core Migration Bundles on Alpine Linux that can be found here:

Running EF Core Migration Bundle on Alpine Linux - problems and optimisations
Entity Framework Core Migration Bundles can greatly ease the process of applying database changes during the deployment of your application.

There's one great optimisation mentioned there - using a custom .NET Environment to skip configuring unnecessary services when building and running the Migrations Bundle. This requires a tiny change in the Startup class of your .NET application (find them in the post above) and these updates to the .gitlab-ci.yml file:

variables:
  RUNTIME_ID: 'win-x86'
  MIGRATIONS_RUNTIME_ID: 'alpine-x64'
  AZURE_STORAGE_CONTAINER: '$$web'
  DOTNET_ENV: 'BuildDeployment'
  
build-backend:
  ...
  before_script:
    ...
    - 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
    ...
    
.deploy-backend:
  ...
  before_script:
    ...
    - 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
    ...

🎉🎉🎉

And it's done! That's it! It was a long ride, but the Gitlab CI/CD configuration is finally complete:

stages:
  - build
  - deploy-staging
  - deploy-production

variables:
  RUNTIME_ID: 'win-x86'
  MIGRATIONS_RUNTIME_ID: 'alpine-x64'
  AZURE_STORAGE_CONTAINER: '$$web'
  DOTNET_ENV: 'BuildDeployment'

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == "master"'

build-frontend:
  stage: build
  image: node:16.9.0
  dependencies: []
  rules:
    - changes:
      - 'client/**/*'
  cache:
    key:
      prefix: 'frontend'
      files:
        - 'client/webapp/package-lock.json'
    paths:
      - 'client/webapp/node_modules'
  before_script:
    - 'cd client/webapp/'
    - 'npm ci'
  script:
    - 'npm run build'
  artifacts:
    name: 'frontend-${CI_COMMIT_SHORT_SHA}'
    paths:
      - 'client/webapp/dist'

build-backend:
  stage: build
  image: mcr.microsoft.com/dotnet/sdk:6.0
  dependencies: []
  rules:
    - changes:
      - 'server/**/*'
  cache:
    key: 'backend'
    paths:
      - '.nuget'
  before_script:
    - 'dotnet restore server/MyApp/src/Api/Api.csproj --packages .nuget/ --runtime ${RUNTIME_ID}'
    - 'dotnet tool install --global dotnet-ef --version 6.0.5'
    - 'mkdir -p ./tmp/build-artifact'
    - 'mkdir -p ./tmp/migrations-artifact'
    - 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
  script:
    - 'dotnet publish server/MyApp/src/Api/Api.csproj -c Release --no-restore --runtime ${RUNTIME_ID} --no-self-contained -o ./tmp/build-artifact'
    - 'cd server/MyApp && dotnet ef migrations bundle --target-runtime ${MIGRATIONS_RUNTIME_ID} --self-contained -o ../../tmp/migrations-artifact/efbundle'
  artifacts:
    name: "backend-$CI_COMMIT_REF_NAME"
    paths:
      - 'tmp/build-artifact'
      - 'tmp/migrations-artifact'

deploy-backend-staging:
  extends: .deploy-backend
  stage: deploy-staging
  environment: staging/backend

deploy-frontend-staging:
  extends: .deploy-frontend
  stage: deploy-staging
  environment: staging/frontend

deploy-backend-production:
  extends: .deploy-backend
  stage: deploy-production
  environment: production/backend

deploy-frontend-production:
  extends: .deploy-frontend
  stage: deploy-production
  environment: production/frontend

.deploy-backend:
  image: mcr.microsoft.com/azure-cli
  rules:
    - changes:
      - 'server/**/*'
  dependencies:
    - build-backend
  before_script:
    - 'pushd ./tmp/build-artifact'
    - 'zip -r ../../publish.zip .'
    - 'popd'
    - 'apk add gcompat'
    - 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
    - 'DB_STRING=`az sql db show-connection-string --client ado.net --server ${AZURE_DB_SERVER} --name ${AZURE_DB_NAME}`'
    - 'DB_STRING=`echo ${DB_STRING:1:-1}`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/<username>/${AZURE_DB_USER}/"`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/<password>/${AZURE_DB_PASSWORD}/"`'
    - 'DB_STRING=`echo ${DB_STRING} | sed "s/$/TrustServerCertificate=true;/"`'
    - 'IP=`curl -s https://api.ipify.org/`'
    - 'az sql server firewall-rule create --resource-group ${AZURE_RG} --server ${AZURE_DB_SERVER} --name Gitlab-Runner --start-ip-address ${IP} --end-ip-address ${IP}'
    - 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
  script:
    - 'az webapp deploy --clean true --resource-group ${AZURE_RG} --name ${AZURE_APPSERVICE} --src-path publish.zip --type zip'
    - './tmp/migrations-artifact/efbundle --connection "${DB_STRING}"'
  when: manual

.deploy-frontend:
  image: mcr.microsoft.com/azure-cli
  rules:
    - changes:
      - 'client/**/*'
  dependencies:
    - build-frontend
  before_script:
    - 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
    - 'STORAGE_KEY=`az storage account keys list --account-name ${AZURE_STORAGE_NAME} --output json --query "[0].value"`'
    - 'az storage container create --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --name ${AZURE_STORAGE_CONTAINER} --public-access blob'
  script:
    - 'az storage blob upload-batch --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --overwrite true --source ./client/myapp/dist/ --destination ${AZURE_STORAGE_CONTAINER}'
  when: manual

This is the last post in the GitLab CI/CD Series. I hope you enjoyed them 😊

Cover photo by Ashkan Forouzani on Unsplash