GitLab CI/CD Series: Putting it all together and making adjustments
This is the fifth post in the GitLab CI/CD Series (oh, how the time flies), where we'll be setting up a build & deploy pipeline using GitLab CI/CD. In the previous posts, we built all the necessary pieces, and now it's time to finally put them all together!
GitLab CI/CD Series Table of Contents
- Building Angular application
- Building .NET API Application and EF Core Migration Bundle
- Deploying Angular application to Azure Storage
- Deploying .NET API to Azure App Service and Applying EF Core Migrations
- Putting it all together and making adjustments (you are here right now)
If you've never worked with GitLab CI/CD before, the first post is a great thing to have a look through, as it covers most of the basics.
Recap & summary
Here are the assumptions about the project:
- The project is hosted on GitLab
- One repository contains both .NET and Angular applications
- The Docker GitLab Runner executor will be used
- The CI/CD pipeline will consist of the following stages:
build,staging, andproduction, with the last two being all about deployment to different environments - The project structure is:
And this the the current look of the gitlab-ci.yml file:
To sum up: the build jobs are created when there are any changes in the
client/ or server/ directories. The dependencies of each build are downloaded from
the caching service before each build is started and uploaded back to it once the build succeeds. The
build-frontend job produces an artifact with the Angular application. The
build-backend job's artifact contains the deployable .NET application and an EF Core Migration
Bundle that is capable of applying database schema and data updates.
Then the Angular application is deployed to Azure Storage. The .NET Application is deployed to Azure Web Service, and the Entity Framework migrations are applied to the database.
Environments
So all the steps are pretty much done, but the script only handles one environment, which is not enough in most cases. All the deployment targets in the gitlab-ci.yml file are taken from variables that were set in the GitLab UI. That's good because those support scoping them to different environments. But how to define that? It's just a matter of adding a environment: <environment-name> section to a job definition. Seems easy? Good, because it is.
Let's update the CI/CD configuration (only changes are shown here):
stages:
- build
- deploy-staging
- deploy-production
deploy-frontend-staging:
stage: deploy-staging
environment: staging
image: mcr.microsoft.com/azure-cli
...
deploy-backend-staging:
stage: deploy-staging
environment: staging
image: mcr.microsoft.com/azure-cli
...
deploy-frontend-production:
stage: deploy-production
environment: production
image: mcr.microsoft.com/azure-cli
...
deploy-backend-production:
stage: deploy-production
environment: production
image: mcr.microsoft.com/azure-cli
...Now the variables in the GitLab UI can be scoped to each environment:
This looks okay...ish, the deployment can now be done to more than one environment. But the gitlab-ci.yml file has gotten really long because the deployment steps are repeated for each environment. The readability of the file dropped enormously and not to mention how difficult it will be to maintain in the future - if there are more environments, then it will be even more difficult.
YAML has a feature called anchors, which can be used to reuse content across the file. It's a great feature, but for someone not very familiar with the syntax it might look a little quirky.
.job_template: &job_configuration
image: mcr.microsoft.com/azure-cli
...
job-1:
<<: *job_configuration
script:
- 'echo TEST'It seems that the GitLab folks thought the same as they introduced a nicer way of reusing configurations:
.job_template
image: mcr.microsoft.com/azure-cli
...
job-1:
extends: .job-template
script:
- 'echo TEST'More documentation about the feature is available at Use extends to reuse configuration sections (there's even a possibility to split the configuration into several files!). Let's apply this to the gitlab-ci.yaml file:
.deploy-frontend:
image: mcr.microsoft.com/azure-cli
...
.deploy-backend:
image: mcr.microsoft.com/azure-cli
...
deploy-frontend-staging:
extends: .deploy-frontend
stage: deploy-staging
environment: staging
deploy-backend-staging:
extends: .deploy-backend
stage: deploy-staging
environment: staging
deploy-frontend-production:
extends: .deploy-frontend
stage: deploy-production
environment: production
deploy-backend-production:
extends: .deploy-backend
stage: deploy-production
environment: productionThat's so much neater.
Side note: Any job whose name starts with the dot (.) character will not be executed on its own.
All might seem well, but after a few deployments, you might notice that one of the deployment jobs fails with the following message:
That's because GitLab assumes that the deploy-backend-* and deploy-frontend-* jobs deploy to the same place and will prevent deploying outdated artifacts. It can be easily fixed - let's change the environment names a bit:
deploy-frontend-staging:
extends: .deploy-frontend
stage: deploy-staging
environment: staging/frontend
deploy-backend-staging:
extends: .deploy-backend
stage: deploy-staging
environment: staging/backend
deploy-frontend-production:
extends: .deploy-frontend
stage: deploy-production
environment: production/frontend
deploy-backend-production:
extends: .deploy-backend
stage: deploy-production
environment: production/backendGreat, but does that mean that all the variables will need to be declared multiple times? No. Have a look at Scope environments with specs - the environment definition in the variable configuration supports wildcards (*).
Now GitLab will also nicely group the environments in the Deployments tab under the Environments option. It's a great place to quickly see what's currently deployed to each environment.
Limiting the number of deployments
So one thing that might not be perfect is that the deployments are triggered automatically for any code change, and deploying everything to production is not always desired. To limit the number of deployments two things can be introduced.
Rules
There's a variety of rules that can be specified that define when the job will be created. Full documentation can be found at rules. There's a myriad of options available, but as an example, we will add two rules:
- jobs should be created only for commits to the
masterbranch - or for commits in a merge request
The rules may be applied to each job definition separately, but they can also be defined for all of them in one place, using the workflow section at the root of the YAML file:
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == "master"'When
Sometimes you might want jobs to be created but not run immediately. That's where the when keyword comes to play. There are a couple of options that you can look up in the documentation, but for this case, the manual option is perfect.
deploy-frontend-staging:
extends: .deploy-frontend
stage: deploy-staging
environment: staging/frontend
when: manual
deploy-backend-staging:
extends: .deploy-backend
stage: deploy-staging
environment: staging/backend
when: manual
deploy-frontend-production:
extends: .deploy-frontend
stage: deploy-production
environment: production/frontend
when: manual
deploy-backend-production:
extends: .deploy-backend
stage: deploy-production
environment: production/backend
when: manualAfter these changes, the build will still be triggered automatically, but the deployments will require manual action through the GitLab UI. Unfortunately, restricting access to deployments is only available in the GitLab Premium tier.
One more thing
This is the last improvement that will be made - I promise. Last week there was a post on this blog about running the EF Core Migration Bundles on Alpine Linux that can be found here:
There's one great optimisation mentioned there - using a custom .NET Environment to skip configuring unnecessary services when building and running the Migrations Bundle. This requires a tiny change in the Startup class of your .NET application (find them in the post above) and these updates to the .gitlab-ci.yml file:
variables:
RUNTIME_ID: 'win-x86'
MIGRATIONS_RUNTIME_ID: 'alpine-x64'
AZURE_STORAGE_CONTAINER: '$$web'
DOTNET_ENV: 'BuildDeployment'
build-backend:
...
before_script:
...
- 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
...
.deploy-backend:
...
before_script:
...
- 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
...🎉🎉🎉
And it's done! That's it! It was a long ride, but the Gitlab CI/CD configuration is finally complete:
stages:
- build
- deploy-staging
- deploy-production
variables:
RUNTIME_ID: 'win-x86'
MIGRATIONS_RUNTIME_ID: 'alpine-x64'
AZURE_STORAGE_CONTAINER: '$$web'
DOTNET_ENV: 'BuildDeployment'
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == "master"'
build-frontend:
stage: build
image: node:16.9.0
dependencies: []
rules:
- changes:
- 'client/**/*'
cache:
key:
prefix: 'frontend'
files:
- 'client/webapp/package-lock.json'
paths:
- 'client/webapp/node_modules'
before_script:
- 'cd client/webapp/'
- 'npm ci'
script:
- 'npm run build'
artifacts:
name: 'frontend-${CI_COMMIT_SHORT_SHA}'
paths:
- 'client/webapp/dist'
build-backend:
stage: build
image: mcr.microsoft.com/dotnet/sdk:6.0
dependencies: []
rules:
- changes:
- 'server/**/*'
cache:
key: 'backend'
paths:
- '.nuget'
before_script:
- 'dotnet restore server/MyApp/src/Api/Api.csproj --packages .nuget/ --runtime ${RUNTIME_ID}'
- 'dotnet tool install --global dotnet-ef --version 6.0.5'
- 'mkdir -p ./tmp/build-artifact'
- 'mkdir -p ./tmp/migrations-artifact'
- 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
script:
- 'dotnet publish server/MyApp/src/Api/Api.csproj -c Release --no-restore --runtime ${RUNTIME_ID} --no-self-contained -o ./tmp/build-artifact'
- 'cd server/MyApp && dotnet ef migrations bundle --target-runtime ${MIGRATIONS_RUNTIME_ID} --self-contained -o ../../tmp/migrations-artifact/efbundle'
artifacts:
name: "backend-$CI_COMMIT_REF_NAME"
paths:
- 'tmp/build-artifact'
- 'tmp/migrations-artifact'
deploy-backend-staging:
extends: .deploy-backend
stage: deploy-staging
environment: staging/backend
deploy-frontend-staging:
extends: .deploy-frontend
stage: deploy-staging
environment: staging/frontend
deploy-backend-production:
extends: .deploy-backend
stage: deploy-production
environment: production/backend
deploy-frontend-production:
extends: .deploy-frontend
stage: deploy-production
environment: production/frontend
.deploy-backend:
image: mcr.microsoft.com/azure-cli
rules:
- changes:
- 'server/**/*'
dependencies:
- build-backend
before_script:
- 'pushd ./tmp/build-artifact'
- 'zip -r ../../publish.zip .'
- 'popd'
- 'apk add gcompat'
- 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
- 'DB_STRING=`az sql db show-connection-string --client ado.net --server ${AZURE_DB_SERVER} --name ${AZURE_DB_NAME}`'
- 'DB_STRING=`echo ${DB_STRING:1:-1}`'
- 'DB_STRING=`echo ${DB_STRING} | sed "s/<username>/${AZURE_DB_USER}/"`'
- 'DB_STRING=`echo ${DB_STRING} | sed "s/<password>/${AZURE_DB_PASSWORD}/"`'
- 'DB_STRING=`echo ${DB_STRING} | sed "s/$/TrustServerCertificate=true;/"`'
- 'IP=`curl -s https://api.ipify.org/`'
- 'az sql server firewall-rule create --resource-group ${AZURE_RG} --server ${AZURE_DB_SERVER} --name Gitlab-Runner --start-ip-address ${IP} --end-ip-address ${IP}'
- 'export ASPNETCORE_ENVIRONMENT=${DOTNET_ENV}'
script:
- 'az webapp deploy --clean true --resource-group ${AZURE_RG} --name ${AZURE_APPSERVICE} --src-path publish.zip --type zip'
- './tmp/migrations-artifact/efbundle --connection "${DB_STRING}"'
when: manual
.deploy-frontend:
image: mcr.microsoft.com/azure-cli
rules:
- changes:
- 'client/**/*'
dependencies:
- build-frontend
before_script:
- 'az login --service-principal -u ${AZURE_SP_ID} -p ${AZURE_SP_SECRET} --tenant ${AZURE_TENANT}'
- 'STORAGE_KEY=`az storage account keys list --account-name ${AZURE_STORAGE_NAME} --output json --query "[0].value"`'
- 'az storage container create --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --name ${AZURE_STORAGE_CONTAINER} --public-access blob'
script:
- 'az storage blob upload-batch --auth-mode key --account-key ${STORAGE_KEY} --account-name ${AZURE_STORAGE_NAME} --overwrite true --source ./client/myapp/dist/ --destination ${AZURE_STORAGE_CONTAINER}'
when: manualThis is the last post in the GitLab CI/CD Series. I hope you enjoyed them 😊
Cover photo by Ashkan Forouzani on Unsplash