For the past few days, I've been working on creating a CI/CD pipeline using GitLab CI/CD. This was something new, but I welcomed the challenge and was very curious about how it stacks up to other systems I've used in the past. In the end, I was pretty satisfied with the outcome, but there were some challenges to overcome.

In this series, I will try to guide you through creating a CI/CD pipeline for a .NET API application with an Angular frontend. The backend will be hosted on Azure App Service, and the frontend will be Azure Static Web App hosted on Azure Storage.

Basics

The documentation available at GitLab CI/CD covers most topics in great detail. There are some cases where it can be a little confusing or lacking proper examples. You can always try to Google the unclear bits. Unfortunately, community support is not as big as for other CI/CD platforms.

The heart of it all is a modest .gitlab-ci.yml file. That's where all the magic configuration happens. As the extension suggests, it is in YAML format which, is nice because of the readability. Having it in the repository is also an advantage because version control is always a nice feature.

The configuration file consists of jobs organised into stages. A job can do anything: build, run tests, or deploy a previously created artifact. Read more about pipelines, stages and jobs at CI/CD pipelines.

The other important aspect is running the defined steps. As of writing this post, GitLab offers 400 free minutes on their shared runners (environments capable of running the jobs, aka build agents). That's certainly nice and could be enough for very tiny projects. Unfortunately, those runners can be slow, so the number of free minutes is not really a good indicator. There will be a separate post about running your own runners, which will be linked here once it's written, I promise!

What's great is that GitLab supports different styles of executing these steps. Full description of what means can be found at Executors. The gist of it is that the defined jobs can be run either:

• locally, executing the defined steps using the shell of the machine where the action happens,
• in a Docker container, with a specified image, so the environment is always fresh and has all the required dependencies,
• in a Virtual Machine, using VirtualBox or Parallels

There are a couple of more options to choose from. There's a great section with instructions on how to figure out the best one for your use case.

GitLab CI/CD also offers everything else that is needed for a production-grade CI/CD system as variables, job artifacts, test cases, etc.

Assumptions

As this is the first post in the series, here are all the assumptions about the project/repository:

• The project is hosted on GitLab
• One repository contains both .NET and Angular applications
• The Docker GitLab Runner executor will be used
• The CI/CD pipeline will consist of the following stages: build, staging, and production, with the last two being all about deployment to different environments
• The project structure is:

client/
└── webapp/
    ├── src/
    ├── angular.json
    ├── package.json
    └── tsconfig.json
server/
└── MyApp/
    ├── src/
    │   ├── Api/
    │   │   ├── Infrastructure
    │   │   ├── Controllers
    │   │   ├── Api.csproj
    │   │   ├── appsettings.json
    │   │   ├── Program.cs
    │   │   └── Startup.cs
    │   ├── Logic/
    │   │   ├── Services
    │   │   └── Logic.csproj
    │   └── Data/
    │       ├── Models
    │       ├── DbContext
    │       └── Data.csproj
    └── MyApp.sln

Let's do it

Let's start with a basic setup:

stages:
  - build

build-frontend:
  stage: build
  image: node:16.9.0
  before_script:
    - 'cd client/webapp'
    - 'npm ci'
  script:
    - 'npm run build'

This is very simple but does its job. The first section is stages, where the different stages and their order is defined. For now, it's only build, but the list will grow as deployments are added.

Then there's the actual job definition. It's named build-frontend because that's what it does. It's assigned to the previously defined build stage. The next property is image. It defines the Docker image that the GitLab Runner will use to run the job. By default, the image is pulled from DockerHub, but other image registries can be configured. More details and configuration examples can be found at Run your CI/CD jobs in Docker containers.

Then it's time to finally define the commands that should be run by the runner. It's possible to specify those in three sections:

• before_script
• script
• after_script

Their names are pretty descriptive, but the last one is slightly different from the others (there is an Additional details section in the link above that goes over the differences). Generally, the commands specified in before_script and script are merged and run sequentially. This can be useful when working with more advanced gitlab-ci.yml features such as templates. There's a good example in this Stack Overflow answer. In the case of our setup, those two sections could have been merged, but it's nice to separate the "setup" part of the job from the "real" steps.

So the job builds the Angular application, great! But the build's output should be captured as a build artifact. To do that, add the following section to the build-frontend job:

  artifacts:
    name: 'frontend-${CI_COMMIT_SHORT_SHA}'
    paths:
      - 'client/webapp/dist'

Now when the job succeeds, the output of the build will be saved and uploaded to GitLab:

There is one important thing to note about the artifacts section. The path must be inside and relative to the root of the repository. Any attempt to publish an artifact from outside of the default directory will fail with a not supported: outside build directory message. Even trying to use the built-in variable CI_BUILDS_DIR will result in that message, which seems a little over-zealous. If it's a problem for you, these two issues in GitLab's issue tracker might help:

• Files exists but can't be used to build artifacts.
• Extend artifact use , not only for build folder

The defined job does what it's supposed to. But it can still be improved. One benefit of using the Docker executor is having a clean environment every time the job runs. However, when building a frontend application, a lot of time is spent on downloading dependencies. There is no reason they couldn't be cached and reused. Thankfully, the GitLab folks thought about that. The following configuration section does just that:

  cache:
    key: 'frontend'
    paths:
      - 'client/webapp/node_modules'

Now, before the scripts are executed, the GitLab runner will check whether its cache contains an entry for the given key - frontend in this case (this can be either a local filesystem cache or a distributed solution like an Amazon S3 bucket). If a cache exists, it will be downloaded and restored to the same directory it was picked from. After the job succeeds, the cache will be packaged and stored in the cache for future use.

The cache key may be dynamic. It might contain any variable, like so: key: 'frontend-${CI_COMMIT_REF_SLUG}' (this one will be postfixed with a "standardised" branch name). But the key can also be generated based on the contents of a file. This is super useful in the case of frontend dependencies because it can be used like this:

  cache:
    key:
      prefix: 'frontend'
      files:
        - 'client/webapp/package-lock.json'
    paths:
      - 'client/webapp/node_modules'

The content of the files given in the files section will be hashed, and that will be appended after the chosen prefix. Note that only two files can be specified as the source of the cache key. But it's not the end of the world as a single job may contain up to four separate caches.

The caching mechanism in GitLab CI/CD is pretty powerful and well-thought-out. Some jobs may only download the cache without ever uploading it back while others will do the opposite. Read more about it at Caching in GitLab CI/CD.

The job now builds the Angular application, collects the build output into an artifact and even caches and resues npm dependencies. That's not bad for just a couple of lines of YAML configuration.

One caveat though is that the build-frontend job will be created even if there are no changes in the client/ directory. Sometimes that might be okay, but this too can be configured:

  rules:
    - changes:
      - 'client/**/*'

Now, the job will only run when there is a change in any of the files in the client/ directory. Neat!

The final gitlab-ci.yml file looks like this:

stages:
  - build

build-frontend:
  stage: build
  image: node:16.9.0
  rules:
    - changes:
      - 'client/**/*'
  cache:
    key:
      prefix: 'frontend'
      files:
        - 'client/webapp/package-lock.json'
    paths:
      - 'client/webapp/node_modules'
  before_script:
    - 'cd client/webapp/'
    - 'npm ci'
  script:
    - 'npm run build'
  artifacts:
    name: 'frontend-${CI_COMMIT_SHORT_SHA}'
    paths:
      - 'client/webapp/dist'

Of course, this simple setup will not work for everyone. But it is a good starting point. From here, you can analyse the bottlenecks and make adjustments as necessary.

Cover photo by Greyson Joralemon on Unsplash