How to use a private Amazon ECR registry with Gitlab without updating runners configuration Blog Devoteam Rebirth

Temps de lecture : 5 minutes

Similarly to my previous article about pulling another private Gitlab project from a repository, this article will explain a basic feature having few or scattered examples on the Internet. This post will give you a complete example. If you would like to have the final solution without reading the steps, you can go directly to the « Propagate credentials using dotenv » section.

Context

Having a complex project, you may want to use pre-package build environments from a private registry to speed up the installs. However, it is not quite simple to set up using Gitlab, the first reason is that Gitlab requires you to have registry credentials prior to starting your job container. If you attempt to use a private ECR registry directly, you will certainly end with the following error:

Running with gitlab-runner 16.4.2 (abcd)
  on ip-1-2-3-4-aws xxx, system ID: yyy
Preparing the "docker" executor
Using Docker executor with image 123456798012.dkr.ecr.eu-west-1.amazonaws.com/my-image:3.10 ...
Pulling docker image 123456798012.dkr.ecr.eu-west-1.amazonaws.com/my-image:3.10 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://123456798012.dkr.ecr.eu-west-1.amazonaws.com/v2/my-image/manifests/3.10": no basic auth credentials (manager.go:237:0s)
ERROR: Job failed: failed to pull image "123456798012.dkr.ecr.eu-west-1.amazonaws.com/my-image:3.10" with specified policies [always]: Error response from daemon: Head "https://123456798012.dkr.ecr.eu-west-1.amazonaws.com/v2/my-image/manifests/3.10": no basic auth credentials (manager.go:237:0s)

Consequently, you must find a solution to have your registry credentials before starting the pipeline. Let’s search on the Internet how to do that and browse the different solutions until the perfect one.

Pipeline credentials (no automation)

The first solution you may find on the Internet is probably the most straightforward: inject a DOCKER_AUTH_CONFIG variable directly in your pipeline variables.

In particular, the DOCKER_AUTH_CONFIG environment variable tells Docker to use specific credentials for given registries. The AWS CLI for ECR have a specific command allowing to generate such a temporary password for Docker authentication, consequently, you can generate it locally by using the below commands:

ECR_PASSWORD = $(aws ecr get-login-password  --region eu-west-1)

The authentication must be a base64 string with the form AWS:<PASSWORD> consequently we must generate the authentication as below

GENERATED_AUTH_CHAIN = $(echo -n "AWS:${ECR_PASSWORD}" | base64)

Then, you can write a DOCKER_AUTH_CONFIG environment variable with the following format:

{
  "auths": {
    "123456798012.dkr.ecr.eu-west-1.amazonaws.com": {
      "auth": "<GENERATED_AUTH_CHAIN>"
    }
  }
}

This solution is not ideal, and probably the worst in term of automation. In particular, the temporary Amazon ECR password is valid for only 12 hours!

This constraint will get you updating this pipeline variable everyday you would like to run your pipelines, which is pretty annoying. Fortunately, there is other solutions.

Runner permissions (not applicable to my usecase)

The most common documented option on the Internet is to give the runner the permission to generate these temporary credentials using the amazon-ecr-credentials-helper software.

I will not provide additional details here, because there is already great posts explaining how to to perform this installation:

https://faun.pub/authenticating-your-gitlab-ci-runner-to-an-aws-ecr-registry-using-amazon-ecr-docker-credential-b4604a9391eb

This solution is great, because credentials will now be generated automatically. However, my company uses shared runners, and we cannot easily change their configuration to have it pulling all our private registries. The technological vision of these shared runners is keeping it really standard with very few customizations.

We are now back to the starting point, I would like to have automatically generated Docker credentials, but I would like to generate it from my pipeline, which… require credentials to run… Don’t leave this page, we are now closer to the final solution, let’s explore a solution updating the pipeline’s environment variables at runtime.

Update project variables at runtime (ok, but not a great option)

A few years ago, looking desperately for a solution, I found this amazing blog post from Michael Herman. He explains something damn ingenious: you actually can generate the Docker credentials in your first CI job (pulling base image from a public registry) and then update your CI/CD project variables using the Gitlab CLI, and let the other jobs run pulling credentials from the environment variable you have set in you project!

Michael’s solution uses AWS access keys, which is not really great from a security point of view. To fix that, I tweaked his script a little bit to use an AWS federated identity for Gitlab.

Below is the complete Michael’ gitlab template with my update. Checkout the Gitlab documentation to get this pipeline work with AWS federation.

docker-login:
  stage: docker-login
  image:
    name: public.ecr.aws/aws-cli/aws-cli:2.13.28
    entrypoint: [""]
  before_script:
    - yum install -y jq
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  script:
    - >
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn arn:aws:iam::13456789012:role/ecr-pull-images
      --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
      --web-identity-token ${GITLAB_OIDC_TOKEN}
      --duration-seconds 3600
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))
    - AWS_PASSWORD=$(aws ecr get-login-password --region eu-west-1) && ENCODED=$(echo -n "AWS:$AWS_PASSWORD" | base64) && PAYLOAD=$( jq -n --arg userpass "$ENCODED" '{"auths": {"123456789012.dkr.ecr.us-east-1.amazonaws.com": {"auth": $userpass}}}' ) && curl --request PUT --header "PRIVATE-TOKEN:$TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/variables/DOCKER_AUTH_CONFIG" --form "value=$PAYLOAD"

Propagate credentials using dotenv (the best solution for my usecase)

The previous solution works well, however, it requires to have a job step messing with the project’s CI/CD variables, and I am not comfortable with that idea. Fortunately, I recently ended up on a comment from Cajetan Rodrigues in one of the hundred tickets existing in the Gitlab bug tracker. Cajetan solved this last concern, let’s see how.

In the ticket’s comments, Cajetan explains how to take advantage of the dotenv report parameter. In particular, you can actually pass the generated credentials through the pipeline directly, without having to update the project variables, by converting an artifact to environment variables for the next pipeline steps. More details can be found in the Gitlab documentation about dotenv.

Adding this tweak to Michael’s solution and including the federation mechanism I integrated on my side: I now have a solution I now consider as the best for that usecase.

An initial job pulls the aws cli image from Amazon public ECR registry. Inside this job, we authenticate to AWS using a federated OIDC token and generate Docker credentials for Amazon ECR
Then, we output a DOCKER_AUTH_CONFIG variable from this job in an artifact output using the Gitlab dotenv report
Finally, we declare a dependency to this initial job on all other jobs requiring to login to Amazon ECR. Doing that, the DOCKER_AUTH_CONFIG will be injected on job initialization, the job can pull image and then start.

The complete .gitlab-ci example is below. Many thanks to Michael Herman and Cajetan Rodrigues for having shared their knowledge on the Internet years ago. If you are looking for the federation configuration instructions in AWS, details are available in the Gitlab documentation.

stages:
  - ecr-login
  - test

# Initial job step: will generate Amazon ECR credentials
# Generated DOCKER_AUTH_CONFIG will be exported in a dotenv report

ecr-login:
  stage: ecr-login
  image:
    name: public.ecr.aws/aws-cli/aws-cli:2.13.28
    entrypoint: [""]
  before_script:
    - yum install -y jq
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  script:
    - >
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn arn:aws:iam::123456789012:role/ecr-pull-images
      --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
      --web-identity-token ${GITLAB_OIDC_TOKEN}
      --duration-seconds 3600
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))
    - |
      AWS_PASSWORD=$(aws ecr get-login-password --region eu-west-1)
      ENCODED_PASSWORD=$(echo -n "AWS:$AWS_PASSWORD" | base64 -w 0)
      AUTH_PAYLOAD=$( jq --indent 0 -n --arg userpass "$ENCODED_PASSWORD" '{"auths": {"123456789012.dkr.ecr.eu-west-1.amazonaws.com": {"auth": $userpass}}}' )
      echo "DOCKER_AUTH_CONFIG=${AUTH_PAYLOAD}" > ${CI_PROJECT_DIR}/auth.env
  artifacts:
    reports:
      dotenv: auth.env

# Sample job using a private registry image in Amazon ECR

terraform-fmt-check:
  stage: test
  image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/my-custom-image:latest
  dependencies:
   - ecr-login
  script:
    - terraform fmt -check=true -diff=true