Towards a CI/CD world without credentials
Did you know about the largest database account leaked ? More than 12 millions account credentials have been leaked.
And beside that, in our traditional CI/CD platforms, we store credentials for cloud deployments (access key/secret key, service account, login/password) somewhere to be in the end available as an environment variable. But sometimes GitLab or GitHub had also some exposure.
Great, what are the other problems ? Here are some:
- Security breach on the CI/CD platform
- Password copy/paste on your desktop computer
- Rotating secrets every 3 months
- Ever lasting credentials available for every one with a simple
echo
Not really fun.
So today i’m writing this article to help you to change one of a common bad habit and work with Cloud without saving any credentials in your CI/CD. To do so we’ll setup an identity provider and generate temporary credentials to limit the risk as much as we can in case of exposure.
Maybe you already read our post on GitLab and Vault, it is similar but we will go a step further and remove the need of a vault.
Prerequisites
For this article you will need to execute commands to setup your cloud platform. So you will need theses:
- AWS
- aws CLI v2
- Write permission in your account.
- GCP
- gcloud CLI
- Write permission in your project(s).
Setup your Cloud
Getting to the point, you will need to setup your favorite Cloud platform to trust your favorite CI/CD. Be smart and pick the only ones regarding your needs.
The identity provider of CI/CD platform is a key element of this mechanism. Because we trust it, it must guaranty that all signed identity must be valid. The other element is the identity federation in your cloud. It is the one that will guaranty the chain of trust.
Setup AWS
1- Create the Identity federation
The IAM Identity provider is a resource in AWS that help you assume a role by trusting an external identity provider. In our case we will use the Open ID Connect identity provider standard.
# If you're using Gitlab
IDENTITY_PROVIDER_HOST='gitlab.com'
# If you're using GitHub
IDENTITY_PROVIDER_HOST='token.actions.githubusercontent.com'
# Trust the Identity Provider
fingerprint="$(openssl s_client -connect "${IDENTITY_PROVIDER_HOST}:443" </dev/null 2>/dev/null |
openssl x509 -fingerprint -noout | awk -F= '{ print $2 }' |
sed 's/://g')"
aws iam create-open-id-connect-provider \
--url "https://${IDENTITY_PROVIDER_HOST}" --thumbprint-list "${fingerprint}" \
--client-id-list "https://${IDENTITY_PROVIDER_HOST}"
2- Create a CI/CD role
To have an access level in AWS, all come with permission policies. These permissions can be attached to a user, and in a federation scenario as currently, we can attach it to a role. In AWS, a role is an entity having temporary credentials tied to a set of permissions in the AWS cloud. The role is used to perform all your AWS cloud actions. And because we trust our created Identity Provider, we allow it to assume a role only if the request is coming from the right GitLab project or GitHub repository.
Note: The IAM Identity provider and the role must be in the same account.
AWS_ACCOUNT_ID='1234567890'
# Role for a specific GitLab project
GITLAB_PROJECT_PATH='your-group/your-project'
aws iam create-role --role-name 'cicd' --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "'"arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${IDENTITY_PROVIDER_HOST}"'"
},
"Condition": {
"StringLike": {
"'"${IDENTITY_PROVIDER_HOST}:sub"'": "'"project_path:${GITLAB_PROJECT_PATH}:ref_type:*:ref:*"'"
}
}
}
]
}'
# Role for specific GitHub repository
GITHUB_REPOSITORY_PATH='your-org/your-repository'
aws iam create-role --role-name 'cicd' --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "'"arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${IDENTITY_PROVIDER_HOST}"'"
},
"Condition": {
"StringLike": {
"'"${IDENTITY_PROVIDER_HOST}:sub"'": "'"repo:${GITHUB_REPOSITORY_PATH}:*"'"
}
}
}
]
}'
3- Attach policy to your role
To be able to perform actions with your role, you should attach policies. Let’s see an example with S3 read only actions. Be careful and respect the Least Privilege Principle.
aws iam attach-role-policy --role-name 'cicd' --policy-arn 'arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess'
You can repeat this command as much as you need to attach all necessaries policies to allow your CI/CD to deploy your infrastructure and application.
Setup GCP
1- Create the Identity Federation
The Workload Identity Federation is a resource used to trust an external identity provider (Here GitLab or GitHub) and then once the Identity has been confirmed, try to assume a targeted service account.
export GCP_WIF_PROJECT_ID='your-project-id-for-workload-identity-federation'
# Create the pool
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools create --location global cicd
# Trust the GitLab Identity Provider
export GITLAB_ORGANIZATION='your-organization'
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools providers create-oidc \
--location=global --workload-identity-pool=cicd gitlab \
--issuer-uri='https://gitlab.com' --allowed-audiences='https://gitlab.com' \
--attribute-mapping='google.subject="gitlab::" + assertion.project_id,attribute.root_group=assertion.sub.extract("project_path:{root_group}/"),attribute.project_id=assertion.project_id' \
--attribute-condition='attribute.root_group == "'"${GITLAB_ORGANIZATION}"'"'
# Trust the GitHub Identity Provider
export GITHUB_ORGANIZATION='your-organization'
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools providers create-oidc \
--location=global --workload-identity-pool=cicd gitlab \
--issuer-uri='https://token.actions.githubusercontent.com' \
--attribute-mapping='google.subject="github::" + assertion.repository_id,attribute.repository_owner=assertion.repository_owner,attribute.repository_id=assertion.repository_id' \
--attribute-condition='attribute.repository_owner == "'"${GITHUB_ORGANIZATION}"'"'
2- Create your service account
To have an access level in GCP, all come with permissions. These permissions can be attached to a user, and in a federation scenario as currently, we can attach it to a service account. In GCP, a service account is an identity having temporary credentials tied to a set of permissions in the GCP cloud.
The service account is used to perform all your google cloud actions. And because we trust our created Workload Identity Pool, we allow it to impersonate a service account only if the request is coming from the right GitLab project or GitHub repository.
export GCP_PROJECT_ID='your-project-id'
export GCP_WIF_PROJECT_ID='your-project-id-for-workload-identity-federation'
export GCP_WIF_PROJECT_NUMBER=$(gcloud projects describe "${GCP_WIF_PROJECT_ID}" --format=value(projectNumber))
# Create your service account
gcloud --project="${GCP_PROJECT_ID}" iam service-accounts create cicd
# Allow impersonation for the gitlab project
gcloud --project="${GCP_PROJECT_ID}" iam service-accounts add-iam-policy-binding "cicd@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/${GCP_WIF_PROJECT_NUMBER}/locations/global/workloadIdentityPools/cicd/attribute.project_id/${GITLAB_PROJECT_ID}"
# Allow impersonation for the github repository
gcloud iam service-accounts add-iam-policy-binding cicd \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/${GCP_WIF_PROJECT_NUMBER}/locations/global/workloadIdentityPools/cicd/attribute.repository_id/${GITHUB_REPOSITORY_ID}"
Note: The Workload Identity Federation and the service account can be in separated projects.
Setup your favorite CI/CD platform
Now that we have setup the trust chain, we will just need to ask for temporary credentials. Lets wrap this into a cloud_authentication.sh
file to be more readable and reusable.
Let’s begin with a schema to explain how it is working.
- Your trusted identity provider will generate a signed token to identify the job and where and when it is running
- The
cloud_authenticate.sh
above will use this signed token to ask for temporary credentials - During the ask for credential, the cloud identity federation will verify the signature of the token to ensure it is valid
- Finally the cloud identity federation generate temporary credentials to send then back to the
cloud_authenticate.sh
Now let’s do it…
Setup your GitLab job
Your-Awesome-Job:
script:
- source cloud_authentication.sh
- echo "your cloud stuff"
Setup your GitHub action job
jobs:
Your-Awesome-Job:
permissions:
id-token: write
steps:
- run: source cloud_authentication.sh
- run: echo "your cloud stuff"
Implement your script for AWS
#!/usr/bin/env bash
# @file cloud_authentication.sh
export AWS_ACCOUNT_ID='00000000' # <= replace with your AWS account id where your role and IAM identity provider is setup
aws_assume_role() {
local credentials
mapfile -t credentials < <(aws_sts_assume_role_with_web_identity "$@" | aws_sts_to_credentials)
if [[ -n "${credentials+x}" ]]; then
export AWS_ACCESS_KEY_ID=${credentials[0]}
export AWS_SECRET_ACCESS_KEY=${credentials[1]}
export AWS_SESSION_TOKEN=${credentials[2]}
true
else
false
fi
}
aws_sts_assume_role_with_web_identity() {
local account_id=$1
local role_name=$2
aws sts assume-role-with-web-identity \
--role-arn "arn:aws:iam::${account_id}:role/${role_name}" \
--role-session-name "$(role_session_name)" \
--web-identity-token "$(jwt_token)" \
--duration-seconds 3600
}
role_session_name() {
if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
echo "GitLabProject-${CI_PROJECT_ID}-Job-${CI_JOB_ID}"
elif [[ -n "${CI+x}" ]]; then # GitHub
echo "GitHubRepository-${GITHUB_REPOSITORY_ID}-Job-${GITHUB_JOB}"
else
exit 2
fi
}
jwt_token() {
if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
echo "${CI_JOB_JWT_V2}"
elif [[ -n "${CI+x}" ]]; then # GitHub
echo "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
else
exit 2
fi
}
aws_sts_to_credentials() {
jq -r '.Credentials | .AccessKeyId,.SecretAccessKey,.SessionToken'
}
aws_assume_role "${AWS_ACCOUNT_ID}" 'cicd'
Implement your script for GCP
#!/usr/bin/env bash
# @file cloud_authentication.sh
export GCP_WORKLOAD_IDENTITY_POOL_PROJECT_NUMBER='0000000000' # <= replace with your GCP project number where workload identity pool is setup
export GCP_PROJECT_ID='your-project-id' # <= replace with your GCP project ID where your service account is setup
gcp_impersonate() {
export GOOGLE_APPLICATION_CREDENTIALS=".gcp_credentials"
gcp_create_impersonation_credentials "$@"
# verify the impersonation is working
gcloud --quiet auth application-default print-access-token >/dev/null
}
gcp_create_impersonation_credentials() {
local workload_identity_pool_project_number=$1
local project_id=$2
local service_account_name=$3
local service_account="${service_account_name}@${project_id}.iam.gserviceaccount.com"
local pool_provider
pool_provider="$(gcp_pool_provider)"
local credential_source_file=".gcp_credential_source"
jwt_token >"${credential_source_file}"
cat >"${GOOGLE_APPLICATION_CREDENTIALS}" <<EOT
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/${workload_identity_pool_project_number}/locations/global/workloadIdentityPools/cicd-pool/providers/${pool_provider}",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${service_account}:generateAccessToken",
"credential_source": {
"file": "${credential_source_file}",
"format": {
"type": "text"
}
}
}
EOT
}
gcp_pool_provider() {
if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
echo "gitlab"
elif [[ -n "${CI+x}" ]]; then # GitHub
echo "github"
else
exit 2
fi
}
jwt_token() {
if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
echo "${CI_JOB_JWT_V2}"
elif [[ -n "${CI+x}" ]]; then # GitHub
echo "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
else
exit 2
fi
}
gcp_impersonate "${GCP_WORKLOAD_IDENTITY_POOL_PROJECT_NUMBER}" "${GCP_PROJECT_ID}" "cicd"
Conclusion
Now you should be able to deploy in AWS or GCP from GitLab or GitHub. I hope you will never ever need to save credentials into your CI/CD platform. If it is still the case, maybe you can store these into a Secret Manager or reproduce the mechanism above for your specific components.
Happy coding.
FAQ
It exists a method much simple for GitHub. Why not using it ?
It is because I like to know what is happening and how things are working.
Haha ! I found a security breach. Temporary credentials can be printed and used as-is. What can you do now ?
Yeah that is true. Just like the credentials saved in the CI/CD platform. But here they expire in 1 hour. So it is eventually more secured than the previous used method.
And Azure ?
Sorry, I’m not able to do it with this cloud provider. Because they are all similar, it should be adaptable.
Sources
If you need to go futher (and maybe simplify, enhance, fix all of theses) here are the links I used to write this article:
- GitLab Docs
- GitHub Docs – About security hardening with OpenID Connect
- AWS Documentation
- Google Cloud Documentation – Enabling keyless authentication from GitHub Actions
- OpenID Connect
- JWT