Fixing CI/CD security issues you didn’t know you had

CI/CD systems like Jenkins/Github Actions/Gitlab CI need permissions to access other systems they need to act on such as cloud providers, k8s, and others. A popular and straightforward solution for this problem is using long lived credentials. While this approach is convenient, it also presents a security risk. Most teams set and forget these credentials without ever rotating them. If a secret is leaked, an attacker can gain access to your production infrastructure.

At first glance you could say “Let’s just run our own runners in our cloud provider”. This does allow you to switch to using instance roles on the runners and getting away from the long lived credentials. “Woohoo, problem solved”! However that’s just scenario 1, now let’s look at larger organizations where we might have various groups with various cloud accounts. In this case we’ll just say each team has one non-prod account and one prod account. How do you prevent teams from accessing other accounts which that host might have access to via an assume-role or the likes without adding a huge amount of complexity within the build system?

One option would be to use runner groups which would be limited to specific teams/repos. However it’s a pretty poor optimization of resources. It means you need to create specific runners for every team making maintenance a pain as well as increase the operational costs which will push organizations to limit runner availability which will drive developer productivity down.

So how do you get shared hosted or self-hosted runners in a way that still satisfies the security needs, a simpler maintenance/scaling model,, and no long lasting keys?

OIDC FOR THE WIN

Whether you use hosted or self-hosted runners, OIDC setup with your cloud provider is the clear option here. It allows you to essentially “auth” your repo as a machine account without long lasting keys. Here’s the general steps to set it up with AWS and Github as an example. We’ll provide other examples with Gitlab, Vault, etc in the future.

1. Set up an identity provider — each AWS account will need it setup for each source.
2. Set up a role which trusts the above identity providers with the repo/org constraints
3. Configure AWS credentials actions to assume the above role
4. Remaining steps in the pipeline act as that role

So what’s actually going on behind the scenes here is basically the external service — in this case Github, however it could easily be Gitlab, Vault, Jenkins, etc. is providing a way for the pipeline to authenticate itself against the service. For Github that’s “https://token.actions.githubusercontent.com”, for gitlab “https://gitlab.com”, and some equivalent for other providers. The pipelines will ask their appropriate hosting to sign a token using their TLS certificate on the branch, actions, or the likes which can be passed to the cloud provider. At this point the cloud provider will verify the token (which allows you to match on the repo) and assume a role and if all things check out, you get a temporary session to act as that role.

Now for some code snippets:

To setup the Github IAM OIDC provider for AWS for usage by repo actions:

data "tls_certificate" "github" {
 url = “https://token.actions.githubusercontent.com”
}
 
resource "aws_iam_openid_connect_provider" "github" {
 url             = “https://token.actions.githubusercontent.com”
 client_id_list  = ["sts.amazonaws.com"]
 thumbprint_list = [data.tls_certificate.github.certificates.0.sha1_fingerprint]
}
 
data "aws_iam_policy_document" "assume-role-policy" {
 statement {
   actions = ["sts:AssumeRoleWithWebIdentity"]
   principals {
     type        = "Federated"
     identifiers = [aws_iam_openid_connect_provider.github.arn]
   }
   condition {
     test     = "StringLike"
     variable = "token.actions.githubusercontent.com:sub"
     values   = ["repo:pelotech/infrastructure:ref:refs/heads/main"]
   }
 }
}
resource "aws_iam_role" "github_ci" {
 name               = "GithubCI-OIDC-TF"
 description        = "GithubCI with OIDC"
 path               = "/ci/"
 assume_role_policy = data.aws_iam_policy_document.assume-role-policy.json
 managed_policy_arns = formatlist(
   "arn:%s:iam::aws:policy/%s",
   data.aws_partition.current.partition,
   ["AdministratorAccess"]
 )
}
output "ROLE_ARN" {
 description = "INFRA Role that needs to be assumed by github actions"
 value       = aws_iam_role.github_ci.arn
}

The only real changes that would need to take place here are the condition values. Replacing “pelotech” with your org name and “infrastructure” with your repo will allow actions on the main branch to assume the “GithubCI-OIDC-TF” role.

And here’s a sample .github/workflows/main.yaml for using this

name: apply-main
on:
 push:
   branches:
     - main
permissions:
 id-token: write
 contents: read
jobs:
 apply-terraform:
   runs-on: ubuntu-latest
   steps:
     - uses: actions/checkout@v2
     - name: Configure AWS credentials for sor account
       uses: aws-actions/configure-aws-credentials@v1
       with:
         role-to-assume: arn:aws:iam::{ACCOUNTID}:role/ci/GithubCI-OIDC-TF
         aws-region: us-west-2
     - name: Setup Terraform
       uses: hashicorp/setup-terraform@v1
       with:
         terraform_version: 1.3.2
     - name: Terraform Init
       run: terraform init
       env:
         TF_WORKSPACE: default
     - name: Terraform Plan
       run: terraform plan
       env:
         TF_WORKSPACE: default
     - name: Terraform Apply
       run: terraform apply -auto-approve
       env:
         TF_WORKSPACE: default

HOW TO BOOTSTRAP

So how do we solve the chicken and egg problem here for initial setup? In this case we need the IAM OIDC provider to exist before we can have a repo with actions configuring the account. What we currently use ourselves are a few trusted folks that can get temporary access to the AWS account(s) and manually runs locally the bootstrap of the IAM OIDC provider and base role for the root terraform. From there the root terraform takes over for all new limited role creation other repos and pipelines. This limits the “manual” tasks to a minimum and limits any permissions exposure.

MODULES TO USE

We’ve started the following project https://github.com/pelotech/terraform-aws-oidc-github to help with the bootstrap. Currently it’s just for github actions, however we’ll be creating the equivilant for gitlab, jenkins, and others.

With the Pelotech crew, we’re on the next versions of dev concepts which allow for complete self-hosted apps, deployments which are based on high-availability, high-scalability, and simple blue/green style deployments. Hit me up if you’re interested in more info about it!

Joachim Hill-Grannec @lindyblues is a Partner at http://www.pelo.tech, a group that helps organizations improve their dev practices and culture. These days you’ll also find him traveling around the world where ever there are waves to surf.