Build a Secure CI/CD Pipeline for Kubernetes Using GitHub Actions and Hashicorp Vault

In this tutorial, you will learn how to use Hashicorp's Vault to build CI/CD pipelines in GitHub Actions to securely build, push, and deploy Docker images to Kubernetes.

You will see how Vault can be used to securely store pipeline secrets like access tokens, API keys, etc. You will then configure a CI/CD pipeline using GitHub Actions to access these secrets stored in Vault and deploy the changes to a Kubernetes cluster. The Vault GitHub action gives the ability to pull secrets from Vault in GitHub Action workflows. This tutorial uses Helm to manage the Kubernetes application.

Objectives:

In this tutorial, you will go through the following steps:

1

Store secrets on Vault

Store sensitive pipeline secrets on Vault.

2

Authenticate GitHub to access Vault

Create Vault token and Vault policy used by GitHub Actions to authenticate to Vault.

3

Configure GitHub Actions

Configure GitHub Actions to use secrets stored in Vault and orchestrate K8s deployments.

4

Test the CI/CD pipeline

Once GitHub Action workflow is configured, trigger the Action and test if the Action works as desired.

5

Troubleshooting

Troubleshoot common issues and errors encountered while using Vault and GitHub Actions.

Prerequisites

Before starting this tutorial, you will need:

• A Kubernetes cluster.

Need a new cluster?

Try our guide <link> to running confidential K8s cluster using enclaive Dynemees.

• kubectl - installed locally and configured to point to your K8s cluster.

• A Vault environment with Vault CLI - self-hosted or HCP Vault Dedicated.

• jq CLI for processing JSON

• git CLI - installed and configured.

• A GitHub account for running GitHub Actions.

• Fork and clone this repo

---

Getting Started

Store secrets on Vault

Before you are able to store/retrieve Vault secrets, make sure your Vault is unsealed. Find out more about sealing/unsealing Vault here.

The GitHub Actions workflow deployed later reads 2 secrets defined at secret/data/<secret_path>. You need to create these Vault key-value secrets, a Vault policy defined to access the secrets, and a Vault token to retrieve the secrets.

Secret Path	Key	Description
token	registry_token	GitHub Personal Access Token to access GitHub Container Registry for pushing and pulling Docker images. This will be passed to the GitHub Action to push images to GCR.
config	kubeconfig	Contents of your Kubeconfig - containing connection details of an existing K8s cluster. This will be passed to the GitHub Action to connect to K8s cluster.

Creating GitHub Personal Access Token

You will need a GitHub Personal Access Token, with write:packages scope, which will be passed to the GitHub Action. To create a GitHub Personal Access token, follow these steps. Make sure to copy your GitHub Personal Access Token so you can store it in Vault.

1. Create the registry_token key-value secret at secret/token path using Vault KV Secrets Engine .

$ vault kv put secret/token registry_token="Github Personal Token you created"

1. Similarly, create the kubeconfig key-value secret at secret/config path.

$ vault kv put secret/config kubeconfig="Your Kubeconfig"

Avoid Kubeconfig configuration with credentials belonging to a cluster-admin user as this would allow a compromised workflow to perform any action on your cluster.

1. Verify the secrets were created.

$ vault kv list secret/

1. In the sample repository, you will find policy.hcl file. You will use this file to create a new Vault policy. This policy grants read capability for paths where the secrets are stored. The policy will be applied to the auth token used by GitHub Action to access the secrets stored in Vault.

policy.hcl

path "secret/data/kubeconfig" {
    capabilities = ["read"]
}

path "secret/data/registry" {
    capabilities = ["read"]
}

1. Export an environment variable GITHUB_REPO_TOKEN to capture the token value created using vault token create with the ci-secret-reader (policy name) policy attached. You will assign this token to the GitHub Repository in the next section.

$ GITHUB_REPO_TOKEN=$(vault token create -policy=ci-secret-reader policy.hcl -format json | jq -r ".auth.client_token")

Authenticate GitHub to access Vault

In this step, you will be using a Vault token and store it as a GitHub repository secret for GitHub Action to authenticate to Vault.

More auth methods

The Vault GitHub Action supports several different authentication methods.

1. To create a repository secret in your GitHub repository, follow these steps.
2. In the Name field of the repository secret, type VAULT_TOKEN .

You will use the repository secret defined here in the GitHub Actions Workflow so make sure you are using the same name in your GitHub Actions Workflow as in the repository secret.

1. From your terminal, copy the token value stored in the GITHUB_REPO_TOKEN variable created earlier and paste that as the value for the VAULT_TOKEN secret. Then click Add Secret.

Now you have configured a GitHub repository with a valid token that can read secrets from Vault server.

Configure GitHub Actions

In the sample repository, you will find .github/workflows/kubernetes.yaml file which defines a GitHub Action workflow. Below are the contents of the file:

.github/workflows/kubernetes.yaml

on:
  push:
    branches:
      - main
jobs: 
  build: 
    runs-on: self-hosted
    permissions:
      packages: write
    steps: 
    - name: Import Secrets
      uses: hashicorp/vault-action@v2
      with:
          url: http://127.0.0.1:8200
          tlsSkipVerify: true
          token: ${{ secrets.VAULT_TOKEN }}
          exportEnv: true
          secrets: |
            secret/data/token registry_token | REGISTRY_TOKEN;
            secret/data/config kubeconfig | KUBECONFIG;
    - name: Login to GHCR
      uses: docker/login-action@v3
      with: 
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ env.REGISTRY_TOKEN }}
    - name: Build and push the Docker image
      uses: docker/build-push-action@v3
      with:
        push: true
        tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
    
  deploy:
    runs-on: self-hosted
    needs: build
    steps:
      - name: Import Secrets
        uses: hashicorp/vault-action@v2
        with:
          url: http://127.0.0.1:8200
          tlsSkipVerify: true
          token: ${{ secrets.VAULT_TOKEN }}
          exportEnv: true
          secrets: |
            secret/data/token registry_token | REGISTRY_TOKEN;
            secret/data/config kubeconfig | KUBECONFIG;
      - name: Checkout repository
        uses: actions/checkout@v4
      - name: Configure Kubeconfig
        uses: azure/k8s-set-context@v4
        with:
          method: kubeconfig
          kubeconfig: ${{ env.KUBECONFIG }}  
      - name: Deploy the Helm chart
        run: |
          helm upgrade --install node-js-app ./helm --create-namespace --namespace ${{ github.event.repository.name }} --set image=ghcr.io/${{ github.repository }}/${{ github.sha }} --set dockerConfigJson.data='\{\"auths\":\{\"ghcr.io\":\{\"username\":\"${{ github.actor }}\"\,\"password\":\"${{ env.REGISTRY_TOKEN }}\"\}\}\}'

This GitHub Action is running on a self-hosted runner, indicated by runs-on: self-hosted on line 8. You can alternatively also use a GitHub-Hosted runner.

The above workflow contains 2 jobs:

build job

This job builds and pushes the sample app's Docker image to GitHub Container Registry. To push the Docker image to GHCR, you will use Docker login-action. This action requires username and a password to log-in to a container registry (the example uses GHCR, but you ca use any other container registry). For password you will need a GitHub Personal Access Token (PAT).

You created a GitHub PAT earlier and stored it as registry_token secret in Vault. This job will use that secret, stored in Vault, to login to GHCR and push images to it.

deploy job

This job deploys the sample NodeJS app to the Kubernetes cluster defined in the Kubeconfig passed to the kubeconfig secret key in Vault. This job uses Azure k8s-set-context action for setting context and retrieving Kubeconfig before deploying to K8s.

The sample app uses Helm to apply K8s manifests declaratively to the K8s cluster. Helm chart will deploy a Deployment (the NodeJS app container) and a LoadBalancer Service to expose the NodeJS application.

Similar to the build job, thedeploy job also uses the registry_token secret in Vault to retrieve the Docker image stored in GHCR.

For the GitHub Action to extract secrets from Vault, you will use the vault-action step. In the sample GHA workflow, the step is configured as follows:

.github/workflows/kubernetes.yaml

steps: 
    - name: Import Secrets
      uses: hashicorp/vault-action@v2
      with:
          url: http://127.0.0.1:8200
          tlsSkipVerify: true
          token: ${{ secrets.VAULT_TOKEN }}
          exportEnv: true
          secrets: |
            secret/data/token registry_token | REGISTRY_TOKEN;
            secret/data/config kubeconfig | KUBECONFIG;

The Import Secrets step is used in both build and deploy jobs as both the jobs require access to secrets stored in Vault. The example step is authenticating GitHub to access Vault running locally at http://127.0.0.1:8200 . You can change this to wherever your vault is running. For example:

.github/workflows/kubernetes.yaml

    - name: Import Secrets
      uses: hashicorp/vault-action@v2
      with:
          url: https://vault.example.com:8200

Once GitHub is authenticated to Vault, you can fetch secrets from the Vault server. The secrets section of the Import Secrets step defines the path where the secrets are stored ( secret/data/token, secret/data/config ) and the key to extract the secrets - registry_token and kubeconfig. By default, this secret value gets exported to GitHub environment variables, REGISTRY_TOKEN and KUBECONFIG, that is useful to the steps that follow.

Test the CI/CD pipeline

Once the workflow has been configured, you can now test the pipeline by performing a source code change as the GitHub Action is configured to run every time a push is made to the main branch.

Trigger the GitHub Action

The first step to test the CI/CD pipeline is triggering the GitHub Action. To trigger the GitHub Action, edit the main.js file. Delete the old contents from main.js and paste the below contents:

main.js

const express = require('express');
const app = express();

app.get("*", (req, res) => {
    res.send("<h1>Hello from Vault!</h1>")
});

app.listen(80, () => {
    console.log("App has started");
});

Next, commit your changes, and then push them to GitHub:

Command Line

a. Stage all modified file:

$ git add .

b. Commit the changes to the local repository:

$ git commit -m "Edit main.js"

c. Push the changes to GitHub:

$ git push origin main

The GitHub-hosted or self-hosted runner polls GitHub for changes, and executes the runner upon detecting changes. Navigate to GitHub's Actions tab and you'll see a new workflow run beginning. After a few seconds, it should show as successfully completed. If your action has failed, refer to the troubleshooting section.

Verifying Deployment to Kubernetes

You will use kubectl to verify the Deployment and the LoadBalancer Service have been deployed and are ready to receive traffic.

Command Line

a. Check K8s Deployments

$ kubectl get deployment -n <Your GitHub Repository Name>
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
<Your GitHub Repository Name>   3/3     3            3           1h

b. Check K8s LoadBalancer Service and obtain the external-IP

$ kubectl get svc -n <Your GitHub Repository Name>
NAME                     TYPE           CLUSTER-IP          EXTERNAL-IP          PORT(S)        AGE
<Your GitHub Repository Name>   LoadBalancer   <some cluster IP>   <some external IP>   80:30421/TCP   1h

The sample app used Helm to dynamically describe name and namespace for the Deployment and the LoadBalancer Service. Make sure to use the name of your GitHub repository where indicated.

Send a curl request to the external IP to verify your app has been successfully deployed:

Command Line

$ curl <external-IP>
StatusCode        : 200
StatusDescription : OK
Content           : <h1>Hello from Vault!</h1>
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    Keep-Alive: timeout=5
                    Content-Length: 105
                    Content-Type: text/html; charset=utf-8

You can make another change to your repository which will trigger the GitHub Action. A new run of the GitHub Action will start and deploy the changes to your K8s cluster.

Troubleshooting

If your GitHub Action run is failing, below are a few things to check:

• Vault is up and running at the address provided in the GitHub Action workflow.

• Vault is unsealed. Read more about sealing/unsealing Vault.
• Secrets are defined in the right path in Vault. Read more about KV Secret Engine API.
• Check for the correctness of Vault token, GitHub Personal Access Token and your Kubeconfig.

Conclusion

One common challenge with GitHub Actions is managing pipeline secrets securely, especially when workflows need access to sensitive data for continuous deployment. In this tutorial, you configured a secured CI/CD pipeline using GitHub Actions and Vault to deploy securely to Kubernetes. Using Vault to store sensitive secrets enhances security while ensuring workflows have the access they need to perform tasks effectively.