Pen, Diary and Glasses, Generation Bass, CC BY 2.0, via flickr
kubernetes talos

Talos Kubernetes - External Secrets and Bitwarden Secrets Manager

Goal

Deploy a custom app to get and cache secrets from a cloud secrets manager (I use Bitwarden Secrets Manager, it has a free tier). I use external-secrets together with bws-cache to cache and handle secrets needed by other apps in my cluster. Checkout that project’s readme to learn more about how it works.

Prerequisite

Talos Kubernetes - Post Install Details (Split DNS)

Instructions

  1. Clone my repo and navigate to /kubernetes/apps in a file explorer or terminal.
  2. Copy the external-secrets folder into your project’s /apps directory.
  3. Navigate to /apps/external-secrets/external-secrets/bitwarden-secrets-manager.
  4. Delete the file secret.sops.yaml: rm secret.sops.yaml.
  5. We are going to create and encrypt our own. Create a new file secret.sops.yaml with touch secret.sops.yaml
  6. Type in or paste in the following:
apiVersion: v1
kind: Secret
metadata:
    name: bws-secret
stringData:
    token:
    ORG_ID:

Important: Do NOT commit this file unless you have verified it has been successfully encrypted by running sops --encrypt --in-place secret.sops.yaml in the following steps. First we need to setup Bitwarden to get our secret access token (token) and organization id (ORG_ID). 6. Create and setup a Bitwarden organization if you need one by following the instructions in Bitwarden’s Organizations Quick Start (It’s free). 7. Create a project and give it a name that identifies it with your cluster. 8. Enable the Secrets Manager. There should be a free option. 9. From within your organization, go to Machine Accounts and create a new access token. Copy the token and paste it into the token: parameter within secret.sops.yaml. 10. Create a secret and assign it to the project you created. Name it Test_secret and put whatever you want as the value. We’ll retrieve this value at the end to verify everything works. 11. You can get your BWS organization ID two ways:

  • From BWS CLI:
    • bws project list / bws project get <project_id> - Your organization ID is shown in the organizationId value of each project returned.
    • bws secret list / bws secret get <secret_id> - Your organization ID is shown in the organizationId value of each secret returned.
  • From browser:
    1. Go to https://vault.bitwarden.com
    2. Open Secrets Manager from the apps list in the top right
    3. Your organization ID is in the URL like this: https://vault.bitwarden.com/#/sm/<BWS org ID> Copy and paste the organization id after the ORG_ID: parameter within secret.sops.yaml.
  1. Save the file, and then from the same directory run: sops --encrypt --in-place secret.sops.yaml and then save it again. Now when you click on secret.sops.yaml in VS Code it should open the decrypted .gitignored file in the editor named .decrypted-secret.sops.yaml (Note: You may need to install the vscode-sops extension into the dev container if this isn’t working). If you select secret.sops.yaml again, you should see that it has been encrypted by sops. Important: Make sure secret.sops.yaml is encrypted before committing.
  2. Commit your changes when you’re ready and wait for flux to deploy everything. Be patient, it may be a few minutes before everything is ready. Check the status with: task kubernetes:resources. The ingress host bws-cache.${SECRET.DOMAIN} should be assigned to the internal ingress VIP address shortly.
  3. Once everything is ready, check that the secret has been pulled in from Bitwarden with: kubectl get secret test-secret -n external-secrets -o jsonpath='{.data.data}' | base64 --decode. The plaintext value you set in step 10 should be printed.

Tips

Fix issues with pulling images:

  1. See the official Kubernetes docs on how to Pull an Image from a Private Registry. It may be necessary to create a secret in the namespace where the pod is being deployed which contains your docker credentials. Follow the instructions in the official docs above to login to docker and create a secret containing your credentials. If you use a Docker credential store have logged in to Docker using a github PAT as the docker password, the command to create the secret could look something like this:

      kubectl create secret docker-registry regcred --docker-server=https://ghcr.io --docker-username=<your-name> --docker-password=<your-github-PAT> --docker-email=<your-email>

  2. The previous command only creates the secret. To get pods to use the secret, you can Add image pull secret to service account. Service accounts are used to apply role based access control (RBAC) to different resources. When we add an image pull secret to the default service account, any pod which uses the default service account will gain access to the secret. Since pods deployed to the default namespace will use the default service account by default, new pods in the default namespace will automatically gain access to the image pull secret. Modify the default service account (official command escaped for windows support):

     kubectl patch serviceaccount default -p "{\"imagePullSecrets\": [{\"name\": \"regcred\"}]}"

  3. Delete any stuck pods. If the problem persists, make sure your credentials are correct and have not expired. Once the registry credential secret is working in the default service account in the default namespace, we can configure the default service account in other namespaces to also use the regcred secret we created. Then, pods deployed in other namespaces will also be able to use your Docker credentials to pull images from private registries. For example, adding the regcred secret to the default service account in the external-secrets namespace can be done by adding -n external secrets to the previous command:

     kubectl patch serviceaccount default -p "{\"imagePullSecrets\": [{\"name\": \"regcred\"}]}" -n external-secrets

It can be helpful to manually reset the cache when you want a secret to be updated. The below commands still work, but are outdated. You can now clear the cache by simply deleting the bws-cache pod.

  1. First, update the secret’s value in the Bitwarden Secrets Manager.
  2. Then, expose the service so we can curl the /reset endpoint. This can be done by port-forwarding the bws-cache service like so: kubectl port-forward service/bws-cache -n external-secrets 5000:5000
  3. In another terminal, run: curl -H "Authorization: Bearer <BWS token>" http://localhost:5000/reset.
  4. Rerun kubectl get secret test-secret -n external-secrets -o jsonpath="{.data.data}" | base64 --decode.
  5. The updated value should be printed in the terminal output.

Next

Use this secrets caching mechanism to deploy metrics, logging, and alerts to monitor cluster health, metrics, and logs.