Managing secrets using sops/age

Store your secrets right in your git repo using sops / age for encryption and decryption.

DO NOT commit unencrypted secrets to Git.

Toolchain

We use sops and age to handle encryption of secrets.

Sops can be downloaded from https://github.com/getsops/sops/releases

Age can be installed from https://github.com/FiloSottile/age#installation

Age is pronounced with a hard G. Like Aghey.

Files

Secrets must be stored in text files, such as YAML or ENV files. In case the secret is managed via Kubernetes, the YAML is typically a manifest file representing a Secret. In case of a frontend application, the secrets are typically stored in a .env file format.

There should be a clear distinction between encrypted and unencrypted secret files. To make this distinction, all encrypted files should have the suffix .enc.yaml or .enc.env, where .enc signifies that the file is encrypted. Example: secrets.enc.yaml or stage.enc.env

To be very safe, is it better to add the plain names of the files to .gitignore so that no one accidentally commits an unencrypted file to git.

Keys

Before we begin to encrypt secrets, we need to create a keypair using age. To do this, run:

age-keygen

The output will look like this:

# created: 2023-05-08T15:38:39-04:00
# public key: age1xxxx
AGE-SECRET-KEY-yyyy

The public key, also known as identity, or recipient, looks like age1xxxx.

The secret key, or simply key, looks like AGE-SECRET-KEY-yyyy.

Store the output of the command (including comments) in a file /home/yourname/.config/sops/age/keys.txt(for Linux), or /Users/yourname/Library/Application Support/sops/age/keys.txt (for Mac)

If you use Bitwarden, it is a good idea to backup this in your vault. It is important you do this before moving forward. If you lose the key, it is NOT possible to recover it. (However, if you lose the public key, you can derive it from the secret key using age-keygen -y)

Encrpytion

Now, we will use sops to encrypt the secret file.

In case of YAML file:

sops -e -a age1xxxx secrets.yaml > secrets.enc.yaml

In case of ENV file:

sops -e -a age1xxxx stage.env > stage.enc.env

Always review the file before committing to git, making sure that the file contents are indeed encrypted.

Decryption

In your local laptop, if you need a edit a secret, you can use sops to directly edit the file in an editor like vi. Run the command:

sops secrets.enc.yaml

In your local laptop, if you want to dump the secret to your terminal to review, you can run:

sops -d secrets.enc.yaml

In order for this to work, you should have the key stored in your keys.txt file as described in earlier section.

Configuration

Consider a Kubernetes secret manifest that looks like this:

apiVersion: v1
kind: Secret
metadata:
    name: myapp-secrets
    namespace: myapp
stringData:
    AUTHORIZATION_USERNAME: edward
    AUTHORIZATION_PASSWORD: MargaretThatcherIs110%Sexy

Encrypt it using the command:

sops -e age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv secrets.yaml

And you will get this result:

apiVersion: ENC[AES256_GCM,data:Efc=,iv:SF0ZIFF+dVm//VweksahH/MIbJajj2TX82jwEooLFG0=,tag:ZJ36YUfz/4YEhaNJmkYZog==,type:str]
kind: ENC[AES256_GCM,data:arfGKcp4,iv:7OzQL9pCqyLTfib3t9OOfldGGxG0igAJi43Gx3q5okY=,tag:9Pjr6GsVMfIYxZ1oZIO2Xw==,type:str]
metadata:
    name: ENC[AES256_GCM,data:ErHyEow3CpuTtoq8tQ==,iv:11MFZgAHuPRcGDL0EWntDT14vM2ix8VKwOy914jXKB8=,tag:SkGF+LTUb4rRaHc1NlUVbg==,type:str]
    namespace: ENC[AES256_GCM,data:8YUTU0U=,iv:hAExMObv3gSXhojEhYcW3hEePEgNVGPkeeJzJIYjetY=,tag:Ez2A4yhXiRj3cMzuNDddzw==,type:str]
stringData:
    AUTHORIZATION_USERNAME: ENC[AES256_GCM,data:PWRGPGHk,iv:P6fXYtS+6ETypBOEcgoNjxakcoYOjwcxqStZDJEU7pM=,tag:a5anLYBaKGjsL8dm3o1ovA==,type:str]
    AUTHORIZATION_PASSWORD: ENC[AES256_GCM,data:3BMUm60XEqoOfHQyif7zTm/s5ZMOnRtoIj4=,iv:DwP5UCDjYONqE7ShI9wM98KD+qo5/YFp01/f/GQznhc=,tag:vJvGS7Rym058/VzODtR+ig==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlMTh0aFRaQ2JjSWtjNHFt
            Q0VLRmJoVm0vMjBTdThSTFRjT3dlWDZDR3lrCjllZ2l5eUJ3bC9iZUEyRU1LSWdk
            OU9ZdTM1K0NHT09LK2YxTjNoNTRDRjAKLS0tIHA5UWh3cVlHNGpWWXBqUm11UFNi
            ZWZsb2RQb2NyY1h4YzZNOE1tempvc2sKgZX1kiUuImKMJa0bTdoWLmiLIrABTDHf
            ZLOi2RUTOh/Ui+mfV1dcYrUV6SKJvlGwwRZEeuuBrG6Jbd5W32A6OQ==
            -----END AGE ENCRYPTED FILE-----            
    lastmodified: "2024-05-21T19:42:28Z"
    mac: ENC[AES256_GCM,data:sHZMYFmP6/HvLGz0LqMQ2kWVvvWVS+K01IoaqeXSkyE3nR9Qi5JJJ6aeRMjfhDygBMGw2P+0zS/6KcPBoeQpvKWbWZFAyWcTZJIx+iI8bIsWivtMOlNydE61C+0xewVIvej3AyKkSyxdSbuDO/1/rLg9KXjkJJ58TtYxz+jk8MA=,iv:eM9viOEs61XQ0f4mkgoHsAi9bxtRUHSQ1IgiPIslAW4=,tag:1iKWK7FlKpzFBH/6wr3AJg==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.8.1

Notice that all keys in the yaml including non-sensitive ones like apiVersion, kind and metadata are also encrypted. This may be underisarable when doing code reviews or otherwise looking at the encrypted file.

To avoid this, create a .sops.yaml (this file is usually placed in the root folder of your repo):

---
creation_rules:
  - path_regex: secrets.yaml
    age: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
    encrypted_regex: ^(data|stringData)$

Now, you can encrypt without specifying any identities:

sops -e secrets.yaml

And you will get this response:

apiVersion: v1
kind: Secret
metadata:
    name: myapp-secrets
    namespace: myapp
stringData:
    AUTHORIZATION_USERNAME: ENC[AES256_GCM,data:Lyj81idG,iv:w77ofABHiYHyup8z/bUMqFbnZ1oL82H1gbgQ2m2ixRQ=,tag:8IIQi4hWgiGRfRP6jYWxRw==,type:str]
    AUTHORIZATION_PASSWORD: ENC[AES256_GCM,data:8Bgfii+UiFTGURyFSvJZ4RZuzRVgMG33RFM=,iv:ToBjzFzrGCbpauXYfa3PNDuxTGerUfmmtNovWah+rNs=,tag:hkTNGymGTtPo6O5q823LQw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCN2toTmFFMS9FdVduNFBU
            eWRKWmcvSnE3Wk8wVzlUZTN4QVdiRUlZSzFZCnlWNytrZTBJcjMzcklWVnFNb3k3
            YjRCdE52dDhiYk1WdHkxNXoxZlZraDQKLS0tIGVCNFdIV0ZuREtjU3NlM0dCcVhn
            QUtwN3BmUGNwYldabTcyQ1dvc1FYTkkKvl41B3MxZVER+BHBGiQJPQW62cm8TsiK
            TIYIqVcJ4HpKdCe6t2KFVnfDHwVJ1BbarUOMGWjMfwaLMpOlB6fRgQ==
            -----END AGE ENCRYPTED FILE-----            
    lastmodified: "2024-05-21T19:49:05Z"
    mac: ENC[AES256_GCM,data:aJAVVBeh0u8o02QH+hnBfewYnVlt+omOQY+IXF7FixWDNtNTRBwMb0JbqAt25rda3P7zELgRd1bfDnClFY7JadolIFN1iwPi8w/hQSv5Vp32N259ukpzxpWozT7Xl7CkR1XaOMvXTTQw2Uv4RfdyyrTxIlPAwsjCfShbiVjFf2k=,iv:niizOhZ9TeEWr1Vms+h/F6wubQZND197q6XpAMMqa7k=,tag:XxAsF1n8nQFtNepf7ePwsg==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.1

As you will note, only the keys under stringData are encrypted.

CI/CD pipelines

For decrypting secrets in CI/CD pipeline, run:

export SOPS_AGE_KEY=AGE-SECRET-KEY-yyyy
sops -d secrets.enc.yaml > secrets.yaml

You can also store the secret key in a Github actions env var and use that here. And finally, you can use the decrypted secrets.yaml in a kubectl apply -f command.