Links

HashiCorp Vault Secrets Manager Setup

HashiCorp Vault Server

Ansible playbooks v2 support storing secrets in HashiCorp Vault server in its kv v2 (key/value version 2) storage backend.
You can either use your existing self-hosted or SaaS hosted HashiCorp Vault server or you can setup a new self-hosted server.

HashiCorp Vault Self-Hosted Server Setup

HashiCorp Vault server administration is a world for itself, so the following is a quick and and a bit dirty setup guide.
Prerequisities:
  • a Linux server
    • reachable via public IP
    • with free and open ports 80 and 443
    • with docker, docker-compose, Python3 installed
  • a domain to point to the server IP (can be 3rd level domain, like vault.example.com)
  • ansible-playbooks and ansible-private repositories available locally
  • docker installed locally for running an Ansible playbook
  • the server record in ansible-private/infrastructure/hosts.ini under group deploy_machines.
  • Existing secrets management storage (LastPass or plaintext files) to access Vault server sudo password (same way you are accessing your portal passwords). Example from hosts.ini: [deploy_machines] depl ansible_host=depl.example.com [deploy_machines:vars] secrets_storage="lastpass" ansible_user=user lastpass_portal_credentials_server="Shared-Server Credentials/{{ ansible_user + '@' + ansible_host }}"
Configure and start the Vault server:
  • Edit my-vars/hashicorp-vault-config.yml in ansible-playbooks repository
    • Copy my-vars/hashicorp-vault-config.sample-do-not-edit.yml to my-vars/hashicorp-vault-config.yml
    • Set Vault server domain
    • Set your email for SSL
  • Run Ansible playbook to configure and start Vault server
    • scripts/infrastructure-hashicorp-vault-install.sh -e my-vars/hashicorp-vault-config.yml --limit depl
  • Record root token and 5 unseal keys
Unseal the Vault:
  • Each time the Vault is (re-)started it is sealed to prevent data theft
  • Navigate to your Vault in the browser (e.g. vault.example.com), (you should be automatically redirected from http to https).
  • Unseal the Vault with any 3 of the 5 unseal keys you have recorded earlier
Enable kv v2 (key/value version 2) storage engine:
  • In Vault navigate to Secrets
  • Click Enable new engine
  • Select KV, click Next
  • Optionally: Set Maximum number of versions
  • Click Enable Engine
As a result new KV version 2 storage engine is created.

HashiCorp Vault Secrets Configuration

Following is just one of possible ways to configure access to your secrets in HashiCorp Vault. In larger Vault setup you might want to employ Vault groups with their own policies etc., but the following is sufficient to get you going as a small team.
Create a policy to access secrets:
  • Login with your root token
  • Navigate to Policies
  • Click Create ACL policy
  • Enter policy Name: skynet-ansible-admins
  • Enter Policy: path "kv/*" { capabilities = ["read", "list"] }
    path "kv/data/ansible-skynet/*" { capabilities = ["create", "read", "update", "delete", "list"] }
    path "kv/metadata/ansible-skynet/*" { capabilities = ["create", "read", "update", "delete", "list"] } # Enable to change own user password
    path "auth/userpass/users/{{identity.entity.aliases.auth_userpass_e8a80ebc.name}}" { capabilities = [ "read", "update" ] allowed_parameters = { "password" = [] } }
  • Click Create policy
Create audit policy:
  • Navigate to Policies
  • Click Create ACL policy
  • Enter policy Name: audit-policy
  • Enter Policy: # 'sudo' capability is required to manage audit devices
    path "sys/audit/*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"] } # To list enabled audit devices, 'sudo' capability is required
    path "sys/audit" { capabilities = ["read", "sudo"] } }
  • Click Create policy
Enable audit logs:
  • In terminal on your server login to Vault (in the prompt use root token recorded earlier): docker exec -it vault vault login
  • Enable Vault audit logs: docker exec vault vault audit enable file file_path=/vault/logs/vault_audit.log
  • Logout by deleting your saved token: docker exec vault rm /root/.vault-token
Enable userpass authentication:
  • Navigate to Access
  • Click Enable new method
  • Select Username & Password
  • Click Next
  • Click Enable Method
Create Vault user(s):
  • Navigate to Vault in your browser
  • Click on terminal icon (top right)
  • Paste the following command to the terminal (replace {username} and {password} with desired values) and hit Enter vault write auth/userpass/users/{username} \ password={password}
  • Repeat for each user
Assign policy to each Entity (user):
  • Navigate to Access
  • Click Entities
  • Search for user via his userpass login
    • Select Lookup by alias name
    • Select userpass
    • Enter user login
  • On the found Entity click Aliases to check login user
  • Click Edit entity
  • In Polices section add skynet-ansible-admins policy
  • Click Save

Backup Self-hosted HashiCorp Vault to AWS S3

When you self host HashiCorp Vault, you should create backups, so you do not loose portal secrets if anything happens to your Vault or your server hosting so you can safely restore your data.
Below we present one way to backup your Vault. The backup consist of a cron job to scan Vault for any new or updated secrets in Vault `kv v2` backend storage. If there is newer or updated secret found than is the latest backup, the script creates a Vault snapshot and uploads the backup to AWS S3 bucket.
NOTE:
Updates to other Vault data other then `kv v2` secrets (e.g. new/updated Vault policies, new users, updated user passwords, etc.) will not trigger the automated backup. You should either trigger the backups manually or update the backup script to handle also those.
First create kv-backup-policy policy with the following content:
# Listing directories and secrets, but not reading their values path "kv/*" { capabilities = ["list"] } # Reading secrets metadata (we need to read update times) path "kv/metadata/*" { capabilities = ["list", "read"] } # Creating Vault snapshots for raft (= Integrated Storage) path "/sys/storage/raft/snapshot" { capabilities = ["read"] }
Enable AppRole in Vault:
  • Login to Vault web UI with root token
  • Enable AppRole: Access > Auth Methods > Enable new method: AppRole > Next
Create actual backup role:
  • In web UI open Vault terminal (terminal icon on top right)
  • Execute in terminal: vault write auth/approle/role/kv-backup-role \ secret_id_ttl=0 \ token_num_uses=0 \ token_ttl=20m \ token_max_ttl=30m \ secret_id_num_uses=0 \ token_policies=kv-backup-policy
Get role ID:
  • In web UI terminal: vault read auth/approle/role/kv-backup-role/role-id
  • Write down role ID, you will need it later for setting environment variables.
Create secret ID:
  • In web UI terminal: vault write -f auth/approle/role/kv-backup-role/secret-id
  • Write down Secret-ID, Secret-ID-Accessor.
Set environment variables for backup script:
  • Create .env file in the directory you have setup HashiCorp Vault by default: /home/user/hashicorp-vault/.env
  • set values for all of the below variables
  • role/secret IDs you should get already from running above scripts
  • HCV kv v2 path should be kv if you have used our default installation
  • AWS values you should use according to your AWS setup HCV_BACKUP_ROLE_ID= HCV_BACKUP_SECRET_ID= HCV_KV_V2_PATH= AWS_S3_BUCKET= AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY=
Copy the backup script:
Setup cron job:
  • run: crontab -e default crontab editor opens
  • to start edit mode press: i
  • paste cronjob entry (to run every 5 minutes): */5 * * * * /home/user/hashicorp-vault/backup-hashicorp-vault.sh
  • exit and save:
    • Escape
    • :wq
    • Enter
Check AWS S3 bucket for the backup:
  • Cron job now runs the Vault job to scan and possibly backup the Vault every 5 minutes.
  • So soon you should see your first Vaul backup in your AWS S3 bucket.

HashiCorp Vault Configuration for Playbooks

Ansible v2 playbooks can connect to self-hosted (setup above) or SaaS hosted Vault.
The playbooks are tested and working with KV v2 (key/value version 2) Vault backend storage.
Configuration: