Sync secrets from Vault to GCP Secret Manager

The Google Cloud Platform (GCP) Secret Manager sync destination allows Vault to safely synchronize secrets to your GCP projects. This is a low footprint option that enables your applications to benefit from Vault-managed secrets without requiring them to connect directly with Vault. This guide walks you through the configuration process.

Prerequisites:

• Ability to read or create KVv2 secrets
• Ability to create GCP Service Account credentials with access to the Secret Manager
• Ability to create sync destinations and associations on your Vault server

Setup

1. If you do not already have a Service Account, navigate to the IAM & Admin page in the Google Cloud console to create a new Service Account with the necessary permissions. Instructions to provision this Service Account via Terraform can be found below.

2. Configure a sync destination with the Service Account JSON credentials created in the previous step. See docs for alternative ways to pass in the credentials parameter.

   $ vault write sys/sync/destinations/gcp-sm/my-dest \
       credentials='@path/to/credentials.json'
       replication_locations='us-east1,us-west1'
   

   Copy

   Output:

   Key                   Value
   ---                   -----
   connection_details    map[credentials:***** replication_locations:us-east1,us-west1]
   name                  my-dest
   type                  gcp-sm
   

Usage

1. If you do not already have a KVv2 secret to sync, mount a new KVv2 secrets engine.

   $ vault secrets enable -path=my-kv kv-v2
   

   Copy

   Output:

   Success! Enabled the kv-v2 secrets engine at: my-kv/
   

2. Create secrets you wish to sync with a target GCP Secret Manager.

   $ vault kv put -mount=my-kv my-secret foo='bar'
   

   Copy

   Output:

   ==== Secret Path ====
   my-kv/data/my-secret
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       <timestamp>
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   

3. Create an association between the destination and a secret to synchronize.

   $ vault write sys/sync/destinations/gcp-sm/my-dest/associations/set \
       mount='my-kv' \
       secret_name='my-secret'
   

   Copy

   Output:

   Key                   Value
   ---                   -----
   associated_secrets    map[kv_1234/my-secret:map[accessor:kv_1234 secret_name:my-secret sync_status:SYNCED updated_at:<timestamp>]]
   store_name            my-dest
   store_type            gcp-sm
   

4. Navigate to the Secret Manager in the Google Cloud console to confirm your secret was successfully created in your GCP project.

Moving forward, any modification on the Vault secret will be propagated in near real time to its GCP Secret Manager counterpart. Creating a new secret version in Vault will create a new version in GCP Secret Manager. Deleting the secret or the association in Vault will delete the secret in your GCP project as well.

Replication policy

GCP can target specific geographic regions to provide strict control on where your applications store data and sync secrets. You can target specific GCP regions for each sync destinations during creation which will limit where Vault writes secrets.

Regardless of the region limits on writes, synced secrets are always readable globally when the client has the required permissions.

Permissions

The credentials given to Vault must have the following permissions to synchronize secrets:

secretmanager.secrets.create
secretmanager.secrets.delete
secretmanager.secrets.update
secretmanager.versions.add
secretmanager.versions.destroy 

Provision service account

Vault needs to be configured with credentials to establish a trust relationship with your GCP project so it can manage Secret Manager secrets on your behalf. The IAM & Admin page in the Google Cloud console can be used to create a new Service Account with access to the Secret Manager.

You can equally use the Terraform Google provider to provision a GCP Service Account with the appropriate policies.

1. Copy-paste this HCL snippet into a secrets-sync-setup.tf file.

   provider "google" {
       // See https://registry.terraform.io/providers/hashicorp/google/latest/docs#authentication-and-configuration to setup the Google Provider
       // for options on how to configure this provider. The following parameters or environment
       // variables are typically used.
   
       // Parameters
       // region = "" (Optional)
       // project = ""
       // credentials = ""
   
       // Environment Variables
       // GOOGLE_REGION (optional)
       // GOOGLE_PROJECT
       // GOOGLE_CREDENTIALS (The path to a service account key file with the
       //                    "Service Account Admin", "Service Account Key Admin",
       //                    "Secret Manager Admin", and "Project IAM Admin" roles
       //                    attached)
   }
   
   data "google_client_config" "config" {}
   
   resource "google_service_account" "vault_secrets_sync_account" {
     account_id  = "gcp-sm-vault-secrets-sync"
     description = "service account for Vault Secrets Sync feature"
   }
   
   // Production environments should use a more restricted role.
   // The built-in secret manager admin role is used as an example for simplicity.
   data "google_iam_policy" "vault_secrets_sync_iam_policy" {
     binding {
       role = "roles/secretmanager.admin"
       members = [
         google_service_account.vault_secrets_sync_account.email,
       ]
     }
   }
   
   resource "google_project_iam_member" "vault_secrets_sync_iam_member" {
     project = data.google_client_config.config.project
     role    = "roles/secretmanager.admin"
     member  = google_service_account.vault_secrets_sync_account.member
   }
   
   resource "google_service_account_key" "vault_secrets_sync_account_key" {
     service_account_id = google_service_account.vault_secrets_sync_account.name
     public_key_type    = "TYPE_X509_PEM_FILE"
   }
   
   resource "local_file" "vault_secrets_sync_credentials_file" {
     content  = base64decode(google_service_account_key.vault_secrets_sync_account_key.private_key)
     filename = "gcp-sm-sync-service-account-credentials.json"
   }
   
   output "vault_secrets_sync_credentials_file_path" {
     value = abspath("${path.module}/${local_file.sync_service_account_credentials_file.filename}")
   }
   

   Copy

2. Execute a plan to validate the Terraform Google provider is properly configured.

   $ terraform init && terraform plan
   

   Copy

   Output:

   (...)
   Plan: 4 to add, 0 to change, 0 to destroy.
   

3. Execute an apply to provision the Service Account.

   $ terraform apply
   

   Copy

   Output:

   (...)
   Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   sync_service_account_credentials_file = "/path/to/credentials/file/gcp-sm-sync-service-account-credentials.json"
   

The generated Service Account credentials file can then be used to configure the Vault GCP Secret Manager destination following the setup steps.

Targeting specific GCP projects

By default, the target GCP project to sync secrets with is derived from the service account JSON credentials or application default credentials for a particular GCP sync destination. This means secrets will be synced within the parent project of the configured service account.

In some cases, it's desirable to use a single service account or workload identity to sync secrets with any number of GCP projects within an organization. To achieve this, you can set the project_id parameter to the target project to sync secrets with:

  $ vault write sys/sync/destinations/gcp-sm/my-dest \
      project_id='target-project-id'

This overrides the project ID derived from the service account JSON credentials or application default credentials. The service account must be authorized to perform Secret Manager actions in the target project.

Access management

You can allow or restrict access to secrets based on IAM conditions against the fully-qualified resource name. For secrets in Secret Manager, a fully-qualified resource name must have the following format:

projects/<project_number>/secrets/<secret_name>

For example, the default secret name template prepends the word vault to the beginning of secret names. To prevent Vault from modifying secrets that were not created by a sync operation, you can use a role binding against the resource name with the startsWith condition:

resource.name.startsWith("projects/<project_number>/secrets/vault")

To prevent out-of-band overwrites, simply add a negative condition with ! on any write-access role bindings not being used by Vault that contain Secret Manager permissions:

!(resource.name.startsWith("projects/<project_number>/secrets/vault"))

To add conditions to IAM principles in GCP, click "+ADD IAM CONDITION" on the Assign Roles screen.

API

Please see the secrets sync API for more details.