When Vault Migration Is Really a Cryptography and Trust-Boundary Problem

The database had made it to the new cloud intact, but the app still could not read its own rows. In front of me were ciphertext blobs, a Vault full of transit keys, and a production token with just enough shape to look useful and no authority to decrypt anything. The fix required drawing the trust boundary, proving which actor could decrypt which value, importing or recreating the right transit keys, and validating through the application code path instead of poking Vault with admin tools.

Why this feels simpler than it is

Vault transit is often described as "encryption as a service." That phrase is accurate enough to be dangerous. The application does not hold raw encryption keys. It sends plaintext to Vault's transit secrets engine and gets ciphertext back, or sends ciphertext and gets plaintext back. The database stores the ciphertext. Vault stores the key material and policy logic. Runtime secrets decide whether the app can ask Vault to decrypt.

That division is the whole point, and it is also why migration conversations go sideways.

Copying the database copies only the locked boxes. If the target Vault has the same transit key material, the same key names, and compatible key versions, those boxes may still open. If it does not, re-encryption is not a ceremony. It is the act of translating data from one trust boundary into another.

Several things can be broken while still looking mostly present. The database can be complete. The Vault server can answer health checks. Some customer keys can exist. A token can be syntactically valid but scoped to the wrong path, the wrong namespace, or the wrong mount. That is how you get a migration that is "95 percent complete" and still cannot read the one field the product needs.

The migration starts with actors, not assets

I stopped treating the migration as a list of buckets, PVCs, and databases. Those matter, but they are inventory. The design question was simpler and less forgiving: who is allowed to turn ciphertext into plaintext?

For this app, the actors were:

The dead scoped token was the giveaway. A dead token is not just an expired credential. In practice, it is evidence that your migration plan depends on an authority you may no longer have. A scoped token makes that worse in a useful way: it may be alive for one mount and useless for another. You can spend a quiet hour proving Vault is up while the app still cannot decrypt a single customer row. Ask me how I acquired this charming little scar.

This is the same kind of boundary problem I keep running into in systems that look operationally simple from the outside. The lesson rhymes with Building Press, Part 1: Read the work, leave the secrets: first identify what must not move casually, then build the workflow around that constraint.

Check what the app actually calls

The first artifact I want in hand is not a cloud diagram. It is the application code or config that names the Vault mount, key prefix, and auth source. A simplified Django-style configuration looked like this after anonymizing names:

VAULT_ADDR = os.environ["VAULT_ADDR"]
VAULT_TOKEN = [REDACTED:secret]"VAULT_TOKEN"]
VAULT_TRANSIT_MOUNT = "transit"
CUSTOMER_KEY_PREFIX = "customer-keys"
 
client = hvac.Client(url=VAULT_ADDR, token=[REDACTED:secret]
 
 
def decrypt_customer_value(key_hash: str, ciphertext: str) -> str:
    response = client.secrets.transit.decrypt_data(
        name=f"{CUSTOMER_KEY_PREFIX}/{key_hash}",
        mount_point=VAULT_TRANSIT_MOUNT,
        ciphertext=ciphertext,
    )
    return base64.b64decode(response["data"]["plaintext"]).decode("utf-8")

That one function answers more than a dozen infrastructure guesses. The key name is per customer. The database value is a Vault transit ciphertext, not some app-local AES envelope. The app needs update capability on the transit decrypt endpoint, because Vault models encrypt and decrypt operations as writes.

A matching policy is boring, which is what you want:

path "transit/decrypt/customer-keys/*" {
  capabilities = ["update"]
}
 
path "transit/encrypt/customer-keys/*" {
  capabilities = ["update"]
}
 
path "transit/keys/customer-keys/*" {
  capabilities = ["read"]
}

Separate key parity from data ownership

One prior check found the target Vault had only a small subset of the per-customer keys. That sounds catastrophic, but the next question matters: do the missing-key customers own encrypted rows?

If a missing key belongs to a dormant customer with no encrypted findings, it is still a cleanup problem, not data loss. If a missing key belongs to a customer with encrypted findings, the target app cannot decrypt those rows. This is where vague migration status reports become expensive.

The useful query shape is a join between distinct encrypted-data owners and the Vault key inventory:

with encrypted_owners as (
  select distinct customer_key_hash
  from findings
  where encrypted_payload is not null
), target_keys as (
  select key_hash
  from vault_key_inventory
  where environment = 'aws-prod'
)
select
  count(*) filter (where target_keys.key_hash is not null) as decryptable_owners,
  count(*) filter (where target_keys.key_hash is null) as blocked_owners
from encrypted_owners
left join target_keys
  on target_keys.key_hash = encrypted_owners.customer_key_hash;

The names here are generic, but the shape is the point. Do not ask whether all historical customers have keys. Ask whether every customer that owns ciphertext has a usable key in the target Vault.

When imported keys remove re-encryption, and when they do not

The recurring question was fair: if we import the keys, do we still need to decrypt and re-encrypt the data?

Sometimes, no. If the target Vault receives the same transit key material under the same key names, with compatible versions, and the ciphertext format references those versions in a way the target understands, the database ciphertext can move as-is. That is the nice path. It is also the path that requires proof, not vibes.

One important constraint before assuming this path is available: exportability depends on how the key was originally configured and on your organization's operational policy. Some organizations forbid exporting transit key material entirely, and a key created without the exportable flag set cannot be exported after the fact. Check both the key configuration and your policy before treating import/export as the default.

Here is the decision table I used:

The trap is treating imported keys as a checkbox. Vault transit ciphertext carries enough metadata for Vault to know which key version should decrypt it. If key rotation happened in the source and the target only has the latest material, old ciphertext can still fail. Importing under a different path changes the API contract even if the raw material is correct. HashiCorp's documentation for transit key export is the place to check the mechanics before assuming key movement is possible in a given setup.

A minimal parity check through the Vault API is useful:

cd ~/vault-migration
export VAULT_ADDR="https://vault.aws.example"
export VAULT_TOKEN="[REDACTED:secret]"
 
vault read -format=json transit/keys/customer-keys/0f3a9c7e \
  | jq '{name: .data.name, min_decryption_version: .data.min_decryption_version, latest_version: .data.latest_version}'

It is still infrastructure validation, though. The stronger test is to run the application decrypt function against a known encrypted row without printing plaintext:

cd /srv/findings-app
export VAULT_ADDR="https://vault.aws.example"
export VAULT_TOKEN="[REDACTED:secret]"
python manage.py shell <<'PY'
from findings.models import Finding
from findings.crypto import decrypt_customer_value
 
row = Finding.objects.exclude(encrypted_payload=None).only(
    "customer_key_hash", "encrypted_payload"
).first()
 
try:
    decrypt_customer_value(row.customer_key_hash, row.encrypted_payload)
    print({"decrypt_ok": True, "key_hash": row.customer_key_hash})
except Exception as exc:
    print({"decrypt_ok": False, "key_hash": row.customer_key_hash, "error": type(exc).__name__})
PY

Notice what this does not print: customer data. A migration validation script that leaks plaintext has managed to fail while succeeding, which is an efficient use of everyone's anxiety.

Why I rejected the tidy shortcuts

The tempting shortcut was to fix everything inside Vault. That sounds clean because it keeps the application untouched. It works only when the problem is actually Vault: missing key material, wrong mount, wrong policy, request-size limits, or a token that can be renewed or replaced.

It does not work when the runtime environment is wired to the wrong secret, when the application computes key names differently than the migration script, or when the database copy came from one environment and the Vault keys came from another. Dev and prod Vault instances are not moral equivalents. They can share hostnames that look similar, mounts that look similar, and policies that look similar. The ciphertext does not care about your naming convention.

The other shortcut was decrypting on the source side, putting static plaintext values into object storage, then importing them later. I only like that for a narrow recovery window with strict controls, because it changes the risk profile entirely: you trade a cryptography migration for a plaintext-handling incident waiting for a calendar invite. If you must do it, encrypt the export with a migration-only key, lock down object storage, expire access quickly, and log reads. Even then, rewrapping directly from source Vault to target Vault is preferable when access allows it.

A rewrap worker has a clearer boundary:

cd ~/vault-migration
export SOURCE_VAULT_ADDR="https://vault.gcp.example"
export TARGET_VAULT_ADDR="https://vault.aws.example"
export SOURCE_VAULT_TOKEN="[REDACTED:secret]"
export TARGET_VAULT_TOKEN="[REDACTED:secret]"
python rewrap_findings.py --batch-size 500 --dry-run
python rewrap_findings.py --batch-size 500 --write

The implementation can stream rows, decrypt with the source transit key, encrypt with the target transit key, and write the new ciphertext. It should be resumable and idempotent, with a marker column or migration table that records key hash, row id, source ciphertext digest, target ciphertext digest, and status. The digest is enough to audit progress without preserving plaintext.

create table finding_rewrap_audit (
  finding_id bigint primary key,
  customer_key_hash text not null,
  source_ciphertext_sha256 text not null,
  target_ciphertext_sha256 text,
  status text not null check (status in ('pending', 'rewrapped', 'failed')),
  error_class text,
  attempted_at timestamptz not null default now()
);
 
create index finding_rewrap_audit_status_idx
  on finding_rewrap_audit (status, attempted_at);

The validation path that actually matters

Final validation has to trigger application code. Not a Vault health check. Not a Terraform output. Not a reassuring list of Kubernetes secrets. The app must read a row, ask Vault to decrypt it using the same runtime secret it will use after cutover, and proceed without special privileges.

There are still costs to this approach. Rewrapping data takes time, adds write load to the database, and needs careful batching and rollback semantics. Imported key validation can be faster, but it demands precise proof that key versions and names match. Application-path validation is slower than a curl against Vault, but it catches the failures users will actually feel.

That tradeoff is acceptable. Cryptography migrations fail in the spaces between systems: a database copied from prod, a token from dev, a Kubernetes secret from last month, a cloud identity nobody thought was part of encryption. I ended up treating this like the operational work in When the cluster was ahead of the code: the system was already telling us where the contract had drifted, but only if we followed the runtime path instead of the inventory list.

FAQ

Why is this unreliable when the database copy succeeds?

The database usually stores ciphertext, not the keys needed to decrypt it. The target environment also needs compatible Vault transit keys, key versions, policies, and a runtime token that the application can use.

Where should I capture Vault key parity?

Capture parity at the transit key path the application uses, such as transit/keys/customer-keys/<hash>. Compare only the key names and version metadata you need, and avoid printing tokens or plaintext.

Do imported Vault transit keys remove the need to re-encrypt?

They can, if the imported key material, names, paths, and versions match what produced the ciphertext. If any of those differ, rewrapping through source decrypt and target encrypt is the safer path.

Why validate through application code instead of Vault CLI?

The Vault CLI often runs with a stronger token than the workload. Application validation proves the deployed pod, secret, policy, database row, and decrypt function work together.

What is a dead scoped token?

It is a token that is expired, revoked, or valid only outside the path you need. In a migration, it means you may have infrastructure access without the authority required to decrypt production ciphertext.