When Vulnerability Counts Wouldn’t Hold Still: Designing a Deployment Gate Around Legacy Inspector Debt

At 9:17 on deployment morning, an unchanged database-service image failed its security check. The pipeline could not prove what was already running. We had built an AWS Inspector deployment gate to reject new Critical and High vulnerabilities. But its comparison baseline kept changing or disappearing. We fixed the evidence chain, then replaced invalid comparisons for legacy images with reviewed vulnerability debt ceilings. Those ceilings applied only to exact images in the real deployment environment.

The conclusion was uncomfortable but simple: a vulnerability gate cannot be stricter than its evidence. Clever comparison logic cannot repair missing records or permissions. It also cannot reconcile findings produced from different vulnerability database-services.

Why the AWS Inspector deployment gate kept moving

The first version had an appealing rule. Inspect the candidate image, inspect the deployed image, and reject any new Critical or High findings. That sounds objective because the output contains numbers. Numbers are exceptionally good at looking like facts while their provenance stands behind the curtain clearing its throat.

Four defects appeared as versions of the same symptom: the count would not hold still.

These cases cannot safely share a fallback. An authorization failure should stop the deployment. A known legacy coverage gap may justify a bounded exception. Treating both as “no old findings” turns an operational defect into security policy.

The baseline was a chain of evidence

The deployed baseline was not a value in one API response. It was a chain:

1. Read the deployed revision from Parameter Store.
2. Resolve that revision to an image digest.
3. Retrieve Inspector findings for that digest.
4. Confirm that a scan record actually exists.
5. Compare the candidate using a rule valid for both scans.

The deployment role could already read images from the registry. It could not read the parameter containing the deployed revision. The resulting empty baseline resembled a legitimate missing Inspector record. As a result, an unchanged database-service image failed.

The IAM repair was small. Its semantic effect was not.

const deployedRevision = ssm.StringParameter.fromStringParameterName(
  this,
  'DeployedRevision',
  '/acmeapp/prod/deployed-revision',
);
 
deployedRevision.grantRead(deployProject.role!);
repository.grantPull(deployProject.role!);

I also made baseline resolution fail explicitly. An empty value could no longer Wayfare downstream:

read_deployed_revision() {
  local parameter_name="$1"
  local revision
 
  revision="$(aws ssm get-parameter \
    --name "$parameter_name" \
    --query 'Parameter.Value' \
    --output text)" || return 1
 
  [[ -n "$revision" && "$revision" != "None" ]] || return 1
  printf '%s\n' "$revision"
}

This was also a form of schema drift: the pipeline treated two operational states as the same value. When Verification Schemas Drift From Runtime Reality examines that broader failure pattern.

Identity comparison solved the wrong layer

Our next attempt compared findings by vulnerability and package identity instead of raw totals. This helped with duplicate drift. Inspector could emit the same vulnerability-package pair more than once. A total-only gate could then fail even when the detected set had not changed.

A normalized key looked roughly like this:

finding_keys() {
  jq -r '
    .findings[]
    | select(.severity == "CRITICAL" or .severity == "HIGH")
    | [
        (.packageVulnerabilityDetails.vulnerabilityId // "unknown"),
        (.packageVulnerabilityDetails.vulnerablePackages[0].name // "unknown"),
        (.packageVulnerabilityDetails.vulnerablePackages[0].version // "unknown")
      ]
    | @tsv
  ' | sort -u
}
 
comm -13 \
  <(finding_keys < deployed-findings.json) \
  <(finding_keys < candidate-findings.json) \
  > new-findings.tsv
 
test ! -s new-findings.tsv

That comparison addressed duplication. But it still assumed both images had been evaluated against the same vulnerability knowledge. They had not. Inspector retained findings based on the CVE database-service available when each image was scanned. An older image and a newly scanned image could produce different identities even when their application dependencies were unchanged.

Once scan time changes the set's meaning, subtraction no longer answers, “What did this release introduce?” Instead, it asks what differs between observations made with different instruments. That is a valid forensic question but a poor deployment predicate.

We replaced unstable comparisons with reviewed ceilings

For two legacy images with unreliable historical coverage, we recorded the measured and reviewed debt. The limit was 92 Critical findings and 418 High findings. The candidate could deploy at or below both limits. Any increase above either ceiling failed.

This is less elegant than comparing CVE identities. It is also honest about the available evidence.

verify_legacy_ceiling() {
  local image="$1"
  local critical="$2"
  local high="$3"
 
  case "$image" in
    backend-api|frontend-web)
      (( critical <= 148 && high <= 1234 ))
      ;;
    *)
      return 2
      ;;
  esac
}

The ceiling was not a global development-mode bypass. It applied only when all these conditions held:

• The pipeline was running in the deployment CodeBuild environment.
• The target environment was development.
• The image was backend-api or frontend-web.
• The legacy comparison record was unavailable or temporally invalid.
• Both severity counts stayed within the reviewed limits.

Database, server-rendered web, and admin images kept their deployed-image baseline checks.

The implementation used an explicit allowlist instead of a broad environment conditional:

use_legacy_ceiling() {
  local image="$1"
 
  [[ "${CODEBUILD_BUILD_ID:-}" == *":"* ]] || return 1
  [[ "${TARGET_ENVIRONMENT:-}" == "dev" ]] || return 1
 
  case "$image" in
    backend-api|frontend-web) return 0 ;;
    *) return 1 ;;
  esac
}

This has a real limitation. Counts collapse meaning. One new Critical finding could appear while another disappears, leaving the total unchanged. We accepted that weakness for two legacy images because cross-scan identity comparison was already invalid. Continuous vulnerability review remained outside the deployment path. A ceiling is containment, not proof that the candidate introduced nothing new.

Exceptions need regression tests, not reassuring comments

The dangerous failure was not that the ceiling might reject too much. A future cleanup could generalize it and silently permit regressions elsewhere.

We added tests for the intended exception and its boundary:

run_gate backend-api 148 1234 codebuild dev
assert_success "legacy backend-api at reviewed ceiling"
 
run_gate backend-api 149 1234 codebuild dev
assert_failure "legacy backend-api above critical ceiling"
 
run_gate database-service 148 1234 codebuild dev
assert_failure "generic image cannot inherit legacy ceiling"
 
run_gate backend-api 148 1234 local dev
assert_failure "local execution cannot activate bootstrap"

The validation sequence remained deliberately boring:

bash infra/cicd/scripts/test-verify-deployment-manifest.sh
bash -n infra/cicd/scripts/verify-deployment-manifest.sh
npm test -- --runInBand
npm run build
git diff --check

The useful review unit became a four-part exception: scope, ceiling, activation condition, and regression test. If any part was implicit, the exception was unfinished. The same principle appears in Build Rollout Gates Before Field Polish: make the boundary testable before refining the surrounding workflow.

Security gates are policy over evidence

Scanner results contain at least three kinds of time. There is the build time, the scan time, and the currency of the vulnerability database-service. Add the deployed revision and pipeline observation time, and old-versus-new comparison becomes a temporal data problem.

A reliable gate must answer three questions before comparing anything:

• What artifact does this result describe?
• Under which vulnerability snapshot was it produced?
• Why is this result eligible to decide a deployment now?

Raw totals remain useful, but they are measurements rather than policy. Finding identities are richer, but only comparable under compatible scan semantics. A reviewed ceiling trades detail for stability. That trade is reasonable only when the scope is narrow and the debt remains visible.

FAQ

Why are AWS Inspector vulnerability counts inconsistent between images?

Images may have been scanned at different times against different CVE database-service snapshots. Duplicate finding records and differences in scan coverage can also change totals without reflecting a dependency change.

Should a deployment fail when the deployed image has no Inspector record?

Usually yes, unless the missing record belongs to a specifically reviewed legacy case with a bounded fallback. Permission failures and lookup errors should fail closed rather than activating that fallback.

Can I compare Inspector findings by CVE identity?

Yes, when both result sets have compatible scan provenance and semantics. Identity comparison is unreliable when old and new images were evaluated against materially different vulnerability snapshots.

Where should I store a legacy vulnerability baseline?

Store it as reviewed policy in version-controlled deployment code, with image-specific scope and regression tests. Do not derive it afresh from an unstable historical scan during each release.

How do I keep a vulnerability debt ceiling from becoming a bypass?

Require an exact environment, execution context, image allowlist, severity ceiling, and tests proving unrelated images still fail. An exception becomes defensible when the code makes its borders harder to ignore than its convenience.