Consolidating AWS Delivery Without Losing the Runtime Boundary

The branch looked harmless until the deployment path split in four directions. server/ still held the staging and production entry points people trusted. Docker Compose, CDK, Ansible, and pipeline work sat around it like a migration that had started before anyone named the boundary. The fix was not to delete half the tree and call history confused. We used an AWS deployment pipeline to gather the delivery pieces under infra/. We kept the runtime boundary intact, and the existing staging and production entry points still worked while the new path became usable.

The argument I would make after doing this: infrastructure consolidation is boundary work before it is cleanup. Treat it as cleanup and every odd directory looks like debt. Treat it as boundary work and some awkward paths become load-bearing contracts. You can move the delivery system around them without making production learn a new shape at the same time.

That distinction mattered because the branch was not just moving files. Docker and Compose declare what runs. CDK declares what AWS resources exist. Ansible describes how machines are prepared. CircleCI and CodePipeline describe what promotes a change. The old server/ layout answered a quieter question: where operators already expect staging and production to be. Those are not interchangeable answers.

Why this was not a folder cleanup

The tempting version was tidy and dangerous. Put all infrastructure in one place. Rename the old directories. Update every script. Delete the compatibility cruft. Enjoy a short burst of moral superiority. Then watch the first deployment fall over because one staging path was still referenced by a runner, a shell script, or a person who remembered how production worked.

I did not want a pretty tree that required a runtime migration first. The safer architecture was additive. Preserve server/ where staging and production expected it. Make infra/ the home for the new delivery machinery.

That diagram is the whole job in miniature. The runtime contract stays legible. The delivery system becomes explicit. The two touch through named paths and generated artifacts rather than assumption.

Keeping the runtime boundary boring on purpose

The existing server/ layout was the production-facing contract. It encoded staging and production assumptions already wired into CircleCI and operational habits. Changing that layout while also introducing CDK, CodeBuild, CodePipeline, and Ansible would have combined two migrations. One migration changes where the app lives. The other changes how it gets delivered.

That coupling makes rollback miserable. If a deploy fails, you no longer know whether the problem is the cloud stack, the build image, the provisioner, the app path, or a renamed directory. The log will usually pick one suspect and lie with confidence.

So the compatibility path stayed. In practice, the old path remains the entry point while the new delivery flow calls through it:

cd /srv/app
./server/bin/deploy-staging.sh --image registry.example.com/site-api:${GIT_SHA}

That command is deliberately dull. It says: the new pipeline decides when to deploy and which image to use. The existing server-side deploy command still owns the last mile. Dull is underrated when production is involved.

Giving delivery one home

The opposite mistake would have been leaving each experiment branch with its own partially correct delivery idea. One branch knows about ARM64 CodeBuild. Another has the CDK stack. Another carries Ansible inventory. Another has Compose changes. That looks flexible until someone has to answer which one is real.

The consolidated shape put delivery assets under infra/:

infra/
  ansible/
    inventory/
    playbooks/
  cdk/
    bin/
    lib/
  compose/
    compose.prod.yml
    compose.staging.yml
  pipeline/
    buildspec.yml
server/
  bin/
  staging/
  production/

The names are not clever. Each directory has a job. infra/cdk defines cloud resources. infra/ansible prepares hosts. infra/compose describes runtime composition. infra/pipeline describes build behavior. server/ remains the runtime surface that existing staging and production paths understand.

Build an AWS deployment pipeline around old entry points

The useful pattern was to let the new system own orchestration while the old system kept the operational contract. That sounds like diplomacy because it is. Repositories accumulate treaties. I used the same bias toward boring runtime contracts in Building Press, Part 6: The datacenter can't post, so the laptop does, where delivery had to respect the machine that could actually publish.

The handoff points matter more than the tools. CDK creates the AWS resources. CodeBuild builds an ARM64 image because the target runtime requires that architecture. A mismatch here would have been the first quiet failure. Ansible prepares the host and writes the environment. Compose starts the application services. The existing server deploy script remains the point where staging and production behavior is selected.

A reduced buildspec shows the boundary:

version: 0.2
 
env:
  shell: bash
  variables:
    IMAGE_REPO: registry.example.com/site-api
 
phases:
  pre_build:
    commands:
      - test -n $CODEBUILD_RESOLVED_SOURCE_VERSION
      - export IMAGE_TAG=${CODEBUILD_RESOLVED_SOURCE_VERSION:0:12}
  build:
    commands:
      - docker buildx build --platform linux/arm64 -t ${IMAGE_REPO}:${IMAGE_TAG} .
  post_build:
    commands:
      - docker push ${IMAGE_REPO}:${IMAGE_TAG}
      - printf '%s\n' ${IMAGE_TAG} > image-tag.txt
artifacts:
  files:
    - image-tag.txt

The build does not SSH into a machine. It does not decide which services restart. It produces an image and a tag. That restraint is the boundary.

Ansible then consumes the tag and prepares the host. It does not pretend to be a build system:

- hosts: app_hosts
  become: true
  vars:
    app_dir: /srv/app
    image_repo: registry.example.com/site-api
  tasks:
    - name: Write deployment image
      copy:
        dest: '{{ app_dir }}/server/.image'
        content: '{{ image_repo }}:{{ image_tag }}\n'
        mode: '0644'
 
    - name: Run staging deploy
      command: ./server/bin/deploy-staging.sh --image {{ image_repo }}:{{ image_tag }}
      args:
        chdir: '{{ app_dir }}'

There is a cost here. The Ansible playbook still knows about server/.image and server/bin/deploy-staging.sh. That is not pure. It is also much easier to verify than a simultaneous path rename, deploy rewrite, and cloud migration.

What I rejected

I considered three cleaner-looking options and disliked all of them for different reasons.

The first option is seductive because it makes the repository look intentional. It also makes the app runtime look like an implementation detail of the infrastructure system. That is backwards. The runtime is what users experience. Delivery exists to move it safely.

The second option is what engineers reach for when old shell scripts have been staring back at us too long. Deleting them is a poor substitute for understanding why they survived.

The third option is the slow failure mode. No one gets paged by branch sprawl the day it happens. They get paged later, when two delivery paths disagree about the same environment. Then the team has to debug archaeology under pressure. The same pattern shows up in When the cluster was ahead of the code: the system can look modern while the deployable contract is still behind it.

Keeping CDK declarative

CDK turns into a dumping ground for operational behavior if you let it. I wanted the TypeScript stack to describe AWS resources and handoffs. I did not want it to hide deployment logic inside construct code. AWS documents the same split at the service level: AWS CDK defines cloud infrastructure, while AWS CodeBuild runs build work.

A simplified stack boundary looked like this:

import * as cdk from 'aws-cdk-lib';
import * as codebuild from 'aws-cdk-lib/aws-codebuild';
import * as codepipeline from 'aws-cdk-lib/aws-codepipeline';
import { Construct } from 'constructs';
 
export class DeliveryStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
 
    const build = new codebuild.PipelineProject(this, 'ArmBuild', {
      environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_7_0,
        privileged: true,
        computeType: codebuild.ComputeType.SMALL,
      },
      buildSpec: codebuild.BuildSpec.fromSourceFilename('infra/pipeline/buildspec.yml'),
    });
 
    new codepipeline.Pipeline(this, 'DeliveryPipeline', {
      pipelineName: 'site-api-delivery',
      restartExecutionOnUpdate: true,
    });
  }
}

This excerpt is incomplete on purpose. The important part is where the build definition lives: infra/pipeline/buildspec.yml. The stack points at it rather than swallowing it. That keeps cloud topology and build procedure close enough to connect, but separate enough to review independently.

cd /srv/app
test -x ./server/bin/deploy-staging.sh
test -d ./server/staging
test -d ./server/production
test -f ./infra/pipeline/buildspec.yml

Where this still has sharp edges

The chosen approach buys safety by carrying compatibility forward. That means the repo temporarily has two centers of gravity: infra/ for delivery and server/ for runtime. New contributors need to understand why both exist. Reviewers also need to keep runtime behavior from drifting into infra/ just because that is where current work is happening.

There is also a sequencing risk. If the team never schedules the follow-up to retire compatibility paths, additive consolidation becomes permanent sediment. The difference between a bridge and a landfill is usually a removal ticket with an owner.

I am comfortable with that tradeoff because the alternative was pretending a repository can be made correct by making it aesthetically consistent. Correctness here meant preserving known deployment behavior while making the AWS deployment pipeline explicit enough to reason about.

FAQ

Why is consolidating delivery infrastructure risky?

It is risky because it often changes delivery tooling and runtime assumptions in the same branch. When the pipeline, host provisioning, build architecture, and app paths all move together, a failure has too many plausible causes.

Where should I put CDK, Ansible, and Compose files?

Put delivery-owned files under a clear infrastructure root such as infra/, but keep production runtime entry points where the running system and operators already expect them until you have a tested migration path.

Why keep compatibility paths during a migration?

Compatibility paths reduce the blast radius. They let the new pipeline orchestrate builds and provisioning while the existing staging and production scripts keep owning the last-mile runtime behavior.

When should old deployment paths be removed?

Remove them after the new path has run successfully, the old entry points have no remaining callers, and the removal itself is reviewed as a separate behavior change.