Cold storage you can't search is a filing cabinet in a warehouse

I moved one original into cold storage, refreshed the asset grid, and watched its thumbnail disappear. The archive had made the asset unusable before anyone even tried to download it. That is the failure mode searchable cold storage must prevent.

The mistake was treating "asset" and "original object" as synonyms. An asset is a record with rights, provenance, metadata, versions, relationships, and several representations. The original is one placement of bytes. Once I separated those concepts, cold storage became a state within the library rather than a second library.

Three invariants for searchable cold storage

A common first implementation changes the object's class and lets the normal read path discover that GET no longer works. The damage extends beyond download:

• Search results omit archived records because the renderer cannot produce a tile.
• Collections and portals replace previews with placeholders.
• A CMS rendition fails because its transform reads the cold master.
• Users learn that "archive" means "make this disappear," so they keep private warm copies elsewhere.

The organisation then pays for both an archive and a shadow library.

dam.rs instead enforces three invariants:

1. Identity stays hot. The asset record, metadata, extracted text, embeddings, thumbnails, detached provenance manifests, and master proxy remain available.
2. Byte placement is independent of restore availability. The original may stay in an archive class while a temporary readable copy exists or expires.
3. Money-spending transitions require durable workflow state. A restore or lifecycle move must survive browser, request, and worker lifetimes, with its cost and status visible.

Only original masters may move to a restore-required class. The browser can therefore show the same result before, during, and after a restore.

Object key namespaces make enforcement cheaper because proxies and thumbnails can be identified as tier-exempt without another database lookup. Namespaces alone are not a guarantee. Lifecycle policies must also be scoped to original-only prefixes or guarded by enforcement that rejects transitions for browsing artifacts. Otherwise a policy that matches everything can still move them.

Storage class and restore state are separate facts

An archived S3 object does not become a Standard object when restored. For S3 Glacier Flexible Retrieval and Deep Archive, a restore creates a temporary accessible copy while the object remains in its archive class. AWS deletes that copy after the requested availability period. The AWS restore documentation describes the distinction.

The database therefore needs two dimensions:

• storage_class records where the durable object lives.
• restore_state records whether a temporary readable copy is requested, ongoing, available, or expired.

The schema refuses an available restore without an expiry:

storage_class      text NOT NULL DEFAULT 'STANDARD',
restore_state      text NOT NULL DEFAULT 'none'
                       CHECK (restore_state IN (
                         'none', 'requested', 'ongoing',
                         'available', 'expired'
                       )),
restore_expires_at timestamptz,
 
CONSTRAINT placements_restore_expiry CHECK (
  restore_state <> 'available' OR restore_expires_at IS NOT NULL
)

Recording a completed restore by changing the class to STANDARD creates a delayed failure. The temporary copy eventually expires, but the row continues to claim instant access.

The type system carries the same distinction. Restore-required behaviour belongs to the storage class instead of string comparisons scattered through handlers:

pub fn requires_restore(self) -> bool {
    matches!(self, Self::Glacier | Self::DeepArchive)
}
 
pub fn min_duration_days(self) -> u32 {
    match self {
        Self::Standard | Self::IntelligentTiering => 0,
        Self::StandardIa | Self::OnezoneIa => 30,
        Self::GlacierIr | Self::Glacier => 90,
        Self::DeepArchive => 180,
    }
}

S3 Glacier Instant Retrieval contains "Glacier" in its name but supports real-time GET. A substring check would send immediate-access objects through an unnecessary restore workflow.

Put billing rules in the state machine

Archive pricing includes byte-month storage, minimum billable duration, retrieval requests, and retrieved bytes. Minimum billable object sizes and metadata overhead may also apply.

Current AWS documentation lists a 90-day minimum for S3 Glacier Instant Retrieval and Flexible Retrieval, and 180 days for Deep Archive. Deleting, overwriting, or transitioning an object early still incurs the remaining duration charge. Glacier Instant Retrieval bills objects at a minimum of 128 KB. The S3 storage-class comparison is the source dam.rs follows.

A small worked comparison makes the mechanism concrete. Consider an illustrative 100 MB original and a 20 KB thumbnail:

• In a warm class, storage is based on their actual sizes under that class's rules.
• In Glacier Instant Retrieval, the 20 KB thumbnail is billed as 128 KB, or 6.4 times its logical size, before retrieval charges. The 100 MB original is far above that minimum, so the size floor is negligible for it.
• If either object leaves a class with a 90-day minimum after 30 days, 60 days of minimum-duration charges remain. For the thumbnail, those charges apply to the 128 KB billable size.

The calculation excludes regional prices because those inputs change. It still shows why large originals are plausible archive candidates while tiny, frequently read thumbnails are poor ones.

Avoid class churn

An "archive after 30 idle days" rule can repeatedly start minimum-duration clocks if opening an asset permanently moves its original to Standard and another idle period moves it cold again. Recent access is not enough information to authorize a transition.

Each placement carries min_duration_until, and the planner refuses another move before that instant:

if let Some(until) = candidate.min_duration_until
    && now < until
{
    return Verdict::Skipped(
        SkipReason::MinDurationNotElapsed { until }
    );
}

A restore is not a warm transition. It leaves the durable class unchanged and creates temporary readability, matching S3's behaviour without restarting a storage-class clock.

Keep tiny browsing artifacts hot

A 20 KB thumbnail billed as 128 KB can cost more in an infrequent-access class than in Standard. It also gains retrieval charges while saving almost no storage. The lifecycle engine excludes proxy, thumbnail, manifest, and staging namespaces before considering price.

Large originals can move while small browsing artifacts stay available. Treating every object attached to an asset as one lifecycle unit loses that advantage.

Make restore a durable workflow

Once an original is cold, the download button cannot pretend the response is merely slow. The API must represent a long-running, priced operation.

dam.rs gives the user a quote before confirmation, including the class, estimated wait, estimated charge, and any restore already in flight. Approval and subsequent progress live in durable state. A worker submits the request and polls until the copy becomes available, then the system notifies the user.

AWS currently documents typical S3 Glacier Flexible Retrieval windows of 1 to 5 minutes for Expedited retrieval of eligible objects, 3 to 5 hours for Standard, and 5 to 12 hours for Bulk. Deep Archive has no Expedited tier. Standard is typically within 12 hours and Bulk within 48. The archive retrieval options also describe request-rate and large-dataset constraints, so the interface presents these windows as estimates rather than deadlines.

A spinner implies that the current HTTP request may finish. A restore can outlive the browser session, the user's working day, or the initiating worker. Durable state and eventual notification are part of the operation, not interface polish.

The restore request is idempotent. A uniqueness rule over the asset version, object placement, and requested retrieval tier permits one active request, while a compare-and-set transition prevents two approvers or workers from issuing the provider restore twice. The row records provider request time, observed state, accessible-until time, last poll, and terminal failure.

Workers reconcile rather than fire and forget. After a crash, another worker resumes polling the same request; a late provider response cannot overwrite a newer terminal state. Delivery checks the current accessible-until value during authorisation because expiry can race with a user opening a completion notification.

A lifecycle plan in dam.rs, using representative development data. Originals are planned separately from tier-exempt thumbnails and proxies before any transition runs.

Keep search off the original

A database row alone does not preserve searchability. Search, preview, and integration paths must avoid accidental reads of the master.

The indexing pipeline uses hot metadata and proxy material. The asset grid receives a thumbnail URL for the hot derivative. The delivery handler checks archive state only when the signed claim names the original. A thumbnail claim should not return 202 Archived because the master beside it is cold.

The boundary is easy to miss because the asset owns both objects. Resolving archive state at the asset level would render every thumbnail in a fully archived library as a pending restore. That would be consistent with the master and wrong for the requested bytes.

Integrations follow the same boundary. A page render must not trigger a restore by surprise. A connector either uses an existing warm rendition, receives an explicit archived response, or has permission to request restores. Restore permission is distinct from read permission because retrieval spends money.

Test the provider, not only the planner

A fake store with a controllable clock is ideal for state-transition tests. It can advance a restore from requested to available, expire the temporary copy, and exercise minimum-duration boundaries in milliseconds. It proves the planner's internal behaviour.

It cannot prove that AWS accepts the request, reports the expected headers, preserves the original storage class, and eventually serves the same bytes.

The repository therefore has a separate AWS conformance command. It is excluded from the ordinary pre-push gate because it uses a real account, creates billable archive objects, and waits on an external restore. In one recorded run in ap-south-1, 20 cases passed, none skipped, and an Expedited Glacier restore became readable in about 76.7 seconds. That is a measured observation, not an SLA.

Zero skips matter because local S3-compatible stores honestly skip restore-completion cases they cannot implement. Against AWS, a skip would indicate that the driver under-declared a capability the backend provides.

The workflow had previously been green while doing no useful work. First it named a Cargo feature that did not exist. Later, missing credentials printed a warning and exited successfully. The command is now manual until a deliberate non-interactive identity is configured, and missing credentials fail instead of manufacturing a green archival claim.

Remaining limits

The design does not make archive access instant. A user who needs an original now may wait hours. Even accurate individual quotes cannot prevent a bad policy from creating a restore queue that hits provider quotas or surprises the budget.

Cost estimates depend on region, provider, retrieval tier, object size, and current price tables. They need versioned inputs and a visible timestamp. A stale estimate is more dangerous than no estimate because it still looks authoritative.

Hot proxies create an operational obligation. If they are lost while the original is in Deep Archive, rebuilding previews requires restores. The browsing substrate needs its own durability, backup, and scrub policy. I apply the same operating principle in Trusting a home server: a local model, monitoring, and backups. Keeping proxies hot is an architecture choice, not a substitute for operating them.

S3-compatible backends also differ. Some echo a storage-class header without changing behaviour. Some do not return server-side checksums on HEAD. Capability declarations and conformance cases must be specific to each driver. Compatibility is a claim that needs evidence.

FAQ

Can an archived digital asset remain searchable?

Yes. Search uses metadata and an index, while the grid uses warm thumbnails or proxies. Only downloading or reprocessing the cold original needs a restore.

Does restoring a Glacier object move it back to Standard?

No. For S3 Glacier Flexible Retrieval and Deep Archive, S3 creates a temporary readable copy and leaves the durable object in its archive class. The application must track restore state and expiry separately from storage class.

Why not archive thumbnails with the original?

Thumbnails are small, frequently read, and may be subject to minimum billable object sizes in colder classes. Keeping them hot preserves browsing and often costs less than tiering them.

What makes a cold-storage DAM trustworthy?

A trustworthy cold-storage DAM keeps the library searchable without reading originals. It accounts for billing constraints in lifecycle decisions, records restores as durable and explicit operations, and verifies backend claims against the named backend. The asset remains present even when its largest representation takes longer to retrieve.