Skip to content

Bulk PDF Agent ARRNC Hosting Correction — Migration Plan

Design: 2026-08-30-bulk-pdf-agent-arrnc-hosting-design.md. Decision: ADR-015.

This is the executable, multi-repository Phase 2 plan. Phase 1 is documentation only and stops at Gate A for explicit architecture approval. No PR below is authorized by the existence of this plan.

1. Safety invariants

Every implementation PR and operation preserves these invariants:

  1. Production is untouched until a distinct promotion plan, review, and Gate C approval.
  2. The Kubernetes Application, ingress, PVC, and reviewed rollback configuration stay intact until ARRNC staging has passed authenticated end-to-end proof. Its Deployment may scale to zero only after the Gate B intake/queue/claim/storage drain; no resource is removed at that cutover.
  3. A queue has at most one active environment consumer. There is no dual-consumer soak.
  4. An ordinary agent sees one exact storage child and no SMB credential or sibling path.
  5. PR code receives no production/staging credential, mount, network, broker account, or Docker control surface.
  6. PR-controlled jobs receive no ARRNC activation identity; privileged activation runs only from protected base-branch code and never executes the PR checkout.
  7. Desired state is declarative. Imperative tools may execute an approved record but do not become a second authority.
  8. All destructive cleanup is canonical-child-scoped, no-follow, idempotent, and evidenced.
  9. A failed mount, sentinel, provenance, capacity, queue-ownership, or restore proof is a STOP, never permission to weaken the contract.
  10. Secrets and participant/project/file identifiers are absent from PRs and proof bundles.
  11. Each live apply has a named operator, rollback owner, observation window, and explicit gate.

2. Dependency graph and Phase 2 waves

flowchart TD
    A[Gate A<br/>approve and merge planning PR] --> W1A
    A --> W1B
    W1A[Wave 1A server-config<br/>storage + persistent staging] --> W2
    W1B[Wave 1B SyRF<br/>runtime/deploy contracts] --> W2
    W1A --> W1C[Wave 1C config service + gatekeeper<br/>bounded preview family]
    W1B --> W1C
    W1C --> W2[Wave 2<br/>preview lifecycle automation]
    W1A --> W3[Wave 3<br/>ARRNC staging dry-run, paused]
    W1B --> W3
    W3 --> B{Gate B<br/>approve live staging cutover}
    B --> W4[Wave 4<br/>single-owner staging cutover + E2E]
    W4 --> W5[Wave 5<br/>Kubernetes drain/retirement + doc cleanup]
    W2 --> W6[Wave 6<br/>preview acceptance]
    W5 --> C{Gate C later<br/>production promotion approval}
    W6 --> C

Wave 1 repository PRs may be developed in parallel, but their contracts are merged in the dependency order shown. Wave 3 is the first ARRNC environment mutation and requires reviewed implementation plus an authorized operator. Wave 4 cannot start without Gate B.

Wave Repository PRs Outcome
1A server-config exact staging child mount, isolated network/scanner, paused persistent slot, quotas/alerts/verification
1B syrf immutable-digest ARRNC deployment inputs, parity health/config, queue-owner controls, authenticated proof harness
1C server-config plus arrnc-api-deploy only where transport changes are required, and university storage owner bounded dynamic preview-family registrar, per-PR storage/allocation, rootless container isolation, atomic fleet admission, and exact gatekeeper execution contract
2 syrf plus server-config approved-PR activation/teardown, digest provenance, configurable concurrent capacity, queueing, and orphan reconciliation
3 server-config operational apply ARRNC staging reconciled paused; mount/ACL/ClamAV/broker/rollback dry-run evidence
4 cluster-gitops plus controlled ARRNC operation intake freeze, queue/claim drain, Kubernetes scale-to-zero, ARRNC sole consumer, authenticated E2E and soak
5 cluster-gitops then syrf snapshot-backed Application/ingress/PVC retirement; remove obsolete chart/workflow/docs authority
6 syrf plus server-config operational proof multiple concurrent previews from approved PRs, isolation/E2E, independent cancellation/teardown, orphan recovery
future separately planned multi-repo production promotion no action until Gate C

3. Numbered migration plan

Step 1 — Lock the cross-repository contract

Owner: SyRF and ARRNC platform maintainers.

Before implementation branches diverge, add shared test vectors/fixtures for:

  • canonical roots production, staging, previews/pr-N and derived container names;
  • invalid zero, zero-padded, negative, overflow, traversal, separator, Unicode, and sibling inputs;
  • persistent versus preview secret/network/storage profiles;
  • paused/enabled consumer states;
  • image repository, digest, source commit, PR, and trust provenance;
  • teardown state-machine idempotency.

The fixtures contain no credentials or live paths beyond approved non-secret mount names. Each repository consumes the same cases or an exact generated copy with checksum.

Proof: contract suites in all affected repositories agree. Rollback: test-only; no environment effect. STOP: any repository needs a different canonical root or container naming rule.

Step 2 — Provision child-scoped storage contracts in server-config

Repository: camaradesuk/server-config. Owner: ARRNC platform.

Expected files:

  • vars/projects.yml;
  • roles/cifs_mounts/defaults/main.yml, tasks/main.yml, handlers/main.yml, and restart unit template;
  • roles/docker_volumes/tasks/main.yml;
  • roles/verify/tasks/check_cifs_mounts.yml, check_cifs_restart_units.yml, check_volumes.yml, and a focused PDF-agent verification task;
  • filter_plugins/tests/test_syrf_pdf_agent_hosting.py.

Preview-storage implementation dependency and owner:

  • the university Camarades storage owner must provide, before preview activation, an audited operation or isolated allocation pool that accepts only canonical PR number and idempotency key, then creates/allocates and retires the exact previews/pr-N child, distinct principal, ACL, quota, and server-side connection/bandwidth/IOPS limits;
  • its scoped capability remains in the ARRNC root-owned control plane, never GitHub, SyRF workflows, or preview code. A pre-provisioned pool is acceptable when allocation, reset/fencing, identity rotation, and audit prevent prior or sibling PR exposure. This missing automation is an implementation dependency to solve, not an alternative-hosting decision or a blocker for the persistent staging PR.

Implement declarative staging and preview-family storage templates:

  1. create exact host roots for staging and previews/pr-N scratch/output;
  2. require direct child SMB shares and distinct root-owned credential scopes;
  3. pin UID/GID and least-privilege ACLs;
  4. create the structured sentinel only after active mount/source/ACL proof;
  5. add per-child byte/inode quota and ARRNC aggregate capacity preflight;
  6. restart only the matching stopped/failed environment container after mount recovery;
  7. prohibit the common Camarades root, sibling traversal, symlinks, and a preview deletion target outside previews/pr-N;
  8. record that production is unchanged and the ARRNC host backup excludes its mount; defer the remote-share backup attestation/restore proof to Gate C; retain staging-delivered bytes while any durable reference can use them, applying 30 days only to proven unreferenced/orphan test data after application-state retirement; set the preview no-backup/24-hour teardown SLO.
  9. verify the storage-owner audit record and server-side connection/bandwidth/IOPS limits before mounting a preview child.

Tests:

  • Ansible syntax and check mode;
  • negative structural tests for broad share roots, reused credentials, missing quota/sentinel, wrong UID, cross-environment restart, and unsafe cleanup paths;
  • Molecule/rehearsal-host mount-loss and local-underlay refusal where available;
  • external ACL tests proving each identity can access only its child.
  • simultaneous bounded preview I/O tests that create/delete/rename/fsync and prove per-preview and aggregate connection/bandwidth/IOPS limits preserve production and staging service health.

Proof before apply: university SMB export accepts each exact child; distinct ACLs deny siblings; quota and backup classifications are approved. Rollback: remove only the new inactive staging/preview declarations; production diff must be empty. STOP before preview activation: an exact child/allocation, distinct identity, ACL, quota, I/O limits, reset/fencing, and audit cannot be delivered without broader authority. Persistent staging implementation and proof continue independently.

Step 3 — Add the persistent staging runtime to server-config

Repository: camaradesuk/server-config. Owner: ARRNC platform.

Expected files:

  • vars/projects.yml;
  • vars/portal-env-overrides.yaml;
  • deployments/arrnc-api.yml;
  • roles/clamav;
  • roles/gatekeeper/templates/services.conf.j2 and container.env.j2;
  • roles/verify/tasks/check_clamav.yml, check_gatekeeper.yml, check_containers.yml;
  • config-service record/reconcile tests.

Declare syrf-pdf-agent-staging and its scanner/network with:

  • the same agent UID, /app/data/output and /app/data/scratch contract;
  • a staging-only broker secret/vhost/queue and storage/PublicBaseUrl values;
  • immutable image/digest, resource/PID/log/scratch limits, restart policy, and structured health;
  • managed/gated/paused state that cannot consume during the dry run;
  • separate network and scanner from production;
  • no changes to syrf-pdf-agent-production.

Tests assert manifest exhaustiveness, env completeness, immutable digest, no secret values in rendered output, isolated network, no ports/privilege/Docker socket, exact volume, paused consumer, and production byte-for-byte semantic stability.

Proof: a no-op production diff and a staging-only check-mode plan. Rollback: remove inactive staging desired state and scanner/network if unused. STOP: config-service cannot represent paused consumption independently of container readiness.

Before activation, the IIS owner also proves that the existing Camarades mapping serves only the exact staging and previews/pr-N children without directory listing, scratch, sibling, or unscanned content. If it does not, add a separately reviewed declarative IIS-owner change to this wave; do not restore the agent-chart static server as a shortcut.

Step 4 — Extend SyRF's runtime and deployment contract

Repository: camaradesuk/syrf. Owner: SyRF maintainers.

Expected files:

  • src/services/pdf-agent/SyRF.PdfAgent/Program.cs and Configuration/PdfAgentOptions.cs;
  • storage, clamd-readiness, broker-binding, lifecycle, and health tests under src/services/pdf-agent/SyRF.PdfAgent.Tests;
  • src/services/pdf-agent/SyRF.PdfAgent/Dockerfile and appsettings.json;
  • .github/workflows/ci-cd.yml and reusable workflow/scripts;
  • shared environment mapping for BulkPdfStorageSettings and public URL.
  • BulkPdfUploadController/new-session admission configuration and active-session tests in the PM API, so maintenance can reject initiation without disabling completion/cancellation endpoints.
  • the S3 notifier's durable release-gate/outbox boundary and its object-version/job reconciliation tests, so preview notification cannot outrun the committed Uploaded job.

Implement only differences necessary for one ARRNC runtime contract:

  1. explicit consumer paused/enabled desired state with no queue subscription while paused;
  2. health output for root/sentinel/mount capability, clamd/signature age, broker identity, consumer ownership, scratch/free space, and journal recovery;
  3. orderly fence/drain/stop and idempotent restart behavior;
  4. immutable digest deployment input and previous-digest rollback;
  5. environment-scoped queue/user/vhost validation and exact root propagation;
  6. resource-compatible read-only/non-root container behavior;
  7. authenticated staging E2E harness using synthetic project/file names and redacted proof output.
  8. a maintenance admission gate that rejects only new multipart sessions while already accepted sessions retain signing, acknowledgement, heartbeat, completion, cancellation, and abandonment until they quiesce.
  9. a trusted, durable notifier release gate keyed by environment, object identity/version, and job ID; it releases only after a read proves the matching job is committed as Uploaded, and it reconciles held events idempotently after crashes or retries.

The release image is identical for staging and production. Environment configuration remains external. The PR does not deploy or enable any environment.

Tests:

  • unit and integration suites for paused startup, single queue binding, duplicate/redelivery, journal replay, mount/clamd/broker loss, disk threshold, and shutdown with an active claim;
  • API tests proving the admission gate rejects initiation but does not turn accepted-session lifecycle endpoints into 404s, plus a cutover drain with an active multipart upload;
  • notifier integration tests that force object completion before job persistence, prove the event remains held, then prove one idempotent release after Uploaded, including crash/retry recovery;
  • container image scan/build and non-root/read-only checks;
  • workflow validation that deploy resolves and records a digest and cannot target production from a staging action.

Rollback: revert the SyRF PR; no live effect before Step 8. STOP: the agent cannot become ready without consuming, or cannot drain/fence deterministically.

Step 5 — Build the bounded preview-family registrar

Primary repository: camaradesuk/server-config. Supporting repository: camaradesuk/arrnc-api-deploy only if the existing execution protocol cannot carry the typed operation. Owners: ARRNC config-service and gatekeeper maintainers.

Expected server-config files:

  • config-service/src/ConfigService/Api/ApiEndpoints.cs and authorization;
  • config-service/src/ConfigService/Operations and Record;
  • config-service/src/ConfigService/Schemas/deployment-manifest.schema.json;
  • config-service/src/ConfigService/Provenance;
  • config-service/src/ConfigService/Reconcile;
  • config-service tests for API, authorization, records, provenance, races, and reconciliation;
  • deployments/arrnc-api.yml and root-controlled templates.

Expected arrnc-api-deploy files if needed:

  • scripts/container-web-deploy and container-web-apply;
  • scripts/install-gatekeeper;
  • config/services.conf.example;
  • shell/PowerShell contract tests.

Implement typed create/pause/resume/retire operations whose only caller-controlled deployment values are canonical PR number, verified SyRF digest/source, desired operation, and idempotency key. Add a typed output-inspect operation whose only additional inputs are persisted preview job ID and expected derived relative output path. Derive every name/path/network/secret/resource value. Verify trusted repo/workflow, open PR, digest attestation, fleet capacity, record ownership, and explicit authorized-maintainer approval bound to the exact current head SHA and digest. Invalidate activation on every synchronize event, recheck the binding immediately before registration, serialize by PR, and audit each transition. Before creating any state, atomically reserve CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota/allocation, ports/routes, network, RabbitMQ, and scanner capacity from a fleet-wide ledger. The configurable concurrency ceiling derives from measured ARRNC capacity and may exceed one; requests queue without partial resources when the aggregate safe budget is unavailable. The registrar uses only the storage owner's scoped provisioning/allocation capability and verifies its audit result. Immediately before a queued request reserves resources and provisions anything, one atomic admission check must revalidate the exact head repository, open/enabled PR state, current head SHA, verified digest, and current authorized-maintainer approval; a stale or ineligible request is discarded with no partial state.

Security tests must prove rejection of arbitrary repositories, tags, commands, paths, ports, networks, env names, secret scopes, every head repository other than the SyRF base repository, stale/closed PRs, per-slot profiles or requested concurrency above hard limits, digest drift, record takeover, traversal, replay with conflicting input, and persistent-slot retirement. They must separately prove that transient aggregate-capacity exhaustion within valid limits queues with no partial state and later admits after capacity is released only after a fresh atomic eligibility and provenance check. Tests close, disable, synchronize, change the digest, and revoke approval while queued and prove none can reserve or provision. Host network tests must also prove deny-by-default preview egress: allow only the exact preview scanner plus minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations; deny host gateways, metadata endpoints, ARRNC/UoE internal CIDRs, persistent environments, and arbitrary internet routes. Root-controlled per-preview tests enforce ingress/egress rate and burst ceilings even for allowed presigned-S3 traffic and prove simultaneous bounded transfers cannot exhaust the ARRNC uplink or degrade production/staging health.

Each explicitly approved same-repository PR receives its own disposable rootless Docker container on ARRNC. Root-controlled templates enforce the pinned digest, non-root UID, dropped capabilities, no-new-privileges, read-only root filesystem where supported, seccomp/AppArmor, private network namespace, no Docker socket, no host namespace/device/admin access, and no production/staging mount, credential, route, or service identity. Any pull request whose head repository is not exactly the SyRF base repository, including every fork, is rejected, as is a stale, closed, or unapproved same-repository head. VM/microVM isolation and /dev/kvm are not prerequisites for this approved-workload policy.

Root applies immutable per-preview and aggregate CPU, memory, PID, block-I/O, runtime-disk, scratch, storage-quota, port/route, network, broker, and scanner ceilings. Bounded write/discard/fsync and simultaneous-preview tests prove those ceilings and acceptable production/staging health.

ARRNC root mounts the exact SMB child and bind-mounts only that child into the matching rootless container. Tests prove exact source, no-follow behavior, sibling/common-root denial, mount-loss fail-closed behavior, cross-preview denial, and ordered container stop before host unmount. SMB credentials never enter any preview container.

The output-inspect operation accepts a short-lived preview-family harness identity only for its registered PR/job/path. Root resolves the expected path no-follow beneath the exact child, enforces byte/rate limits, and passes it to an isolated trusted type/size validator and scanner. It returns only redacted size/hash/scan proof: no raw bytes, listing, arbitrary path, storage credential, or mount capability reaches the workflow. Audit/replay tests bind the result to the preview record; teardown revokes the capability before removing the container and child/allocation.

The scanner updater has a distinct trusted identity and an allowlist only to a pinned internal signature mirror. Tests prove refresh, mirror failure, stale-signature fail-closed behavior, and that the PR agent cannot reach the mirror.

Registry/host tests must reject a manifest compressed total or signed build/verifier unpacked-size attestation above reviewed ceilings, insufficient Docker data-root headroom, and teardown of a layer referenced by another slot. Reconciliation must reclaim proven preview-only layers, detect leakage, and avoid broad pruning when ownership is ambiguous. Broker tests must drive simultaneous preview clients through connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits and prove persistent-environment health is unchanged; requests queue when aggregate capacity is unavailable, or a separately reviewed dedicated preview broker supplies the boundary.

arrnc-api-deploy must remain a constrained actuator. If the typed policy cannot stay in server-config, STOP for architecture review rather than duplicate it.

Rollback: disable preview-family API and reconcile no new records; existing preview records remain paused for explicit safe teardown. Persistent slots are unaffected.

Step 6 — Wire trusted PR preview lifecycle

Repositories: camaradesuk/syrf and camaradesuk/server-config. Owners: SyRF CI and ARRNC platform.

Expected SyRF files:

  • .github/workflows/pr-preview.yml;
  • .github/workflows/_preview-gitversion.yml;
  • .github scripts that detect service changes and clean preview resources;
  • CLAUDE.md preview runner/security boundary.

Activation requires an eligible same-repository PR and explicit preview configuration. The workflow that executes PR code builds the PDF-agent image, resolves its digest, and publishes only the attestation. It receives no ARRNC, registrar, storage, broker, or protected-environment credential. A separate protected workflow loaded from the base branch, using pinned dependencies and a short-lived preview-family-only identity, re-verifies PR state, capacity, and an approval/configuration record for the exact head SHA and digest before calling the registrar. Every synchronize event invalidates that record, and activation rechecks the binding immediately before registration. It never checks out or executes PR code, then waits for paused readiness/isolation proof before enabling the preview's synthetic-only Bulk consumer. Forks never reach the ARRNC path.

The protected harness alone uses a synthetic preview identity and protected API path to create the matching Project, initiate the persisted upload session/job, upload and acknowledge every approved non-sensitive multipart fixture part, and complete the session. Only after the durable Uploaded job records the exact environment, size, hash, and policy required by BulkPdfUploadClaimConsumer may the trusted notifier release gate publish the exact object/version event. The gate must already have durably captured an early S3 event, survive a crash, and release it once and only once after the committed job is visible. External policy disables human upload initiation for previews. PR code receives no selector or credential for production, clinical, participant, or other sensitive objects.

Cleanup on close/config removal invokes the idempotent registrar teardown, removes the exact preview image reference and only proven unreferenced preview layers, revokes the output-inspect capability, and waits for durable container, image, credential, mount, storage, and record absence. It removes the disposable rootless container, writable runtime state, per-preview network, routes, ports, broker identity, scanner allocation, and exact storage allocation. It atomically releases only resources whose absence is proved. Any unsafe remainder stays charged to the owning reservation, or moves atomically to an explicit quarantine debit against the same aggregate fleet budget, until reclamation is proved. An hourly server-side reconciler covers missed GitHub events and alerts at one hour; the 24-hour SLO is a hard incident threshold. A later preview creates a fresh container from its verified immutable digest with fresh identities and allocations; runtime state is never reused across PRs without the reviewed reset, fencing, and identity-rotation contract.

Tests:

  • workflow trigger/permission/secret-flow static validation;
  • trusted versus fork event matrices;
  • proof that a same-repository PR modifying either workflow cannot obtain the activation identity or replace privileged base-branch code/dependencies;
  • synchronize/stale-approval tests proving a new head SHA cannot reuse a prior opt-in, review, or digest binding;
  • proof that PR code cannot alter the host egress policy and cannot reach host gateways, metadata, ARRNC/UoE internal CIDRs, persistent-environment endpoints, or arbitrary internet destinations;
  • simultaneous bounded-transfer proof that root-controlled per-preview and aggregate ingress/egress rate and burst ceilings protect the shared ARRNC uplink and persistent-service health;
  • simultaneous bounded-I/O proof that host-enforced per-preview and aggregate block-device bandwidth/IOPS ceilings cover writable runtime data and scratch without degrading production/staging storage health;
  • simultaneous-client proof that broker connection/channel, message-size, publish-rate, queue-length/bytes, storage, and scanner limits fail closed without degrading persistent environments or another preview;
  • oversized-image and accumulated-layer proof for activation ceilings, reserved Docker headroom, preview-only reclamation, and refusal to prune persistent/shared layers;
  • container removal/recreation proof showing that a surviving process, stale identity, network, mount, message, route, allocation, or runtime state cannot cross PR assignments;
  • synthetic-only input tests rejecting human upload initiation and every object/prefix outside the protected preview fixture set, proving multipart upload/acknowledgement/completion produces the matching synthetic Project and durable Uploaded job with environment/size/hash/policy before the release gate publishes; force the S3 event to arrive first and prove durable hold, crash recovery, and one idempotent post-Uploaded publication, with no sensitive content in proof artifacts;
  • proof that IIS/public routes deny the preview child and the bounded host-side inspect operation accepts only the registered PR/job/derived path, returns redacted type/size/hash/scan proof, and cannot list/read siblings, return raw bytes, leak credentials, or survive teardown;
  • create/close/reopen/retag/reseed races and duplicate events;
  • atomic fleet-admission contention proving requests queue with no partial state when any safe aggregate budget is unavailable;
  • at least two simultaneous previews proving distinct mounts, credentials, names, networks, routes, messages, inspection capabilities, and cleanup authority;
  • independent failure, cancellation, and orphan reconciliation proving one preview cannot inspect, stop, revoke, unmount, delete, or release resources owned by another preview;
  • configuration tests proving the concurrency ceiling derives from measured capacity rather than a hard-coded single slot;
  • exact cleanup denial for another PR and persistent environments;
  • broker/secret revocation and mounted-child absence.

Rollback: disable new activation, enumerate and fence/retire every admitted preview record, retain quarantine debits until absence is proved, and restore the previous workflow. STOP: PR code can reach production/staging secrets, networks, mounts, scanner, or Docker control.

Step 7 — Review implementation before any apply

All Wave 1 and 2 PRs must be green and review-clean. Attach a redacted cross-repo matrix showing each ADR requirement, implementation diff, automated test, operator proof, and rollback command.

The user reviews:

  • exact server-config check-mode diff;
  • production no-change evidence;
  • capacity/quota/backup owner sign-off;
  • registrar threat model and negative tests;
  • staging cutover and restore runbooks;
  • proposed maintenance window and named operators.

Merging code/config does not authorize apply. Gate B is requested only after Step 8 dry-run evidence exists.

Step 8 — Apply ARRNC staging paused and collect dry-run proof

Repositories: no new design change; execute the reviewed server-config workflow. Owner: authorized ARRNC operator. Environment: staging slot only.

Preflight:

  • verify hostname and inventory target;
  • record current production semantic state and live Kubernetes staging state;
  • verify queue/claims without changing them;
  • verify exact child/ACL/quota/backup classifications externally;
  • confirm that pre-cutover rollback stops/removes the inactive staging slot; identify a previous digest only if it is independently proven to support paused startup with no queue subscription.

Apply the staging mount, scanner/network, secrets, and container with consumer paused and Bulk intake still served by Kubernetes. Run mount/source/sentinel/sibling denial, quota, EICAR, signature freshness, broker-auth-with-zero-consumer, public URL isolation, resource, restart, log, alert, and stopped/absent rollback. Do not start a legacy digest during this dry run. A previous-digest rehearsal is allowed only after that digest has separately passed the paused, zero-subscription contract.

Required evidence:

  • production semantic diff empty;
  • ARRNC staging desired equals reconciled digest/config;
  • queue consumer remains the existing Kubernetes consumer only;
  • no writes outside the staging child;
  • no production credential reachable by the staging container;
  • rollback removes or stops the inactive slot without changing the sole Kubernetes consumer; if a compatible prior digest exists, its optional rehearsal remains paused with zero subscriptions.

STOP and roll back the new inactive slot on any mismatch. Do not touch Kubernetes.

Step 9 — Gate B: approve staging activation

The user explicitly approves or rejects the live handover using Step 8 evidence, the final cluster-gitops drain diff, the maintenance window, and named rollback owners.

Approval covers only:

  • temporary staging Bulk intake freeze;
  • Kubernetes staging PDF-agent scale-to-zero with PVC/Application/ingress retained;
  • ARRNC staging consumer activation;
  • authenticated staging proof and soak;
  • later retirement steps if their evidence gates pass.

It does not cover production.

Step 10 — Perform the single-owner staging cutover

Repositories/operations: cluster-gitops drain PR plus reviewed ARRNC state transition. Owners: GKE and ARRNC operators together.

  1. Close only new staging Bulk session admission while existing multipart sessions retain sign/acknowledge/heartbeat/complete/cancel/abandon paths.
  2. Prove active upload sessions quiescent, pause new notifier publication, and capture a durable inventory binding S3 objects to outbox/event/job state. Require every accepted job terminal; Uploaded or another known nonterminal state is not a drain.
  3. Drain ready/unacknowledged messages, claims, and journal with exactly one Kubernetes consumer, then enumerate/checksum and perform an initial PVC transfer through the reviewed controlled path into the ARRNC staging child, and prove canonical-path, checksum, collision, stored-reference, and candidate IIS-read equivalence. Stop on any unexplained difference.
  4. Switch the reviewed staging PublicBaseUrl/routing configuration to the ARRNC child while the Kubernetes static server still runs, and prove existing and newly generated references resolve from ARRNC.
  5. Merge/sync a narrowly scoped cluster-gitops change that sets the staging PDF-agent Deployment to zero while retaining its Application, ingress, Service, PVC, and values.
  6. With the old consumer fenced, perform a final delta transfer, prove no post-fence PVC writes, and reconcile every S3 object/outbox entry with terminal state or a preserved durable queue event. Prove zero consumers and unchanged queue/claim/read invariants.
  7. Enable ARRNC staging consumer, prove exactly one consumer and correct environment identity, then resume notifier publication.
  8. Reopen staging new-session admission and execute the authenticated proof suite.
  9. Observe through the agreed soak with queues, claims, errors, mount, clamd, resources, and public reads monitored.

Authenticated proof covers clean and EICAR uploads, S3/notifier/Rabbit/agent/PM/SignalR/CSV/Study/ IIS path, duplicate delivery, process kill, clamd outage, broker outage, mount outage, disk threshold, queue recovery, and idempotency. It also proves previous-digest rollback only when an independently verified compatible ARRNC digest exists; for the initial activation it instead proves stopped/absent pre-cutover rollback and the retained Kubernetes rollback path after cutover.

Rollback:

  1. intake off;
  2. fence ARRNC and drain or establish safe recoverability;
  3. prove zero ARRNC consumers;
  4. enumerate ARRNC output created since cutover and reconcile it back to the retained PVC through the reviewed inverse transfer, with checksum/collision and stored-reference proof;
  5. scale Kubernetes back to one by reverting GitOps and prove the static server ready against the reconciled PVC;
  6. restore the previous PublicBaseUrl/routing configuration;
  7. prove sole Kubernetes consumer, PVC health, and old/new link continuity;
  8. intake on and monitor.

STOP: any ambiguity in consumer, claim, journal, queue, storage, or rollback ownership.

Step 11 — Retire Kubernetes staging reversibly

Repository: camaradesuk/cluster-gitops. Owner: GKE platform. Dependency: Step 10 accepted soak.

PR 11A keeps the Deployment at zero, disables/removes ingress exposure, and preserves the PVC and rollback metadata. Before merge, enumerate/checksum PVC contents and reconcile them against ARRNC staging. Create a VolumeSnapshot or equivalent controlled archive and prove restoration into an isolated claim. Record owner, expiry, and restore instructions.

PR 11B, after the retention/soak gate, removes:

  • syrf/environments/staging/pdf-agent/config.yaml and values.yaml;
  • the generated Argo CD Application and ingress/Service/Deployment ownership;
  • obsolete preview/production PDF-agent Kubernetes configuration;
  • the PVC only after snapshot/archive proof and the explicit expiry checkpoint.

ApplicationSet effects from argocd/applicationsets/syrf.yaml and syrf-previews.yaml must be shown in rendered diff. Argo orphan warnings are reconciled; no manual delete is used.

Tests/proof:

  • Helm/ApplicationSet render before/after;
  • Argo dry-run/diff and sync health;
  • queue/claim/ARRNC consumer evidence before and after;
  • snapshot restore and file checksum proof;
  • Git revert plus restore recreates the prior staging shape without queue consumption until explicitly enabled.

Rollback: fence ARRNC, revert GitOps, and restore the tested artifact without enabling the old consumer or route. Inventory every ARRNC file written after artifact capture, perform a collision-safe checksummed inverse transfer into the restored claim, and prove every current BulkPdfDeliveredPath/public reference resolves there before enabling the old static route and sole Kubernetes consumer. STOP: no restorable artifact, unexplained file difference, queued/in-flight work, or Argo prune outside the expected resource set.

Step 12 — Remove obsolete SyRF Kubernetes authority

Repository: camaradesuk/syrf. Owner: SyRF maintainers. Dependency: Step 11 accepted.

Remove the PDF-agent Helm deployment target and PVC/static-server tests/templates if no longer used, delete preview GitOps deployment outputs for this service, and update:

  • src/services/pdf-agent/.chart;
  • .github/workflows/pr-preview.yml and CI chart packaging paths;
  • CLAUDE.md;
  • ADR-012 and the 2026-08-11 design/plan supersession notices;
  • feature and operator documentation.

Keep the Docker image, runtime tests, ARRNC deployment, and application storage-authority contracts. Documentation validation and a repository-wide search must find no operative statement that directs a PDF agent to Kubernetes/PVC.

Rollback: restore documentation/chart only if Kubernetes rollback is explicitly selected. STOP: another active workflow still consumes the chart or cluster-gitops still references it.

Step 13 — Prove concurrent dynamic previews end to end

Repositories/operations: trusted SyRF PR and ARRNC preview-family registrar. Owners: SyRF CI and ARRNC operator. Dependency: Waves 1C and 2 green; staging accepted.

Use at least two non-sensitive, explicitly approved same-repository PR fixtures concurrently:

  1. opt in, approve, build, attest, and register each immutable digest while proving atomic reservation and queueing when the configured aggregate budget is exhausted;
  2. prove each exact previews/pr-N ACL/sentinel/quota and sibling/cross-preview denial;
  3. prove isolated network/scanner/broker credentials, per-preview and aggregate resource caps, and host-enforced block-device bandwidth/IOPS ceilings under simultaneous bounded I/O;
  4. prove IIS and all public routes deny every preview child, then use the protected harness for each fixture to create the matching synthetic Project, initiate/upload/acknowledge/complete every fixture part, force notification to reach the durable release gate before job persistence, prove it remains held until Uploaded and then publishes exactly once, and run clean and EICAR authenticated preview uploads; use each scoped host-side inspect operation to obtain redacted type/size/hash/external-scan proof for malformed output without publishing or returning its bytes;
  5. while one preview remains healthy, fail the other and prove isolation, then recover it; while the recovered preview remains healthy, cancel the first and prove that containers, mounts, credentials, names, networks, routes, messages, inspection, queues, storage, and cleanup effects remain confined to their owning PR and persistent staging remains healthy, then recreate the cancelled preview from its immutable digest and fresh identities;
  6. deliberately suppress cleanup for one preview, prove the reconciler completes only that orphaned preview while the other remains healthy, and prove any quarantined remainder stays charged to the fleet budget;
  7. close/disable every remaining fixture and prove consumer fence, queue/claim convergence, credential revocation, inspection-capability revocation, container/network/runtime removal, unmount, exact-child retirement, record removal, verified absence, and only then the corresponding atomic fleet-reservation release.

No production/staging secret-name material may appear inside the preview container even if values are redacted. The preview must be unable to connect to persistent agent/scanner networks.

Rollback: disable preview activation globally; enumerate, fence, and safely retire every fixture admitted by the concurrent rehearsal; retain any quarantined resource debit until absence is proved. STOP: cleanup exceeds 24 hours, cross-preview or persistent-environment isolation fails, fleet admission creates partial state, or the registrar needs broader authority.

Step 14 — Prepare production as a separate future phase

Do not execute under this plan. After staging and preview acceptance, write a new production promotion plan using current evidence. It must include production child/backup proof, queue/claim handover, existing-file/IIS compatibility, capacity, maintenance/rollback owners, previous digest, observability, and an explicit Gate C request.

No prior approval, merged implementation, staging apply, or preview success implies Gate C.

4. Cross-repository acceptance matrix

Requirement Automated proof Operator proof Activation blocker
Exact child isolation canonical/path/ACL negative tests direct mount, sibling denial, no-follow write/delete any root/sibling access
No production credential leakage rendered-secret scope and workflow tests container env/mount/network inspection any persistent secret reachable by PR code
Immutable image parity digest/provenance tests reconciled digest; compatible prior-digest rollback when one exists, otherwise stopped/absent and retained-Kubernetes rollback mutable tag or unverified commit
ClamAV parity readiness/EICAR/signature tests clean + EICAR, signature-age alert stale/unreachable scanner
RabbitMQ isolation vhost/queue/user and paused-state tests consumer count, ready/unacked/error/claim evidence more than one or wrong consumer
Resource/capacity declared-limit and concurrency tests ARRNC free capacity, quota and failure alert unbounded slot or missing quota
Failure recovery journal/redelivery/idempotency tests process/mount/broker/clamd/disk rehearsals duplicate effect or ambiguous claim
Kubernetes rollback rendered GitOps/snapshot tests isolated snapshot restore and checksum untested restore or unexpected prune
Preview teardown race/idempotency/path tests event and orphan-reconciler teardown residue, credential, mount, or record remains
Production untouched semantic-diff test before/after inventory evidence any production change

5. Required PR metadata and evidence

Every implementation PR states:

  • its wave, dependencies, repository owner, and exact environment effect;
  • files and contracts changed;
  • tests and negative tests run;
  • whether merge triggers an apply (default: no);
  • rollback and STOP conditions;
  • production impact, which must be none before Gate C;
  • links to evidence using redacted identifiers.

Operational evidence records timestamps, commit/digest, desired/reconciled state, consumer/queue/ claim status, checks performed, outcome, and named operator/rollback owner. It never records credentials, participant information, project names, or uploaded filenames.

6. Phase gates and completion

Phase 1 is complete only when this planning PR:

  • contains the superseding ADR, design, numbered plan, and stale-authority cleanup;
  • passes documentation/contract checks;
  • has no unresolved valid review findings;
  • is green and review-clean;
  • is not merged;
  • is presented for explicit Gate A approval with the exact Phase 2 waves.

Phase 2 begins only after Gate A. It stops again at Gate B before live staging cutover. Production is a future separately planned phase that stops at Gate C.