Release and recovery
Publish through CI, verify the live release and respond to operational incidents.
Repository review: . Public operational reference; acceptance remains to be recorded by the receiving owner.
Release through CI and verify the result
- Prepare a scoped change on main. Read before editing, preserve unrelated work, stage only the requested slice and use a meaningful conventional commit. Check that generated files and publication boundaries match the intended change.
- Run local checks. Complete the relevant validation checks and inspect the final diff. Repository working notes, confidential evidence and exploratory reports must not become public routes incidentally.
- Push under the applicable release authority.
make deploypushesmain; the AWS caller workflow selects the validation lanes. It does not upload the localdist/. - Observe the whole release. Follow
.github/workflows/deploy-aws.ymlandsite-release.yml. Required contracts and the release candidate must both succeed before the exact validated artifact can deploy. - Check the live identity and behaviour. Read the published
/release.jsonand compare itssourceCommitwith the intended commit. Check the changed page, critical navigation, dependent assets and any affected service. Record the release/run identity and the observed result.
The deployment uses GitHub OIDC to assume the approved AWS role. It publishes content-addressed assets first, then changed mutable objects and targeted CloudFront invalidations. Old immutable assets remain available to already-open pages. Do not assign immutable cache policy to HTML, manually copy a different build over the release, or delete the prior assets as routine cleanup.
Infrastructure changes have a separate infra.yml pipeline. Its ordered CloudFormation deployments and runtime packaging are part of the implementation. Bootstrap resources are deliberately outside that pipeline. A coupled infrastructure/site change needs the documented order and checks in both workflows.
Incident triage and recovery
First distinguish a static-site problem from a participant-service incident or a local tool failure. Capture the release commit, affected feature, start time and non-sensitive error evidence. Assign an incident owner. Preserve participant data and previous immutable assets; a redeploy of HTML cannot restore a stateful service's data.
| Symptom | First checks | Recovery boundary |
|---|---|---|
| Release failed before deployment | Read the failing job and retained failure evidence. Confirm whether deployment ran and compare the live release identity. | Fix the failure and rerun the normal pipeline. Do not assume a failed build changed production. |
| Bad page or client behaviour after release | Compare live and intended commits, changed routes, asset responses and cache evidence. | Revert the specific faulty change on main, preserving later accepted work, then validate and release through CI. Verify the resulting live commit and behaviour. |
| Local dev or model query fails | Check the owned server, selected port, Node/pnpm, Java/Python, upstream input revision and tool output. | Restart only the owned process or rebuild the affected projection. Do not start another server against shared state or change production to fix a local issue. |
| Signup, approval, acknowledgement or onboarding fails | Check the corresponding function, queue/dead-letter state, logs, provider health and enablement configuration with authorised access. | Use the owning service's idempotent retry/review path. For an onboarding operation parked for attention, use the operator review described below. Verify each real postcondition before resending messages or declaring provisioning complete. |
| Comments unavailable or data recovery needed | Check gateway/session services, the Artalk task and persistence health. | Preserve the SQLite/Litestream persistence model and single-writer invariant. Restore only under an agreed, tested data-recovery plan. |
| Infrastructure or identity access failure | Check the last infrastructure run, CloudFormation events, owner access and the relevant provider integration. | Use the authorised infrastructure owner. Do not delete/recreate stacks, alter trust or rotate shared credentials as speculative troubleshooting. |
Rollback implementation limit: there is no general one-command production/data rollback established by this handover. The normal site recovery is a reviewed forward commit or scoped revert followed by the same CI release. Site artifacts currently have short retention, so do not assume an old workflow artifact remains downloadable. A revert that affects infrastructure, stored data or third-party side effects needs a separate recovery decision.
Backup evidence: checked-in templates enable point-in-time recovery on key registration/participant stores and versioning on relevant persistence buckets. A retained migration-evidence bucket is explicitly not an automated backup service. Confirm live settings, retention, recovery-point/recovery-time objectives and a successful isolated restore with the operator. Repository code and template declarations are not restore-test evidence.
Alert delivery: the operations stack wires an SNS alarm topic; email subscriptions must be confirmed. An existing topic or configured alarm does not prove somebody receives notifications. Transfer the recipient ownership and demonstrate delivery, including the agreed Teams route where used.
Onboarding review and reconciliation
scripts/onboarding-review.mjs is the existing operator interface for uncertain SharePoint grants. With authorised AWS/Microsoft access and the approved credential profile, node scripts/onboarding-review.mjs inspect <operationId> reads the operation and encrypted receipts and compares uncertain grants with live membership. It does not change the operation; its output belongs in the restricted incident record.
The same script's resolve operation is a write: it records a human verdict for grant-contributor or grant-index and requeues the operation. Use absent only when the grant did not land, owned when evidence establishes this workflow owns it, and manual for a grant it must preserve. Read the script's argument contract and the live evidence before an authorised resolution; a successful membership lookup alone does not establish ownership. Recheck the resulting operation and intended postconditions.
config/aws/hubspot-approval-stack.yaml defines an enabled reconciliation rule every 15 minutes. Verify that rule, its invocation permission, worker health and recent outcomes in the receiving account. The template does not prove that reconciliation runs today, and a scheduled retry does not replace human resolution of ambiguous effects.
Credential recovery: preserve the existing participant receiptEncryptionKey when rotating the Microsoft certificate. It protects the ownership receipts used for cleanup. The runtime credential loader checks certificate expiry before returning the key, so expired credentials can block cleanup and notices. Assign certificate-expiry monitoring and renewal to a named owner and deputy, retain key custody and recovery instructions privately, and verify receipt continuity after rotation. ADR-0085 records that independent recovery remains unverified; do not assume a new key can read old receipts.
Participant data, retention and queue replay
Agree who receives privacy requests and owns retention, expiry, withdrawals, erasure and provider suppression. Check the actual maintenance schedule and outcomes; documented policy is not evidence that a daily sweep is active. Participant approval, email preference, operational access and a marketing unsubscribe have distinct effects and must not be substituted for one another.
Inspect failed consumer events and their live source state before a bounded replay. The worker must recheck approval, expiry, deletion and suppression; do not restore an old grant or resend old messages from a queue snapshot. A data restore needs quarantine and reconciliation with subsequent erasures, withdrawals and security suspensions before anything returns to service. ADR-0084 records this boundary; complete the restricted retention/recovery runbook and demonstrate it with the data owner.