# Operations Administration is CLI plus declarative configuration only. There is no HTTP admin API, no default credential, and nothing is auto-created at startup. This is the de-facto admin interface and the runbook for its day-two tasks. ## Purpose For an operator: the full command surface, the declarative `config sync` file, the audit stream, key and certificate rotation, and backup and restore. ## The admin model - The `migrate` subcommand owns the schema and is the only writer of it. - The `serve` subcommand serves OAuth and never migrates and never provisions. - Every mutation (scopes, users, entitlements, redirect URIs, resource servers, client attachments, client access-token format) is audited on the `audit` tracing target. - Secrets are read from stdin, a file, or the environment, never from argv. A secret is printed only when the operator asked for a one-time value. ## Command reference The binary is `d-oauth-server` (`cargo run --bin d-oauth-server -- `). Exit codes: `0` success, `1` command failure, `2` usage error. ### serve and migrate ```bash d-oauth-server serve d-oauth-server migrate d-oauth-server config check ``` - `serve` loads config, connects Postgres and Redis, asserts the Redis durability contract, and runs the TLS public listener plus the plaintext health listener. It never migrates. It drains on `SIGTERM`. - `migrate` applies forward-only migrations under sqlx's advisory lock and prints `schema current: N known migration(s)` (`N` is the embedded count, not the delta just applied). - `config check` validates configuration and prints `config ok: issuer=...`. ### Users ```bash d-oauth-server user create --email [--admin] [--password-stdin] [--password-file ] d-oauth-server user enroll-totp --email [--show-secret] d-oauth-server user disable --email d-oauth-server user unlock --email d-oauth-server user grant-scope --email --scope d-oauth-server user revoke-scope --email --scope d-oauth-server user scopes --email ``` - `create` reads the password from stdin by default or from `--password-file`. The two sources are mutually exclusive. `--admin` sets `is_admin`. A created user always starts with `must_change_password = true`. A `--password-file` must not be world-readable (use mode `0600`); a world-readable file is rejected before it is read. Passwords are held in wiped-on-drop memory and never appear in logs, errors, or audit events. - `enroll-totp` generates an encrypted TOTP secret and prints the secret and the `otpauth://` URI exactly once. The secret prints only to an interactive terminal; a redirected stdout (pipe, file, or job log) refuses unless `--show-secret` is passed to acknowledge the disclosure. Register the secret in an authenticator now. - `disable` first purges all of the user's Redis ephemeral state (login sessions and their `:meta` sidecars, access/refresh tokens and their grants, authorization codes, device records with their `:cnf`/`:poll` sidecars and `user_code` mappings, and the lockout counters (account plus per-IP)), then flips the Postgres `disabled` flag. If Redis is unavailable the command fails and the user stays enabled (fail closed). Disabling alone does not touch PAR records, replay markers, or rate-limit counters, which carry no subject. - `unlock` clears the user's login lockout counters (the per-account counter and every per-IP composite bucket) without touching the account. The user must exist; a Redis outage fails closed and clears nothing. Unknown-label buckets carry no account label and are never touched. The per-IP sweep uses `SCAN`, so on Cluster topologies it covers the contacted node only and buckets on other nodes may survive; re-run against each node or wait out the window. - `grant-scope` requires the scope to exist in the catalog. - `revoke-scope` is idempotent. ### Scopes ```bash d-oauth-server scope create [--description ] d-oauth-server scope list d-oauth-server scope enable d-oauth-server scope disable d-oauth-server scope delete ``` `create` upserts: it creates a scope, updates its description, or re-enables a disabled one. `delete` deletes a scope that nothing references, and disables it when `user_scopes` references it, so entitlements are not silently destroyed. The enabled scope catalog is what `GET /.well-known/oauth-authorization-server` (discovery document) advertises as `scopes_supported`. ### Redirect URIs and client attachments ```bash d-oauth-server redirect-uri create [--kind authorize|post_logout] [--description ] d-oauth-server redirect-uri list d-oauth-server redirect-uri enable [--kind authorize|post_logout] d-oauth-server redirect-uri disable [--kind authorize|post_logout] d-oauth-server redirect-uri delete [--kind authorize|post_logout] d-oauth-server client attach-redirect-uri --client-id --uri [--kind authorize|post_logout] d-oauth-server client detach-redirect-uri --client-id --uri [--kind authorize|post_logout] d-oauth-server client redirect-uris --client-id d-oauth-server client set-access-token-format --client-id --format opaque|jwt ``` A redirect URI is a global catalog entry; a client attaches to it. `/authorize` matches only the client's attached and enabled `authorize` set. A private-use URI scheme must contain reverse-DNS dot notation (draft-16 §2.3.1): `com.example.app:/` is accepted and `myapp:/` is rejected at creation. Detaching or disabling a URI stops future redirects to it; already issued tokens are unaffected. Deleting a referenced URI disables it instead. Legacy rows predate the §2.3.1 dot check, so a catalog that still holds a dotless private-use URI (for example `myapp:/cb`) makes `config sync` fail: `redirect::upsert` validates the scheme (`src/cli/redirect.rs:170`) before the idempotency short-circuit (`src/cli/redirect.rs:178`, via `src/cli/sync.rs:168`), so the sync aborts with `invalid input` instead of reporting no changes. `redirect-uri disable` and `redirect-uri delete` bypass the upsert check, so disable or delete the legacy row first, then re-run the sync. `kind` selects the URI's role. `authorize` (the default) is an authorization-response callback. `post_logout` is a bare return target used only by `POST /logout` to send the browser back to the client after the session ends (`GET /logout` is read-only and issues no redirect). `POST /logout` redirects only while exactly one enabled `post_logout` URI is registered for the client and the client is active. Zero registered enabled URIs or more than one, and `POST /logout` performs no redirect: the target must be unambiguous or nothing happens. Registering a second enabled `post_logout` URI is a configuration error, not a silent last-writer-wins. ### Resource servers and client access-token format ```bash d-oauth-server resource-server create [--description ] d-oauth-server resource-server list d-oauth-server resource-server enable d-oauth-server resource-server disable d-oauth-server resource-server delete ``` A resource server is an RFC 8707 resource indicator (the `resource` parameter and the `aud` target of an access token). `identifier` matches exactly — no normalization, sub-path, or wildcard. It must be an absolute `https` URI with a host; a `*`, userinfo, fragment, or query is rejected, so one catalog row can never stand for an entire origin or a non-fetchable identifier. `create` upserts: it creates the row, updates its description, or leaves an unchanged row alone; it never silently re-enables an operator-disabled row, and `list` shows both states. `delete` removes the row outright (nothing references the catalog directly). Disabling or deleting an identifier stops new authorizations and, at mint time, refresh/device grants already bound to it: refresh fails `invalid_target` with the refresh token still live, device poll fails `invalid_target` with the code consumed, and auth-code redemption fails before consuming the code. Re-enable the identifier and the client re-authorizes (or retries the live refresh token). `client set-access-token-format --client-id --format opaque|jwt` is the OPERATOR-ONLY opt-in to JWT access tokens for one client; `opaque` (the OAuth 2.1 baseline and column default) is what every dynamically registered client gets. `--format` accepts exactly `opaque` or `jwt`, and `jwt` is honored only while JWT access tokens are enabled by configuration. Dynamic client registration can never set this: a registration (`POST /register`) body that carries `access_token_format` is rejected with `invalid_client_metadata`. The JWT signing key is never printed by any command. ### Declarative config ```bash d-oauth-server config sync --file d-oauth-server config check ``` `config sync` applies a JSON file idempotently. A second identical run reports `config sync: no changes`. See the format below. ### Admin bootstrap ```bash d-oauth-server admin bootstrap --email [--password-file ] [--force] ``` Creates the first admin under an advisory lock in one transaction. It refuses while any enabled admin exists unless `--force` is given (a loud warning and an audit event). The password comes from `--password-file`, from stdin, or — when neither yields a value — a generated 256-bit password that prints exactly once and only to a TTY (a non-TTY stdout with a generated password is an error, so the secret can never leak into job logs). A `--password-file` must not be world-readable (use mode `0600`). The full replacement procedure is in `RUNBOOK.md`. ### Initial access tokens (DCR) ```bash d-oauth-server initial-access-token issue --uses --ttl [--label ] [--show-secret] ``` Mints a one-time-use DCR initial-access token. The raw token is printed once; only its keyed hash and remaining-use budget are stored. The token prints only to an interactive terminal; a redirected stdout refuses unless `--show-secret` is passed to acknowledge the disclosure. The token is required when `allow_open_registration` is false. ### Key generation and health ```bash d-oauth-server keygen [--alg es256] d-oauth-server healthcheck [--url ] ``` `keygen` emits an ES256 keypair for test fixtures and client-side DPoP keys. It is never an authorization-server signing key; the server publishes no `jwks_uri`. `--alg` defaults to `es256` (the only algorithm). `healthcheck` probes the health listener over raw HTTP and exits 0 or 1; it is the container `HEALTHCHECK`. `--url` defaults to `http://127.0.0.1:8081/health/live` (liveness). ### PII and TOTP migration commands ```bash d-oauth-server pii backfill d-oauth-server pii verify d-oauth-server pii rotate d-oauth-server totp rewrap ``` All four take no options and read the key ring from configuration. `pii backfill`, `pii verify`, and `pii rotate` require a migration-role DSN (`D_OAUTH_DATABASE__URL`); they scan every row and print counts only, never a value. - `pii backfill` seals every remaining plaintext PII value, blanks the plaintext columns, and upgrades every TOTP blob to v3. It refuses while `pii_dual_write=true`, and must run with the app tier scaled to zero. - `pii verify` reports the four cutover checks: no plaintext remains, every `_enc` opens under a retained key, every TOTP blob is v3, and every stored blind index matches. It exits non-zero when any check is dirty. - `pii rotate` re-encrypts every `_enc` value and recomputes every blind index under the active PII key. Run it under a write freeze with the previous key retained. It is idempotent: a re-run re-encrypts only stale blobs. - `totp rewrap` re-wraps every v3 per-row DEK under the active TOTP key without touching the data ciphertext. It is idempotent. ### PII encryption rollout and cutover The expansion is additive. The backfill blanks the plaintext columns but this release does **not** drop them; dropping the now-inert columns is a later release. Follow this order exactly: 0. Apply the expansion migration with the PII key configured: `d-oauth-server migrate`. The migration drops `NOT NULL` on the plaintext columns and builds the new indexes, so run it in a maintenance window. See [DATABASE.md](DATABASE.md#expand-and-contract). 1. Set `pii_dual_write=true` (`D_OAUTH_PII_DUAL_WRITE=true`). 2. Roll every replica so all of them dual-write and emit v2 TOTP blobs. 3. Set `pii_dual_write=false` and scale the app tier to zero, so no writer runs during the backfill. 4. Run `d-oauth-server pii backfill`. 5. Run `d-oauth-server pii verify`; it must exit 0. 6. Scale the app tier back up. Startup readiness is fail-closed: if `pii_dual_write=false` and any blanked plaintext column still holds a value, `GET /health/ready` returns `503` and the replica stays out of service. While the rollout window is open, CLI `d-oauth-server user enroll-totp` and login-driven TOTP enrollment MUST run with `D_OAUTH_PII_DUAL_WRITE=true`. With it `false` they write v3 TOTP blobs, which a version N-1 replica cannot read, so the user's next TOTP check fails on an old replica. Keep dual-write on for the whole window; turn it off only at the scale-to-zero step above. ## Resource-server onboarding (RFC 9728) This is the operator-side companion to [RS-CONTRACT.md](RS-CONTRACT.md), the normative contract for the resource server (RS). The AS hosts no RS route and publishes no protected-resource metadata: onboarding an RS is a catalog step on the AS plus a publish, challenge, and rotation step the RS team owns. Do the four steps in order; the RS team can follow [RS-CONTRACT.md](RS-CONTRACT.md) from the document alone once the identifier and issuer below are fixed. ### 1. Register the resource identifier in the AS catalog The catalog is the RFC 8707 resource-indicator allowlist and the `aud` target of an access token. Register the RS's exact identifier (an absolute `https` URL, no fragment, no wildcard): ```bash d-oauth-server resource-server create https://api.example.com --description "Orders API" d-oauth-server resource-server list ``` `identifier` matches exactly, with no normalization, sub-path, or wildcard; a `*` anywhere is rejected at creation, as are a non-`https` scheme, a missing host, userinfo, a fragment, or a query, so one catalog row can never stand for an entire origin or a non-fetchable identifier. `create` upserts (it creates the row, updates its description, or leaves an unchanged row alone) and never silently re-enables an operator-disabled row. `list` shows both the `enabled` and `disabled` states. Use `resource-server enable|disable ` to flip a row and `resource-server delete ` to remove it. All four mutations are audited (`resource_server.create|enable|disable|delete`). The same steps are declarative through `config sync`; a `resource_servers` entry with the identifier, description, and `enabled` flag is applied idempotently (see [config sync file format](#config-sync-file-format)). For tokens to be audience-restricted, set `resource_indicators_enabled=true` so the AS honors the RFC 8707 `resource` parameter at the authorization, PAR, token, and device endpoints; otherwise the RS cannot rely on the `aud` check alone (RFC 9728 §7.4). See [CONFIGURATION.md](CONFIGURATION.md). ### 2. Publish the RS protected-resource metadata document The RS team publishes `GET /.well-known/oauth-protected-resource` at the resource host (RFC 9728 §3); when the identifier carries a path, the terminating slash is removed and the path suffix is appended (`/.well-known/oauth-protected-resource/resource1`). The document is served as `application/json` with a `200 OK` (RFC 9728 §3.2) and its `resource` member must be byte-for-byte identical to the identifier registered in step 1 (RFC 9728 §3.3, §7.3). The operator's job is to hand the RS team two values: - `authorization_servers`: the AS `issuer` value (for example `https://auth.example.com`). - `scopes_supported`: the runtime scope catalog this AS actually enables, read with `scope list`. It is a supported set, not a request-everything hint (RFC 9728 §7.2); advertising a scope the AS does not enable only misleads clients. The full member set and an example document are in [RS-CONTRACT.md](RS-CONTRACT.md#the-protected-resource-metadata-document). Optionally, advertise the RS identifier on the AS side through the RFC 9728 §4 `protected_resources` member: set `protected_resources_enabled=true` and list the identifier in `protected_resource_ids`. Both default off and empty, so the AS advertises no `protected_resources` until enabled. The load is fail-closed: the flag requires `resource_indicators_enabled=true` and a non-empty, duplicate-free list of absolute `https` identifiers with no wildcard, userinfo, fragment, or query. `resource_metadata` is never an AS metadata member (RFC 9728 §4, §5.1); it belongs only on the RS challenge in step 3. ### 3. Set the `WWW-Authenticate` challenge on the RS The RS returns the challenge on an unauthenticated or unacceptable request (RFC 9728 §5.1, RFC 6750 §3): ```http HTTP/1.1 401 Unauthorized WWW-Authenticate: Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource" ``` The parameter value is the absolute `https` URL of the document from step 2, derived from the same identifier; the RS never invents it ad hoc, and it is never sent alongside a success. When the RS later needs clients to pick up a changed metadata document, it re-sends a fresh challenge and clients re-fetch and re-validate on receipt (RFC 9728 §5.2). The challenge may be combined with other parameters (for example `max_age`, RFC 9470) and with schemes other than `Bearer` such as `DPoP` (RFC 9449). ### 4. Rotate the identifier and keys - Resource identifier: register the new identifier with `resource-server create` and keep the old row `enable`d until no live token references it, then `resource-server delete` the old identifier. The RS team re-publishes its metadata document with the new `resource` and re-sends the challenge; clients re-validate against the new value (RFC 9728 §3.3, §5.2). Never leave two enabled identifiers standing for the same deployment if the audience check is meant to be exact. - AS token pepper: rotating the pepper invalidates every live opaque token. For an RS that introspects (RFC 7662), that means re-authentication, not a silent accept; overlap the old pepper for at least `ttl.refresh_secs` (see [Key and certificate rotation](#key-and-certificate-rotation)). - JWT signing keys: when JWT access tokens are enabled, retain the previous `jwt_signing_previous_keys` across the rotation window so the RS can still verify in-flight tokens; the RS caches the AS JWKS with a positive lifetime and re-fetches on a fresh challenge (RFC 9728 §7.10, §5.2). Never rotate to an empty key set. - After any rotation, re-run the discovery flow end to end: call the resource with no token, read the challenge, fetch and validate the metadata, then re-authenticate. The full flow is in [RS-CONTRACT.md](RS-CONTRACT.md#discovery-flow). ## MCP client registration An MCP client needs a client identity at the authorization server. There are three supported paths, in order of preference. They are per-client choices; the one server-wide setting they share is whether the host `localhost` counts as an RFC 8252 loopback redirect (`allow_localhost_redirect`). ### 1. CIMD (primary, no registration) Client ID metadata documents let a client identify itself with an `https://` URL and skip registration entirely. 1. Set `cimd_enabled=true`. If the client redirects to `http://localhost:{port}`, also set `allow_localhost_redirect=true`. 2. Restart the server, then confirm the advertised capability: ```bash curl -s https://auth.example.com/.well-known/oauth-authorization-server \ | jq '{client_id_metadata_document_supported, token_endpoint_auth_methods_supported}' ``` The output must carry `client_id_metadata_document_supported: true` and `none` inside `token_endpoint_auth_methods_supported`. With `cimd_enabled` off the first member is absent, not `false`. 3. The client hosts a JSON document at an `https://` URL and uses that exact URL as its `client_id`. The document must validate. A minimal accepted example: ```json { "client_id": "https://mcp.example.com/oauth/client-metadata.json", "client_name": "Example MCP client", "redirect_uris": ["https://mcp.example.com/oauth/callback"], "token_endpoint_auth_method": "none", "grant_types": ["authorization_code", "refresh_token"], "scope": "api:read", "dpop_bound_access_tokens": false } ``` Every `scope` token must already be enabled in the runtime catalog (`scope list`). `client_id` must equal the URL byte for byte. The authorization server fetches the document on the first request and caches it for 15 minutes; a changed document is picked up after that window. No `/register` call and no initial-access token are involved. A `private_key_jwt` document must carry an inline `jwks`; `jwks_uri` is rejected. See [OAUTH.md](OAUTH.md#client-id-metadata-documents-cimd). ### 2. Open dynamic client registration This lets any caller register a client with `POST /register` and no token. It needs all of the following: - `allow_open_registration=true`. - A non-`catalog` `dcr_redirect_policy`: `catalog_loopback` or `open`. - For an `http://localhost:{port}` callback, `allow_localhost_redirect=true`. The default `dcr_redirect_policy = catalog` rejects any redirect URI that is not already in the redirect catalog, even when open registration is on. Choose `catalog_loopback` for catalog URIs plus dynamic loopback URIs, or `open` for any absolute `https` URI plus loopback. Literal `127.0.0.1` and `[::1]` are always loopback; `localhost` is loopback only with the flag on. Open registration is anonymous client creation, so prefer it only where that risk is acceptable. ### 3. IAT-gated dynamic client registration With `allow_open_registration=false` (the default), the client must present a one-time initial-access token: ```bash d-oauth-server initial-access-token issue --uses --ttl --label mcp ``` The raw token prints once, and only to an interactive terminal unless `--show-secret` is passed. The client calls `POST /register` with `Authorization: Bearer `: ```bash curl -s https://auth.example.com/register \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "redirect_uris": ["https://mcp.example.com/oauth/callback"], "token_endpoint_auth_method": "none", "grant_types": ["authorization_code", "refresh_token"], "scope": "api:read" }' | jq . ``` Each use authorizes one registration call, whether or not the metadata later validates. The redirect policy still applies: with the default `catalog` policy, the redirect URI must already be cataloged. For a `localhost` callback, use a non-`catalog` policy and set `allow_localhost_redirect=true`. There is no client-create command in the CLI. A client is created only by `POST /register` (with an IAT or under open registration), and the CLI manages existing clients through `client attach-redirect-uri`, `client detach-redirect-uri`, `client redirect-uris`, and `client set-access-token-format`. ## config sync file format The file is JSON. Every section is optional and defaults to empty. ```json { "scopes": [ { "name": "orders:read", "description": "Read orders", "enabled": true } ], "redirect_uris": [ { "uri": "https://app.example.com/callback", "enabled": true }, { "uri": "https://app.example.com/signed-out", "kind": "post_logout", "enabled": true } ], "resource_servers": [ { "identifier": "https://api.example.com", "description": "Orders API", "enabled": true } ], "user_scopes": [ { "email": "operator@example.com", "scope": "orders:read" } ], "client_redirect_uris": [ { "client_id": "example-client", "uri": "https://app.example.com/callback" }, { "client_id": "example-client", "uri": "https://app.example.com/signed-out", "kind": "post_logout" } ], "client_access_token_formats": [ { "client_id": "example-client", "access_token_format": "jwt" } ] } ``` Scope names must match the RFC 6749 scope-token grammar. `openid` is reserved. Redirect URIs must be absolute and are matched exactly at `/authorize` (or, for `kind: post_logout`, at `POST /logout`). `kind` defaults to `authorize`. Resource identifiers are exact RFC 8707 resource indicators; a `*` wildcard is rejected. `client_access_token_formats` sets the operator-only per-client access-token format (`opaque` or `jwt`); it is the same operator opt-in as `client set-access-token-format` (dynamic registration can never set it). Apply and re-apply: ```bash d-oauth-server config sync --file ./oauth-config.json d-oauth-server config sync --file ./oauth-config.json ``` The first reports the counts of what it created; the second reports no changes. Upserts use `IS DISTINCT FROM`, so an unchanged row does not bump `updated_at`. ## Audit events CLI mutations go to the `audit` target with `event = "cli.mutation"`, an `operation`, and a `subject`. The operations are `migrate`, `config.sync`, `user.create`, `user.enroll_totp`, `user.disable`, `user.unlock`, `user.grant_scope`, `user.revoke_scope`, `scope.create`, `scope.enable`, `scope.disable`, `scope.delete`, `redirect_uri.create`, `redirect_uri.enable`, `redirect_uri.disable`, `redirect_uri.delete`, `resource_server.create`, `resource_server.enable`, `resource_server.disable`, `resource_server.delete`, `client.attach_redirect_uri`, `client.detach_redirect_uri`, `client.set_access_token_format`, `initial_access_token.issue`, `pii.backfill`, `pii.rotate`, `totp.rewrap`, `admin.count.changed`, and the `admin.bootstrap.*` family. `pii verify` and `user scopes` are read-only and emit no mutation event. HTTP events on the same target include `login.success`, `login.failure`, `consent`, `token.issue`, `token.revoke`, `token.refresh_reuse_detected`, and `dcr.register`. No event carries a token, password, OTP, or secret. Tracing is JSON. Route it to your log pipeline and alert on `token.refresh_reuse_detected` and repeated `login.failure`. ## Key and certificate rotation ### Token pepper Opaque token hashes are stored under a version id, default `v1`, set by `token_pepper_id`. The key ring resolves a stored id newest-first across a retained chain, so verification keeps working while an old pepper is retained. The current loader supplies one active pepper; to rotate without invalidating live tokens, retain the previous key id for at least the refresh lifetime (`ttl.refresh_secs`, default 30 days) before removing it. New tokens are always minted under the active pepper. ### TLS certificate `serve` loads the certificate chain and key once at startup. Rotate by replacing the mounted material and performing a rolling restart; there is no hot reload, by design. ### JWT access-token signing key Reference: RFC 9068 §2.1, §4; RFC 7517 §5. This key is optional and off by default. It is unrelated to `keygen`, which emits ES256 material for test fixtures and client-side DPoP only; do not use `keygen` output here unless it happens to be exactly 32 bytes. `jwt_signing_key` is consumed as the literal UTF-8 bytes of the configured string, not decoded as base64 or hex. It must be exactly 32 bytes. Generate printable material with: ```bash openssl rand -hex 16 ``` That prints 32 hexadecimal characters, which are 32 bytes of key material. The inline value is trimmed of leading and trailing whitespace; a `*_FILE` mount is read verbatim except that trailing CR/LF are removed, so do not add spaces. An all-zero or out-of-range P-256 scalar still fails at load with `InvalidJwtSigningKey`. To enable the feature and mint JWTs for one client: 1. Set the environment: ```bash export D_OAUTH_JWT_ACCESS_TOKENS_ENABLED=true export D_OAUTH_JWT_SIGNING_KEY="$(openssl rand -hex 16)" export D_OAUTH_JWT_SIGNING_KID=v1 export D_OAUTH_JWT_ACCESS_TOKEN_SIGNING_ALG=ES256 ``` `D_OAUTH_JWT_SIGNING_KEY_FILE` mounts the same material as a file. The algorithm is `ES256` (default) or `EdDSA`; both use 32 raw bytes and there is no RSA option. Enabling the switch without a key fails at boot with `MissingJwtSigningKey`. 2. Roll the replicas and confirm the public key set: ```bash curl -fsS "$D_OAUTH_ISSUER/jwks.json" | jq -r '.keys[] | "\(.kid) \(.alg) \(.kty)"' ``` The current key appears first. While the feature is off, the route still answers `200` with `{"keys":[]}`; it is cached for 300 s. 3. Opt a client in, per client, only after the deployment is ready: ```bash d-oauth-server client set-access-token-format --client-id --format jwt ``` Registration can never set this; `opaque` remains the column default. To rotate: 1. Generate a new key and a new `kid`. 2. Make the new key active. Move the old key into `jwt_signing_previous_keys`, a list of `{kid, key}` objects supplied in the `D_OAUTH_CONFIG` file (the loader does not accept this list as one environment string): ```yaml jwt_access_tokens_enabled: true jwt_signing_key: "" jwt_signing_kid: v2 jwt_signing_previous_keys: - kid: v1 key: "" ``` Keep `jwt_access_token_signing_alg` unchanged unless every retained key is regenerated: all entries in the ring share that one value. 3. Roll every replica, so each one signs with `v2` and verifies `v1`. 4. Confirm both kids are published, current first: ```bash curl -fsS "$D_OAUTH_ISSUER/jwks.json" | jq -r '.keys[].kid' ``` 5. Keep the previous key until every token it signed has expired. Access tokens live `ttl.access_secs` (300 s by default); refresh tokens stay opaque and do not use this key. After that window, remove the previous entry and roll the replicas again. A JWKS document cached by a resource server can lag the removal by up to 300 s, which is harmless because no new token names the old kid. The old key verifies only. It never signs once it is in `jwt_signing_previous_keys`, and a duplicate kid (active or previous) is rejected at load. ## PII and TOTP key rotation Both the PII field-encryption key and the TOTP encryption key are versioned in the ring, and each has an optional retained previous key. Keep the previous key id configured until every value has been migrated, then retire it. A previous key without a distinct `*_previous_key_id`, or without its active key, is a load-time error. ### TOTP encryption key The TOTP blob carries its key id, and the ring resolves a retained TOTP key by id. To rotate: 1. Put the old key in `totp_previous_key` / `totp_previous_key_file` with a distinct `totp_previous_key_id`, and set the new `totp_encryption_key` / `totp_encryption_key_file` as active. 2. Roll the replicas so the ring loads both keys. 3. Run `d-oauth-server totp rewrap`. Every v3 blob is re-wrapped under the new key. Version 1 and 2 blobs have no DEK and still decrypt through the legacy path, so re-enroll those users with `d-oauth-server user enroll-totp` before the old key is dropped. 4. Remove the previous key once no blob names it. ### PII field-encryption key The blind index is key-dependent, so equality and uniqueness only hold within one key generation. Rotate under a write freeze: 1. Put the old key in `pii_previous_key` / `pii_previous_key_file` with a distinct `pii_previous_key_id`, and set the new `pii_encryption_key` / `pii_encryption_key_file` as active. 2. Roll the replicas, then scale the app tier to zero (the write freeze). No writer may run while the indexes are recomputed. 3. Run `d-oauth-server pii rotate`. It re-encrypts every `_enc` value and recomputes every blind index under the new key in bounded transactions; a duplicate across key generations aborts that batch instead of overwriting a row. It is idempotent and resumable because the previous key is retained. 4. Run `d-oauth-server pii verify`; it must exit 0. 5. Remove the previous key and scale the app tier back up. ## Backup and restore Postgres holds all durable state, so a normal backup covers it: ```bash pg_dump --format=custom "$DATABASE_URL" > oauth.dump pg_restore --clean --if-exists --dbname "$DATABASE_URL" oauth.dump ``` Restore into a migrated database, then run `migrate` to confirm the schema version matches the binary. Redis is ephemeral and need not be backed up: codes, tokens, sessions, and replay markers are re-created as clients re-authenticate, and a Redis loss does not lose clients, users, scopes, or redirect URIs. Registration access token hashes are in Postgres, so DCR client management also survives a Redis loss. ## Security properties - No HTTP admin surface and no default credential. - Every mutation is audited. - Bootstrap refuses a second admin without `--force`, which is loud and audited. - A restricted session (forced password change or missing TOTP) cannot reach `/authorize`, the OAuth endpoints, or admin-gated actions. - Secrets stay out of argv, logs, and audit events. - The container is non-root with a read-only rootfs and no shell, and every secret is a mounted file, so `kubectl exec` / `docker exec` reads no credential from the environment. Exec still bypasses the application's own authorization — the binary *is* the admin CLI — so it must be restricted. - Grant `pods/exec` only to a break-glass role and enable Kubernetes audit logging for the `pods/exec` subresource. ## Edge cases - Deleting a referenced scope or redirect URI disables it rather than leaving a dangling foreign key. - A second `config sync` is a no-op; a changed field is an update, not a new row. - Rotating the token pepper without retaining the old key invalidates every live opaque token. Plan the overlap. - `admin bootstrap` with a non-TTY stdout and no password source refuses rather than print a generated password into job logs. `user enroll-totp` and `initial-access-token issue` refuse a redirected stdout the same way unless `--show-secret` is passed. - A `--password-file` must use mode `0600` (or any mode without the world-readable bit); a world-readable password file is rejected before it is read. ## Operational runbook - First install: `migrate`, then `admin bootstrap`, then complete the forced password change and TOTP enrollment, then create a named admin. See `RUNBOOK.md`. - Routine provisioning: `config sync` for scopes, redirect URIs, and entitlements; `user create` for users. - Key rotation: overlap the pepper for at least the refresh lifetime. Rotate the TOTP key with `totp rewrap`, and the PII key with a write freeze plus `pii rotate` then `pii verify`; see [PII and TOTP key rotation](#pii-and-totp-key-rotation). - Backup: `pg_dump` Postgres; Redis is not backed up. - Access: prefer running admin commands as a one-shot Job (or the local CLI against the database) over `exec` into a running replica. If `exec` is needed, restrict `pods/exec` to a break-glass role and audit the subresource. - Health: liveness `GET /health/live`, readiness `GET /health/ready` on the plaintext health listener. - Durability barrier: a `RotatedUnbarriered` refresh response means the originating rotation applied but the barrier failed after the write, so the returned pair is the live one; every other barrier failure stays fail-closed (`503`). ## File and directory references - `src/cli.rs` for the command tree and flags. - `src/cli/ops.rs`, `src/cli/user.rs`, `src/cli/scope.rs`, `src/cli/redirect.rs`, `src/cli/resource.rs`, `src/cli/sync.rs`, `src/cli/admin.rs`, `src/cli/secret.rs` for implementations. - `src/cli/pii/` for `pii backfill|verify|rotate` and `src/cli/totp.rs` for `totp rewrap`. - `src/crypto/field.rs` for the PII field codec and blind index. - `src/middleware/audit.rs` for the HTTP audit events. - `RUNBOOK.md` for the bootstrap and replacement procedure.