d-OAuth-Server/docs/OPERATIONS.md

Operations

Administration is CLI plus declarative configuration only. There is no HTTP admin API, no default credential, and nothing is auto-created at startup. This is the de-facto admin interface and the runbook for its day-two tasks.

Purpose

For an operator: the full command surface, the declarative config sync file, the audit stream, key and certificate rotation, and backup and restore.

The admin model

• The migrate subcommand owns the schema and is the only writer of it.
• The serve subcommand serves OAuth and never migrates and never provisions.
• Every mutation (scopes, users, entitlements, redirect URIs, resource servers, client attachments, client access-token format) is audited on the audit tracing target.
• Secrets are read from stdin, a file, or the environment, never from argv. A secret is printed only when the operator asked for a one-time value.

Command reference

The binary is d-oauth-server (cargo run --bin d-oauth-server -- <command>). Exit codes: 0 success, 1 command failure, 2 usage error.

serve and migrate

d-oauth-server serve
d-oauth-server migrate
d-oauth-server config check

• serve loads config, connects Postgres and Redis, asserts the Redis durability contract, and runs the TLS public listener plus the plaintext health listener. It never migrates. It drains on SIGTERM.
• migrate applies forward-only migrations under sqlx's advisory lock and prints schema current: N known migration(s) (N is the embedded count, not the delta just applied).
• config check validates configuration and prints config ok: issuer=....

Users

d-oauth-server user create --email <email> [--admin] [--password-stdin] [--password-file <path>]
d-oauth-server user enroll-totp --email <email> [--show-secret]
d-oauth-server user disable --email <email>
d-oauth-server user unlock --email <email>
d-oauth-server user grant-scope --email <email> --scope <scope>
d-oauth-server user revoke-scope --email <email> --scope <scope>
d-oauth-server user scopes --email <email>

• create reads the password from stdin by default or from --password-file. The two sources are mutually exclusive. --admin sets is_admin. A created user always starts with must_change_password = true. A --password-file must not be world-readable (use mode 0600); a world-readable file is rejected before it is read. Passwords are held in wiped-on-drop memory and never appear in logs, errors, or audit events.
• enroll-totp generates an encrypted TOTP secret and prints the secret and the otpauth:// URI exactly once. The secret prints only to an interactive terminal; a redirected stdout (pipe, file, or job log) refuses unless --show-secret is passed to acknowledge the disclosure. Register the secret in an authenticator now.
• disable first purges all of the user's Redis ephemeral state (login sessions and their :meta sidecars, access/refresh tokens and their grants, authorization codes, device records with their :cnf/:poll sidecars and user_code mappings, and the lockout counters (account plus per-IP)), then flips the Postgres disabled flag. If Redis is unavailable the command fails and the user stays enabled (fail closed). Disabling alone does not touch PAR records, replay markers, or rate-limit counters, which carry no subject.
• unlock clears the user's login lockout counters (the per-account counter and every per-IP composite bucket) without touching the account. The user must exist; a Redis outage fails closed and clears nothing. Unknown-label buckets carry no account label and are never touched. The per-IP sweep uses SCAN, so on Cluster topologies it covers the contacted node only and buckets on other nodes may survive; re-run against each node or wait out the window.
• grant-scope requires the scope to exist in the catalog.
• revoke-scope is idempotent.

Scopes

d-oauth-server scope create <name> [--description <text>]
d-oauth-server scope list
d-oauth-server scope enable <name>
d-oauth-server scope disable <name>
d-oauth-server scope delete <name>

create upserts: it creates a scope, updates its description, or re-enables a disabled one. delete deletes a scope that nothing references, and disables it when user_scopes references it, so entitlements are not silently destroyed. The enabled scope catalog is what GET /.well-known/oauth-authorization-server (discovery document) advertises as scopes_supported.

Redirect URIs and client attachments

d-oauth-server redirect-uri create <uri> [--kind authorize|post_logout] [--description <text>]
d-oauth-server redirect-uri list
d-oauth-server redirect-uri enable <uri> [--kind authorize|post_logout]
d-oauth-server redirect-uri disable <uri> [--kind authorize|post_logout]
d-oauth-server redirect-uri delete <uri> [--kind authorize|post_logout]
d-oauth-server client attach-redirect-uri --client-id <id> --uri <uri> [--kind authorize|post_logout]
d-oauth-server client detach-redirect-uri --client-id <id> --uri <uri> [--kind authorize|post_logout]
d-oauth-server client redirect-uris --client-id <id>
d-oauth-server client set-access-token-format --client-id <id> --format opaque|jwt

A redirect URI is a global catalog entry; a client attaches to it. /authorize matches only the client's attached and enabled authorize set. A private-use URI scheme must contain reverse-DNS dot notation (draft-16 §2.3.1): com.example.app:/ is accepted and myapp:/ is rejected at creation. Detaching or disabling a URI stops future redirects to it; already issued tokens are unaffected. Deleting a referenced URI disables it instead.

Legacy rows predate the §2.3.1 dot check, so a catalog that still holds a dotless private-use URI (for example myapp:/cb) makes config sync fail: redirect::upsert validates the scheme (src/cli/redirect.rs:170) before the idempotency short-circuit (src/cli/redirect.rs:178, via src/cli/sync.rs:168), so the sync aborts with invalid input instead of reporting no changes. redirect-uri disable and redirect-uri delete bypass the upsert check, so disable or delete the legacy row first, then re-run the sync.

kind selects the URI's role. authorize (the default) is an authorization-response callback. post_logout is a bare return target used only by POST /logout to send the browser back to the client after the session ends (GET /logout is read-only and issues no redirect). POST /logout redirects only while exactly one enabled post_logout URI is registered for the client and the client is active. Zero registered enabled URIs or more than one, and POST /logout performs no redirect: the target must be unambiguous or nothing happens. Registering a second enabled post_logout URI is a configuration error, not a silent last-writer-wins.

Resource servers and client access-token format

d-oauth-server resource-server create <identifier> [--description <text>]
d-oauth-server resource-server list
d-oauth-server resource-server enable <identifier>
d-oauth-server resource-server disable <identifier>
d-oauth-server resource-server delete <identifier>

A resource server is an RFC 8707 resource indicator (the resource parameter and the aud target of an access token). identifier matches exactly — no normalization, sub-path, or wildcard. It must be an absolute https URI with a host; a *, userinfo, fragment, or query is rejected, so one catalog row can never stand for an entire origin or a non-fetchable identifier. create upserts: it creates the row, updates its description, or leaves an unchanged row alone; it never silently re-enables an operator-disabled row, and list shows both states. delete removes the row outright (nothing references the catalog directly).

client set-access-token-format --client-id <id> --format opaque|jwt is the OPERATOR-ONLY opt-in to JWT access tokens for one client; opaque (the OAuth 2.1 baseline and column default) is what every dynamically registered client gets. --format accepts exactly opaque or jwt, and jwt is honored only while JWT access tokens are enabled by configuration. Dynamic client registration can never set this: a registration (POST /register) body that carries access_token_format is rejected with invalid_client_metadata. The JWT signing key is never printed by any command.

Declarative config

d-oauth-server config sync --file <path>
d-oauth-server config check

config sync applies a JSON file idempotently. A second identical run reports config sync: no changes. See the format below.

Admin bootstrap

d-oauth-server admin bootstrap --email <email> [--password-file <path>] [--force]

Creates the first admin under an advisory lock in one transaction. It refuses while any enabled admin exists unless --force is given (a loud warning and an audit event). The password comes from --password-file, from stdin, or — when neither yields a value — a generated 256-bit password that prints exactly once and only to a TTY (a non-TTY stdout with a generated password is an error, so the secret can never leak into job logs). A --password-file must not be world-readable (use mode 0600). The full replacement procedure is in RUNBOOK.md.

Initial access tokens (DCR)

d-oauth-server initial-access-token issue --uses <n> --ttl <seconds> [--label <text>] [--show-secret]

Mints a one-time-use DCR initial-access token. The raw token is printed once; only its keyed hash and remaining-use budget are stored. The token prints only to an interactive terminal; a redirected stdout refuses unless --show-secret is passed to acknowledge the disclosure. The token is required when allow_open_registration is false.

Key generation and health

d-oauth-server keygen [--alg es256]
d-oauth-server healthcheck [--url <url>]

keygen emits an ES256 keypair for test fixtures and client-side DPoP keys. It is never an authorization-server signing key; the server publishes no jwks_uri. --alg defaults to es256 (the only algorithm). healthcheck probes the health listener over raw HTTP and exits 0 or 1; it is the container HEALTHCHECK. --url defaults to http://127.0.0.1:8081/health/live (liveness).

PII and TOTP migration commands

d-oauth-server pii backfill
d-oauth-server pii verify
d-oauth-server pii rotate
d-oauth-server totp rewrap

All four take no options and read the key ring from configuration. pii backfill, pii verify, and pii rotate require a migration-role DSN (D_OAUTH_DATABASE__URL); they scan every row and print counts only, never a value.

• pii backfill seals every remaining plaintext PII value, blanks the plaintext columns, and upgrades every TOTP blob to v3. It refuses while pii_dual_write=true, and must run with the app tier scaled to zero.
• pii verify reports the four cutover checks: no plaintext remains, every _enc opens under a retained key, every TOTP blob is v3, and every stored blind index matches. It exits non-zero when any check is dirty.
• pii rotate re-encrypts every _enc value and recomputes every blind index under the active PII key. Run it under a write freeze with the previous key retained. It is idempotent: a re-run re-encrypts only stale blobs.
• totp rewrap re-wraps every v3 per-row DEK under the active TOTP key without touching the data ciphertext. It is idempotent.

PII encryption rollout and cutover

The expansion is additive. The backfill blanks the plaintext columns but this release does not drop them; dropping the now-inert columns is a later release. Follow this order exactly:

1. Apply the expansion migration with the PII key configured: d-oauth-server migrate. The migration drops NOT NULL on the plaintext columns and builds the new indexes, so run it in a maintenance window. See DATABASE.md.
2. Set pii_dual_write=true (D_OAUTH_PII_DUAL_WRITE=true).
3. Roll every replica so all of them dual-write and emit v2 TOTP blobs.
4. Set pii_dual_write=false and scale the app tier to zero, so no writer runs during the backfill.
5. Run d-oauth-server pii backfill.
6. Run d-oauth-server pii verify; it must exit 0.
7. Scale the app tier back up. Startup readiness is fail-closed: if pii_dual_write=false and any blanked plaintext column still holds a value, GET /health/ready returns 503 and the replica stays out of service.

While the rollout window is open, CLI d-oauth-server user enroll-totp and login-driven TOTP enrollment MUST run with D_OAUTH_PII_DUAL_WRITE=true. With it false they write v3 TOTP blobs, which a version N-1 replica cannot read, so the user's next TOTP check fails on an old replica. Keep dual-write on for the whole window; turn it off only at the scale-to-zero step above.

Resource-server onboarding (RFC 9728)

This is the operator-side companion to RS-CONTRACT.md, the normative contract for the resource server (RS). The AS hosts no RS route and publishes no protected-resource metadata: onboarding an RS is a catalog step on the AS plus a publish, challenge, and rotation step the RS team owns. Do the four steps in order; the RS team can follow RS-CONTRACT.md from the document alone once the identifier and issuer below are fixed.

1. Register the resource identifier in the AS catalog

The catalog is the RFC 8707 resource-indicator allowlist and the aud target of an access token. Register the RS's exact identifier (an absolute https URL, no fragment, no wildcard):

d-oauth-server resource-server create https://api.example.com --description "Orders API"
d-oauth-server resource-server list

identifier matches exactly, with no normalization, sub-path, or wildcard; a * anywhere is rejected at creation, as are a non-https scheme, a missing host, userinfo, a fragment, or a query, so one catalog row can never stand for an entire origin or a non-fetchable identifier. create upserts (it creates the row, updates its description, or leaves an unchanged row alone) and never silently re-enables an operator-disabled row. list shows both the enabled and disabled states. Use resource-server enable|disable <identifier> to flip a row and resource-server delete <identifier> to remove it. All four mutations are audited (resource_server.create|enable|disable|delete).

The same steps are declarative through config sync; a resource_servers entry with the identifier, description, and enabled flag is applied idempotently (see config sync file format).

For tokens to be audience-restricted, set resource_indicators_enabled=true so the AS honors the RFC 8707 resource parameter at the authorization, PAR, token, and device endpoints; otherwise the RS cannot rely on the aud check alone (RFC 9728 §7.4). See CONFIGURATION.md.

2. Publish the RS protected-resource metadata document

The RS team publishes GET /.well-known/oauth-protected-resource at the resource host (RFC 9728 §3); when the identifier carries a path, the terminating slash is removed and the path suffix is appended (/.well-known/oauth-protected-resource/resource1). The document is served as application/json with a 200 OK (RFC 9728 §3.2) and its resource member must be byte-for-byte identical to the identifier registered in step 1 (RFC 9728 §3.3, §7.3). The operator's job is to hand the RS team two values:

• authorization_servers: the AS issuer value (for example https://auth.example.com).
• scopes_supported: the runtime scope catalog this AS actually enables, read with scope list. It is a supported set, not a request-everything hint (RFC 9728 §7.2); advertising a scope the AS does not enable only misleads clients.

The full member set and an example document are in RS-CONTRACT.md.

Optionally, advertise the RS identifier on the AS side through the RFC 9728 §4 protected_resources member: set protected_resources_enabled=true and list the identifier in protected_resource_ids. Both default off and empty, so the AS advertises no protected_resources until enabled. The load is fail-closed: the flag requires resource_indicators_enabled=true and a non-empty, duplicate-free list of absolute https identifiers with no wildcard, userinfo, fragment, or query. resource_metadata is never an AS metadata member (RFC 9728 §4, §5.1); it belongs only on the RS challenge in step 3.

3. Set the WWW-Authenticate challenge on the RS

The RS returns the challenge on an unauthenticated or unacceptable request (RFC 9728 §5.1, RFC 6750 §3):

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource"

The parameter value is the absolute https URL of the document from step 2, derived from the same identifier; the RS never invents it ad hoc, and it is never sent alongside a success. When the RS later needs clients to pick up a changed metadata document, it re-sends a fresh challenge and clients re-fetch and re-validate on receipt (RFC 9728 §5.2). The challenge may be combined with other parameters (for example max_age, RFC 9470) and with schemes other than Bearer such as DPoP (RFC 9449).

4. Rotate the identifier and keys

• Resource identifier: register the new identifier with resource-server create and keep the old row enabled until no live token references it, then resource-server delete the old identifier. The RS team re-publishes its metadata document with the new resource and re-sends the challenge; clients re-validate against the new value (RFC 9728 §3.3, §5.2). Never leave two enabled identifiers standing for the same deployment if the audience check is meant to be exact.
• AS token pepper: rotating the pepper invalidates every live opaque token. For an RS that introspects (RFC 7662), that means re-authentication, not a silent accept; overlap the old pepper for at least ttl.refresh_secs (see Key and certificate rotation).
• JWT signing keys: when JWT access tokens are enabled, retain the previous jwt_signing_previous_keys across the rotation window so the RS can still verify in-flight tokens; the RS caches the AS JWKS with a positive lifetime and re-fetches on a fresh challenge (RFC 9728 §7.10, §5.2). Never rotate to an empty key set.
• After any rotation, re-run the discovery flow end to end: call the resource with no token, read the challenge, fetch and validate the metadata, then re-authenticate. The full flow is in RS-CONTRACT.md.

MCP client registration

An MCP client needs a client identity at the authorization server. There are three supported paths, in order of preference. They are per-client choices; the one server-wide setting they share is whether the host localhost counts as an RFC 8252 loopback redirect (allow_localhost_redirect).

1. CIMD (primary, no registration)

Client ID metadata documents let a client identify itself with an https:// URL and skip registration entirely.

1. Set cimd_enabled=true. If the client redirects to http://localhost:{port}, also set allow_localhost_redirect=true.

2. Restart the server, then confirm the advertised capability:

   curl -s https://auth.example.com/.well-known/oauth-authorization-server \
   | jq '{client_id_metadata_document_supported, token_endpoint_auth_methods_supported}'
   

The output must carry client_id_metadata_document_supported: true and none inside token_endpoint_auth_methods_supported. With cimd_enabled off the first member is absent, not false.

1. The client hosts a JSON document at an https:// URL and uses that exact URL as its client_id. The document must validate. A minimal accepted example:

   {
   "client_id": "https://mcp.example.com/oauth/client-metadata.json",
   "client_name": "Example MCP client",
   "redirect_uris": ["https://mcp.example.com/oauth/callback"],
   "token_endpoint_auth_method": "none",
   "grant_types": ["authorization_code", "refresh_token"],
   "scope": "api:read",
   "dpop_bound_access_tokens": false
   }
   

Every scope token must already be enabled in the runtime catalog (scope list). client_id must equal the URL byte for byte. The authorization server fetches the document on the first request and caches it for 15 minutes; a changed document is picked up after that window. No /register call and no initial-access token are involved. A private_key_jwt document must carry an inline jwks; jwks_uri is rejected. See OAUTH.md.

2. Open dynamic client registration

This lets any caller register a client with POST /register and no token. It needs all of the following:

• allow_open_registration=true.
• A non-catalog dcr_redirect_policy: catalog_loopback or open.
• For an http://localhost:{port} callback, allow_localhost_redirect=true.

The default dcr_redirect_policy = catalog rejects any redirect URI that is not already in the redirect catalog, even when open registration is on. Choose catalog_loopback for catalog URIs plus dynamic loopback URIs, or open for any absolute https URI plus loopback. Literal 127.0.0.1 and [::1] are always loopback; localhost is loopback only with the flag on. Open registration is anonymous client creation, so prefer it only where that risk is acceptable.

3. IAT-gated dynamic client registration

With allow_open_registration=false (the default), the client must present a one-time initial-access token:

d-oauth-server initial-access-token issue --uses <n> --ttl <seconds> --label mcp

The raw token prints once, and only to an interactive terminal unless --show-secret is passed. The client calls POST /register with Authorization: Bearer <token>:

curl -s https://auth.example.com/register \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "redirect_uris": ["https://mcp.example.com/oauth/callback"],
    "token_endpoint_auth_method": "none",
    "grant_types": ["authorization_code", "refresh_token"],
    "scope": "api:read"
  }' | jq .

Each use authorizes one registration call, whether or not the metadata later validates. The redirect policy still applies: with the default catalog policy, the redirect URI must already be cataloged. For a localhost callback, use a non-catalog policy and set allow_localhost_redirect=true.

There is no client-create command in the CLI. A client is created only by POST /register (with an IAT or under open registration), and the CLI manages existing clients through client attach-redirect-uri, client detach-redirect-uri, client redirect-uris, and client set-access-token-format.

config sync file format

The file is JSON. Every section is optional and defaults to empty.

{
  "scopes": [
    { "name": "orders:read", "description": "Read orders", "enabled": true }
  ],
  "redirect_uris": [
    { "uri": "https://app.example.com/callback", "enabled": true },
    { "uri": "https://app.example.com/signed-out", "kind": "post_logout", "enabled": true }
  ],
  "resource_servers": [
    { "identifier": "https://api.example.com", "description": "Orders API", "enabled": true }
  ],
  "user_scopes": [
    { "email": "operator@example.com", "scope": "orders:read" }
  ],
  "client_redirect_uris": [
    { "client_id": "example-client", "uri": "https://app.example.com/callback" },
    { "client_id": "example-client", "uri": "https://app.example.com/signed-out", "kind": "post_logout" }
  ],
  "client_access_token_formats": [
    { "client_id": "example-client", "access_token_format": "jwt" }
  ]
}

Scope names must match the RFC 6749 scope-token grammar. openid is reserved. Redirect URIs must be absolute and are matched exactly at /authorize (or, for kind: post_logout, at POST /logout). kind defaults to authorize. Resource identifiers are exact RFC 8707 resource indicators; a * wildcard is rejected. client_access_token_formats sets the operator-only per-client access-token format (opaque or jwt); it is the same operator opt-in as client set-access-token-format (dynamic registration can never set it).

Apply and re-apply:

d-oauth-server config sync --file ./oauth-config.json
d-oauth-server config sync --file ./oauth-config.json

The first reports the counts of what it created; the second reports no changes. Upserts use IS DISTINCT FROM, so an unchanged row does not bump updated_at.

Audit events

CLI mutations go to the audit target with event = "cli.mutation", an operation, and a subject. The operations are migrate, config.sync, user.create, user.enroll_totp, user.disable, user.unlock, user.grant_scope, user.revoke_scope, scope.create, scope.enable, scope.disable, scope.delete, redirect_uri.create, redirect_uri.enable, redirect_uri.disable, redirect_uri.delete, resource_server.create, resource_server.enable, resource_server.disable, resource_server.delete, client.attach_redirect_uri, client.detach_redirect_uri, client.set_access_token_format, initial_access_token.issue, pii.backfill, pii.rotate, totp.rewrap, admin.count.changed, and the admin.bootstrap.* family. pii verify and user scopes are read-only and emit no mutation event.

HTTP events on the same target include login.success, login.failure, consent, token.issue, token.revoke, token.refresh_reuse_detected, and dcr.register. No event carries a token, password, OTP, or secret.

Tracing is JSON. Route it to your log pipeline and alert on token.refresh_reuse_detected and repeated login.failure.

Key and certificate rotation

Token pepper

Opaque token hashes are stored under a version id, default v1, set by token_pepper_id. The key ring resolves a stored id newest-first across a retained chain, so verification keeps working while an old pepper is retained. The current loader supplies one active pepper; to rotate without invalidating live tokens, retain the previous key id for at least the refresh lifetime (ttl.refresh_secs, default 30 days) before removing it. New tokens are always minted under the active pepper.

TLS certificate

serve loads the certificate chain and key once at startup. Rotate by replacing the mounted material and performing a rolling restart; there is no hot reload, by design.

JWT access-token signing key

Reference: RFC 9068 §2.1, §4; RFC 7517 §5.

This key is optional and off by default. It is unrelated to keygen, which emits ES256 material for test fixtures and client-side DPoP only; do not use keygen output here unless it happens to be exactly 32 bytes.

jwt_signing_key is consumed as the literal UTF-8 bytes of the configured string, not decoded as base64 or hex. It must be exactly 32 bytes. Generate printable material with:

openssl rand -hex 16

That prints 32 hexadecimal characters, which are 32 bytes of key material. The inline value is trimmed of leading and trailing whitespace; a *_FILE mount is read verbatim except that trailing CR/LF are removed, so do not add spaces. An all-zero or out-of-range P-256 scalar still fails at load with InvalidJwtSigningKey.

To enable the feature and mint JWTs for one client:

1. Set the environment:

   export D_OAUTH_JWT_ACCESS_TOKENS_ENABLED=true
   export D_OAUTH_JWT_SIGNING_KEY="$(openssl rand -hex 16)"
   export D_OAUTH_JWT_SIGNING_KID=v1
   export D_OAUTH_JWT_ACCESS_TOKEN_SIGNING_ALG=ES256
   

D_OAUTH_JWT_SIGNING_KEY_FILE mounts the same material as a file. The algorithm is ES256 (default) or EdDSA; both use 32 raw bytes and there is no RSA option. Enabling the switch without a key fails at boot with MissingJwtSigningKey.

1. Roll the replicas and confirm the public key set:

   curl -fsS "$D_OAUTH_ISSUER/jwks.json" | jq -r '.keys[] | "\(.kid) \(.alg) \(.kty)"'
   

The current key appears first. While the feature is off, the route still answers 200 with {"keys":[]}; it is cached for 300 s.

1. Opt a client in, per client, only after the deployment is ready:

   d-oauth-server client set-access-token-format --client-id <id> --format jwt
   

Registration can never set this; opaque remains the column default.

To rotate:

1. Generate a new key and a new kid.

2. Make the new key active. Move the old key into jwt_signing_previous_keys, a list of {kid, key} objects supplied in the D_OAUTH_CONFIG file (the loader does not accept this list as one environment string):

   jwt_access_tokens_enabled: true
   jwt_signing_key: "<new 32-byte key>"
   jwt_signing_kid: v2
   jwt_signing_previous_keys:
    - kid: v1
      key: "<old 32-byte key>"
   

Keep jwt_access_token_signing_alg unchanged unless every retained key is regenerated: all entries in the ring share that one value.

1. Roll every replica, so each one signs with v2 and verifies v1.

2. Confirm both kids are published, current first:

   curl -fsS "$D_OAUTH_ISSUER/jwks.json" | jq -r '.keys[].kid'
   

3. Keep the previous key until every token it signed has expired. Access tokens live ttl.access_secs (300 s by default); refresh tokens stay opaque and do not use this key. After that window, remove the previous entry and roll the replicas again. A JWKS document cached by a resource server can lag the removal by up to 300 s, which is harmless because no new token names the old kid.

The old key verifies only. It never signs once it is in jwt_signing_previous_keys, and a duplicate kid (active or previous) is rejected at load.

PII and TOTP key rotation

Both the PII field-encryption key and the TOTP encryption key are versioned in the ring, and each has an optional retained previous key. Keep the previous key id configured until every value has been migrated, then retire it. A previous key without a distinct *_previous_key_id, or without its active key, is a load-time error.

TOTP encryption key

The TOTP blob carries its key id, and the ring resolves a retained TOTP key by id. To rotate:

1. Put the old key in totp_previous_key / totp_previous_key_file with a distinct totp_previous_key_id, and set the new totp_encryption_key / totp_encryption_key_file as active.
2. Roll the replicas so the ring loads both keys.
3. Run d-oauth-server totp rewrap. Every v3 blob is re-wrapped under the new key. Version 1 and 2 blobs have no DEK and still decrypt through the legacy path, so re-enroll those users with d-oauth-server user enroll-totp before the old key is dropped.
4. Remove the previous key once no blob names it.

PII field-encryption key

The blind index is key-dependent, so equality and uniqueness only hold within one key generation. Rotate under a write freeze:

1. Put the old key in pii_previous_key / pii_previous_key_file with a distinct pii_previous_key_id, and set the new pii_encryption_key / pii_encryption_key_file as active.
2. Roll the replicas, then scale the app tier to zero (the write freeze). No writer may run while the indexes are recomputed.
3. Run d-oauth-server pii rotate. It re-encrypts every _enc value and recomputes every blind index under the new key in bounded transactions; a duplicate across key generations aborts that batch instead of overwriting a row. It is idempotent and resumable because the previous key is retained.
4. Run d-oauth-server pii verify; it must exit 0.
5. Remove the previous key and scale the app tier back up.

Backup and restore

Postgres holds all durable state, so a normal backup covers it:

pg_dump --format=custom "$DATABASE_URL" > oauth.dump
pg_restore --clean --if-exists --dbname "$DATABASE_URL" oauth.dump

Restore into a migrated database, then run migrate to confirm the schema version matches the binary. Redis is ephemeral and need not be backed up: codes, tokens, sessions, and replay markers are re-created as clients re-authenticate, and a Redis loss does not lose clients, users, scopes, or redirect URIs. Registration access token hashes are in Postgres, so DCR client management also survives a Redis loss.

Security properties

• No HTTP admin surface and no default credential.
• Every mutation is audited.
• Bootstrap refuses a second admin without --force, which is loud and audited.
• A restricted session (forced password change or missing TOTP) cannot reach /authorize, the OAuth endpoints, or admin-gated actions.
• Secrets stay out of argv, logs, and audit events.
• The container is non-root with a read-only rootfs and no shell, and every secret is a mounted file, so kubectl exec / docker exec reads no credential from the environment. Exec still bypasses the application's own authorization — the binary is the admin CLI — so it must be restricted.
• Grant pods/exec only to a break-glass role and enable Kubernetes audit logging for the pods/exec subresource.

Edge cases

• Deleting a referenced scope or redirect URI disables it rather than leaving a dangling foreign key.
• A second config sync is a no-op; a changed field is an update, not a new row.
• Rotating the token pepper without retaining the old key invalidates every live opaque token. Plan the overlap.
• admin bootstrap with a non-TTY stdout and no password source refuses rather than print a generated password into job logs. user enroll-totp and initial-access-token issue refuse a redirected stdout the same way unless --show-secret is passed.
• A --password-file must use mode 0600 (or any mode without the world-readable bit); a world-readable password file is rejected before it is read.

Operational runbook

• First install: migrate, then admin bootstrap, then complete the forced password change and TOTP enrollment, then create a named admin. See RUNBOOK.md.
• Routine provisioning: config sync for scopes, redirect URIs, and entitlements; user create for users.
• Key rotation: overlap the pepper for at least the refresh lifetime. Rotate the TOTP key with totp rewrap, and the PII key with a write freeze plus pii rotate then pii verify; see PII and TOTP key rotation.
• Backup: pg_dump Postgres; Redis is not backed up.
• Access: prefer running admin commands as a one-shot Job (or the local CLI against the database) over exec into a running replica. If exec is needed, restrict pods/exec to a break-glass role and audit the subresource.
• Health: liveness GET /health/live, readiness GET /health/ready on the plaintext health listener.
• Durability barrier: a RotatedUnbarriered refresh response means the originating rotation applied but the barrier failed after the write, so the returned pair is the live one; every other barrier failure stays fail-closed (503).

File and directory references

• src/cli.rs for the command tree and flags.
• src/cli/ops.rs, src/cli/user.rs, src/cli/scope.rs, src/cli/redirect.rs, src/cli/resource.rs, src/cli/sync.rs, src/cli/admin.rs, src/cli/secret.rs for implementations.
• src/cli/pii/ for pii backfill|verify|rotate and src/cli/totp.rs for totp rewrap.
• src/crypto/field.rs for the PII field codec and blind index.
• src/middleware/audit.rs for the HTTP audit events.
• RUNBOOK.md for the bootstrap and replacement procedure.