Added security guidance

docs/CONFIGURATION.md

@@ -145,6 +145,15 @@ server refuses to start. The shipped cluster manifests set
 `D_OAUTH_TOKEN_PEPPER_FILE` and `D_OAUTH_TOTP_ENCRYPTION_KEY_FILE` rather than
 the inline keys; see [DEPLOYMENT.md](DEPLOYMENT.md).
 
+These two values are the server's master keys and are supplied entirely by the
+deployment. There is a hard custody requirement that they never live in the
+database or in the same backup as the database; a KMS/HSM or a secret manager
+should hold them and inject them through a `*_FILE` path. The server has no
+built-in KMS integration and never fetches a key itself. The full contract and
+the forbidden locations are in
+[SECURITY.md](SECURITY.md#master-key-custody); the operator procedure is in
+[DEPLOYMENT.md](DEPLOYMENT.md#master-key-custody-and-kmshsm).
+
 ## Failure policy keys
 
 Security endpoints are always fail-closed. Setting `open` on any of them is a

docs/DATABASE.md

@@ -181,6 +181,93 @@ during the rollout. Do not do it.
 - Foreign keys with `ON DELETE CASCADE` keep the join tables consistent, and a
   scope referenced by `user_scopes` is disabled rather than deleted.
 
+## Access hardening
+
+This section records the controls an operator can put around the database. The
+server does not create, ship, or enforce any of them; each is a deployment
+choice. What the server does do is connect with one role and store secrets
+hashed or encrypted, as described in [Security properties](#security-properties).
+
+### Risk baseline: no application audit trail
+
+The schema tables above are domain state. None is an audit log, and the server
+writes no per-access row outside them, so a reader of the schema can run a
+`SELECT` or a `COPY ... TO` and leave nothing behind in the database. A dump is
+silent by construction. Tamper-evidence has to come from PostgreSQL and the
+operator's logging, not from an application table.
+
+### Least privilege
+
+`database.url` (`src/config/db.rs`) carries a single DSN, so by default the
+server connects with one role for both migrations and runtime. Two separate
+roles reduce the blast radius:
+
+| Role | Privileges | Used by |
+| --- | --- | --- |
+| Migration role | Owns the schema and holds the `citext` privilege. `CREATE EXTENSION IF NOT EXISTS citext` needs superuser or trusted-extension rights, so this role must have them (see [The citext extension](#the-citext-extension)). | `migrate` |
+| Runtime role | `SELECT`, `INSERT`, `UPDATE`, `DELETE` on the six domain tables, plus read access to `_sqlx_migrations`. No DDL and no superuser. | `serve` |
+
+The migration role runs once per release; the runtime role is the only one the
+long-lived replicas hold. The server neither creates nor enforces roles. An
+operator grants them and points each subcommand's DSN at the right one.
+
+### Connection and transport
+
+- Production requires `sslmode=verify-full` with a pinned CA
+  (`database.root_cert_path`, `/certs/db-ca.crt`); validation fails closed at
+  startup if it is missing.
+- Prefer a private network or a unix socket and keep the Postgres listener off
+  the public internet. The shipped manifest uses the in-cluster Service DNS
+  name, not an external endpoint.
+- Passwords are file-mounted rather than embedded in the DSN
+  (`deploy/k8s/secret.yaml`).
+
+### Audit and tamper-evidence
+
+- `pgaudit` with, for example, `pgaudit.log = 'write,ddl,role'` records the
+  statements that matter here: data changes, schema changes, and role changes.
+- `log_connections` and `log_disconnections`, plus `log_statement = 'ddl'`,
+  capture who connected and when.
+- Ship Postgres logs to a separate append-only or central store, so a dump
+  cannot also erase its own record.
+- Alert on `pg_dump`, `COPY ... TO`, and any role or permission change.
+- WAL archiving (`archive_mode`) gives a durable, append-only record of every
+  change and lets a restore reach a point in time.
+
+The server contributes no audit rows of its own, so coverage depends entirely on
+the Postgres configuration and the operator's log pipeline.
+
+### Encrypted backups
+
+- `pg_dump`, `pg_basebackup`, and pgBackRest output must be encrypted at rest.
+- Custody the backup key separately from the database, and never co-locate it
+  with the master keys ([SECURITY.md](SECURITY.md#master-key-custody)).
+- Store backups off-site and test restores.
+- A backup is a full copy of the plaintext PII columns exactly as a live dump
+  is, so it carries the same disclosure risk and needs the same handling.
+
+### Postgres-level encryption
+
+Three layers, complementary rather than substitutes:
+
+| Layer | Scope | Protects against | State here |
+| --- | --- | --- | --- |
+| Disk/volume encryption | Cloud EBS/PD or filesystem | Stolen or mislaid media | Operator choice, not shipped |
+| TDE (`pg_tde`) | Whole database files | Stolen data files at rest | Where the engine supports it, not shipped |
+| `pgcrypto` | Selected columns | A dump of those columns | An alternative, not the current implementation |
+
+The application already does its own crypto before a row is written: passwords
+and client secrets are Argon2id PHC, and TOTP secrets are AES-256-GCM under
+HKDF-SHA256 (`src/crypto/totp.rs`). `pgcrypto` is an alternative for the
+plaintext PII columns, but it moves key custody into the SQL path and is not
+used today.
+
+### See also
+
+- [DEPLOYMENT.md](DEPLOYMENT.md#database-access-and-backups) for the connection,
+  TLS, and backup runbook.
+- [SECURITY.md](SECURITY.md#master-key-custody) for key custody.
+
 ## Edge cases
 
 - Deleting a scope or redirect URI that is still referenced does not cascade

docs/DEPLOYMENT.md

@@ -284,6 +284,36 @@ The migrate Job mounts the same files. To rotate a password, update it in
 Postgres/Redis and the Secret together, then roll the replicas; see
 [CONFIGURATION.md](CONFIGURATION.md) and [OPERATIONS.md](OPERATIONS.md).
 
+### Master key custody and KMS/HSM
+
+The token pepper and the TOTP-encryption key are the server's master keys. The
+binary never fetches a key and carries no cloud KMS or HSM SDK; it reads the
+material from `D_OAUTH_TOKEN_PEPPER_FILE` / `D_OAUTH_TOTP_ENCRYPTION_KEY_FILE`
+(or the inline forms), so custody is entirely the deployment's responsibility.
+Treat this as a hard requirement, not a preference:
+
+- **Never co-locate the key with the database.** The keys must not live in a
+  Postgres table, in the database backup, or in any archive that also carries a
+  dump. A dump plus its backup must not yield them.
+- **Never bake the key into the image or the ConfigMap.** The shipped manifests
+  keep only placeholder keys in `secret.yaml` and mount them `0400` on tmpfs at
+  `/run/secrets`; a secret-shaped key in `configmap.yaml` or a PEM in any
+  manifest is rejected by `scripts/check_manifests.py`.
+- **Prefer a KMS/HSM or a secret manager.** A key held in a cloud KMS
+  (AWS KMS, GCP KMS, Azure Key Vault), Vault Transit, or a PKCS#11 HSM is the
+  strongest option; a secret manager (external-secrets, sealed-secrets, Vault
+  agent) that projects the value to a `0400` tmpfs file is the acceptable
+  minimum. Either way the value reaches the process only through the `*_FILE`
+  path and never through the environment.
+- **Keep the backup key separate too.** If the database backup is encrypted, its
+  key is a second secret and is custodied under the same rule, apart from the
+  backup it protects.
+
+Rotation keeps the chain honest: retaining the old pepper key id is what lets
+live tokens verify until they expire, and retaining the old TOTP key id is only
+long enough to re-enroll users. The full contract, the forbidden locations, and
+the reasoning are in [SECURITY.md](SECURITY.md#master-key-custody).
+
 ### Network policy
 
 `networkpolicy.yaml` selects pods labeled
@@ -315,6 +345,45 @@ allowance, which would restore arbitrary exfiltration. A deployment that does
 not can leave the default-deny in place and treat remote `request_uri` as
 unavailable. The tension is recorded in [TODO.md](TODO.md).
 
+### Database access and backups
+
+The schema is plaintext PII plus hashed or encrypted secrets, so the database
+and every copy of it are in scope for the full-disclosure threat model. The
+server opens one DSN as one role; it does not create roles, audit rows, or
+encrypted backups, so those controls live with the operator. Apply at least:
+
+- **Separate the roles.** Run migrations as a schema-owning role that holds the
+  `citext` privilege the `users` migration needs, and run replicas as a runtime
+  role limited to `SELECT`/`INSERT`/`UPDATE`/`DELETE` on the six tables — no
+  DDL, no superuser. The migration Job and the Deployment can then use different
+  credentials.
+- **Encrypt the transport and pin the CA.** Production requires
+  `sslmode=verify-full` with `database.root_cert_path` pointed at the Postgres
+  CA (`/certs/db-ca.crt` in the shipped ConfigMap). Keep the listener on a
+  private network or a unix socket, never the public internet.
+- **Make access auditable and tamper-evident.** Enable `pgaudit`
+  (`pgaudit.log = 'write,ddl,role'`), `log_connections`, `log_disconnections`,
+  and `log_statement = 'ddl'`; ship the log stream to an append-only or central
+  store the database role cannot rewrite; enable WAL archiving (`archive_mode`)
+  as a durable record; and alert on `pg_dump`, `COPY TO`, and role or
+  permission changes. A read-only dump otherwise leaves no trace.
+- **Encrypt the backups and their keys.** `pg_dump`, `pg_basebackup`, and
+  pgBackRest output must be encrypted at rest, stored off-site, and restore
+  tested. The backup is a full copy of the plaintext PII columns, so treat it
+  exactly like a live dump; the backup key is custodied separately and never
+  with the database (see
+  [SECURITY.md](SECURITY.md#master-key-custody)).
+- **Consider Postgres-level encryption.** Disk or volume encryption (cloud
+  EBS/PD or a filesystem layer) protects stolen media; `pg_tde` or an engine's
+  TDE protects the data files; `pgcrypto` can encrypt selected PII columns at
+  the SQL layer. The application already encrypts TOTP secrets with AES-256-GCM
+  and hashes passwords and client secrets with Argon2id, so `pgcrypto` is an
+  alternative, not the current design, and it moves key custody into the SQL
+  path.
+
+The controls, the privilege split, and the `pgcrypto` trade-off are described in
+[DATABASE.md](DATABASE.md#access-hardening).
+
 ### Multi-replica validation
 
 A local compose file is the intended way to prove shared-state behavior: three

docs/SECURITY.md

@@ -171,6 +171,53 @@ Reference: RFC 9126, RFC 9101.
 - There is no authorization-server JWT-signing key and no `jwks_uri`. The
   `keygen` command emits keys for test fixtures and client-side DPoP only.
 
+## Master key custody
+
+- The token pepper and the TOTP-encryption key are the server's master keys.
+  They live in the versioned `KeyRing` (`src/keys.rs`), loaded once at startup
+  from configuration. Nothing re-reads or fetches them later.
+- **Hard requirement.** A master key must never be stored in Postgres, captured
+  in the same backup or restore as Postgres, baked into an image layer, placed
+  in a ConfigMap, or injected into the process environment by the shipped
+  manifests. A database dump plus its backup must not yield either key.
+- Losing the token pepper invalidates every live opaque access and refresh
+  token, because only keyed HMAC hashes are persisted and the pepper is the
+  HMAC key.
+- The real config surface is small: inline `token_pepper` /
+  `totp_encryption_key`, or the mounted-file forms `token_pepper_file` /
+  `totp_encryption_key_file`. When both are set the inline value wins
+  (`resolve_secret` in `src/config/build.rs`). The key ids default to `v1`
+  through `token_pepper_id` / `totp_key_id` (`src/config/raw.rs`).
+- The shipped cluster manifests never use the inline or environment forms: the
+  Secret file-mounts the keys `0400` on a tmpfs volume at `/run/secrets`, and
+  the loader reads them through `D_OAUTH_TOKEN_PEPPER_FILE` /
+  `D_OAUTH_TOTP_ENCRYPTION_KEY_FILE` (`deploy/k8s/secret.yaml`,
+  `deploy/k8s/configmap.yaml`). The ConfigMap carries only non-secret settings.
+- **The server does not talk to a KMS or an HSM.** It has no cloud SDK and no
+  key-fetch code path. Sourcing a key from a KMS/HSM or a secret manager is the
+  deployment's responsibility; this section is a documented contract, not an
+  implemented feature.
+
+Custody, in order of preference:
+
+- **Preferred.** The key is held in a KMS/HSM or a secret manager that never
+  co-locates the key with the database backup: cloud KMS, Vault Transit, or a
+  PKCS#11 HSM.
+- **Acceptable.** A secret manager writes a `0400` tmpfs file that the server
+  reads through its `*_FILE` key. This is what the shipped manifests expect.
+- **Forbidden.** The key is in the database, in a database backup, in an image
+  layer, in the ConfigMap, or in the process environment of the shipped
+  manifests.
+
+Custody and rotation interact. Retain the old pepper key id for at least the
+refresh lifetime, or live tokens stop verifying; retain the old TOTP key id
+only long enough to re-enroll users, then drop it. The full order is above; a
+rotation that removes an id early is a deliberate invalidation of what that key
+protects.
+
+The operator procedure for custody and KMS/HSM sourcing lives in
+[DEPLOYMENT.md](DEPLOYMENT.md#master-key-custody-and-kmshsm).
+
 ## Fail-closed policy
 
 The seven security endpoints (`authorize`, `token`, `introspect`, `revoke`,

docs/TODO.md

@@ -2,9 +2,10 @@
 
 This is a backlog of deferred hardening work. Each item is a candidate to be
 planned and implemented later; none of them is a current defect that makes the
-server unusable. The items came out of a threat-model review that asked one
-question: if an attacker walks off with a full dump of the PostgreSQL database,
-what can they still read or crack, and what should we do about it?
+server unusable. The items came out of a threat-model review that asked two
+questions: what an attacker can still read or crack from a full dump of the
+PostgreSQL database, and what an attacker already inside the image's trust
+boundary can reach.
 
 This document only records the gaps. It does not describe anything that is
 already implemented, and it does not propose code changes here.
@@ -60,24 +61,6 @@ list is deliberately short and ordered by value.
   key, which makes rotation incremental and limits a single leak's blast radius.
 - **Files.** `src/crypto/totp.rs`, `src/keys.rs`.
 
-### 3. KMS/HSM-backed master key
-
-- **Risk.** The master keys live in environment variables, files, or a
-  secrets manager, but nothing enforces where they are stored or keeps them
-  out of the same backup as the database.
-- **Proposed change.** Document, and optionally support, a master key held in a
-  KMS or HSM. At a minimum, document the hard requirement that the key never
-  lives with the database backup.
-- **Files.** `src/config/`, `docs/DEPLOYMENT.md`, `docs/SECURITY.md`.
-
-### 4. Database access hardening
-
-- **Risk.** There is no tamper-evidence or audit trail for database access, and
-  no guidance on encrypted backups or Postgres-level encryption such as TDE or
-  `pgcrypto`. A dump could be taken by a read-only job and leave no trace.
-- **Proposed change.** Document the controls and, where they fit, implement them.
-- **Files.** `docs/DATABASE.md`, `docs/DEPLOYMENT.md`.
-
 ## Container and host hardening
 
 The attack surface and the controls already in place are documented in
@@ -86,7 +69,7 @@ It answers a second threat-model question: if an attacker is already inside the
 image's trust boundary, how could they reach the container or the host, and how
 should we reduce that risk? The remaining vectors are narrow.
 
-### 5. NetworkPolicy versus remote `request_uri`
+### 3. NetworkPolicy versus remote `request_uri`
 
 - **Risk.** The default-deny egress blocks outbound HTTPS, so the JAR
   `request_uri` fetch (`src/jose/jar/fetch.rs`) cannot reach a publicly routable
@@ -98,7 +81,7 @@ should we reduce that risk? The remaining vectors are narrow.
   option because it restores arbitrary exfiltration.
 - **Files.** `deploy/k8s/networkpolicy.yaml`, `docs/DEPLOYMENT.md`.
 
-### 6. Narrow Postgres/Redis egress to their destination
+### 4. Narrow Postgres/Redis egress to their destination
 
 - **Risk.** `networkpolicy.yaml` allows egress to *any* address on ports 5432 and
   6379 (link-local excluded), so those ports are an exfiltration channel to an
@@ -113,14 +96,13 @@ should we reduce that risk? The remaining vectors are narrow.
 
 Ranked by value against effort:
 
-- **Medium:** items 1, 2, 3. Item 1 is medium/high effort because it needs a
+- **Medium:** items 1 and 2. Item 1 is medium/high effort because it needs a
   migration plus a blind index with uniqueness and rotation handling.
-- **Lower, mostly operational:** item 4.
-- **Container/host hardening:** item 5 is the concrete correctness risk; item 6
+- **Container/host hardening:** item 3 is the concrete correctness risk; item 4
   is the egress-narrowing follow-up.
 
 Recommendation: implement item 1 next. It closes the largest plaintext
-disclosure; items 2 and 3 then reduce the cost of a future key compromise.
+disclosure; item 2 then reduces the cost of a future key compromise.
 
 ## Open questions
 
@@ -129,6 +111,6 @@ disclosure; items 2 and 3 then reduce the cost of a future key compromise.
   in the target deployment? This decides whether envelope encryption (item 2)
   is worth building at all.
 - Which target environments rely on remote JAR `request_uri` objects? This
-  decides whether item 5 needs a scoped egress rule or only documentation.
-- Do Postgres and Redis run in-cluster? This decides whether item 6 can narrow
+  decides whether item 3 needs a scoped egress rule or only documentation.
+- Do Postgres and Redis run in-cluster? This decides whether item 4 can narrow
   egress to their namespace/selector.