Moved to native arm runner

.github/workflows/supply-chain.yml

@@ -2,13 +2,16 @@
 #
 # Per-PR legs: `manifest-lint` (kustomize build + scripts/check_manifests.py,
 # including the L7-vs-L4 mTLS pairing check).
-# Per-push-to-master legs: multi-arch `docker buildx bake` with SBOM +
-# provenance, trivy/grype gating HIGH/CRITICAL, keyless cosign signing, and
-# digest pinning of `deploy/k8s/kustomization.yaml` (strict lint re-run).
-# Release tags (`v*`): the same signed multi-arch build is published under
-# semver tags (X.Y.Z, X.Y, X, `latest`; a pre-release tag gets only X.Y.Z) and
-# a GitHub Release is created. `pin-digest` does not run on a tag (it tracks
-# master).
+# Per-push-to-master legs: the image is built per-arch on native runners
+# (`build` matrix: linux/amd64 on ubuntu-24.04, linux/arm64 on ubuntu-24.04-arm,
+# no QEMU), each pushed by digest with SBOM + provenance; the `merge` job then
+# assembles the multi-arch manifest (`imagetools create`), scans it
+# (trivy/grype gate HIGH/CRITICAL, syft SBOM artifact), keyless-signs the index
+# with cosign, and `pin-digest` writes the index digest into
+# `deploy/k8s/kustomization.yaml` (strict lint re-run).
+# Release tags (`v*`): the merged multi-arch image is published under semver
+# tags (X.Y.Z, X.Y, X, `latest`; a pre-release tag gets only X.Y.Z) and a GitHub
+# Release is created. `pin-digest` does not run on a tag (it tracks master).
 # Scheduled legs (weekly rebuild, Monday 04:00 UTC): rebuild + rescan + resign
 # so base-image digests advance even without code changes.
 #
@@ -54,26 +57,82 @@ jobs:
         run: python3 scripts/check_manifests.py
 
   build:
-    name: build
+    name: build (${{ matrix.arch }})
     if: github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
-    runs-on: ubuntu-24.04
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            platform: linux/amd64
+            arch: amd64
+          - runner: ubuntu-24.04-arm
+            platform: linux/arm64
+            arch: arm64
+    runs-on: ${{ matrix.runner }}
     permissions:
       contents: read
       packages: write
       id-token: write
-    outputs:
-      digest: ${{ steps.bake.outputs.digest }}
     steps:
       - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
-      - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
       - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
       - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Multi-arch bake with SBOM + provenance
+      - name: Build ${{ matrix.arch }} natively and push by digest
         id: bake
+        run: |
+          set -euo pipefail
+          IMAGE_TAGS="${REGISTRY_IMAGE}" docker buildx bake image \
+            --set "image.platform=${{ matrix.platform }}" \
+            --set "image.output=type=image,push=true,push-by-digest=true,name-canonical=true" \
+            --sbom=true \
+            --provenance=mode=max \
+            --metadata-file /tmp/bake-metadata.json
+          digest=$(python3 -c "
+          import json
+          meta = json.load(open('/tmp/bake-metadata.json'))
+          print(meta['image']['containerimage.digest'])
+          ")
+          mkdir -p "${RUNNER_TEMP}/digests"
+          touch "${RUNNER_TEMP}/digests/${digest#sha256:}"
+      - name: Upload ${{ matrix.arch }} digest
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: digests-${{ matrix.arch }}
+          path: ${{ runner.temp }}/digests/*
+          if-no-files-found: error
+          retention-days: 1
+
+  merge:
+    name: merge
+    needs: [build]
+    if: github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      packages: write
+      id-token: write
+    outputs:
+      digest: ${{ steps.merge.outputs.digest }}
+    steps:
+      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Download per-arch digests
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6
+        with:
+          path: ${{ runner.temp }}/digests
+          pattern: digests-*
+          merge-multiple: true
+      - name: Resolve publish tags
+        id: tags
         run: |
           set -euo pipefail
           if [[ "${GITHUB_REF}" == refs/tags/v* ]]; then
@@ -93,26 +152,28 @@ jobs:
             tags="${REGISTRY_IMAGE}:sha-${GITHUB_SHA::12}"
           fi
           echo "tags=${tags}" >> "${GITHUB_OUTPUT}"
-          IMAGE_TAGS="${tags}" docker buildx bake image \
-            --set image.output="type=registry" \
-            --sbom=true \
-            --provenance=mode=max \
-            --metadata-file /tmp/bake-metadata.json
-          python3 -c "
-          import json
-          meta = json.load(open('/tmp/bake-metadata.json'))
-          node = meta['image']['containerimage.digest']
-          open('${GITHUB_OUTPUT}', 'a').write(f'digest={node}\n')
-          "
+      - name: Create multi-arch manifest (imagetools create)
+        id: merge
+        working-directory: ${{ runner.temp }}/digests
+        run: |
+          set -euo pipefail
+          tag_args=()
+          IFS=',' read -ra tags <<< "${{ steps.tags.outputs.tags }}"
+          for tag in "${tags[@]}"; do tag_args+=(-t "${tag}"); done
+          docker buildx imagetools create "${tag_args[@]}" \
+            $(printf "${REGISTRY_IMAGE}@sha256:%s " *)
+          digest=$(docker buildx imagetools inspect "${tags[0]}" \
+            --format '{{json .Manifest.Digest}}' | tr -d '"')
+          echo "digest=${digest}" >> "${GITHUB_OUTPUT}"
       - name: SBOM (syft, committed as artifact)
         uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0
         with:
-          image: ${{ env.REGISTRY_IMAGE }}@${{ steps.bake.outputs.digest }}
+          image: ${{ env.REGISTRY_IMAGE }}@${{ steps.merge.outputs.digest }}
           artifact-name: sbom.spdx.json
       - name: Scan (trivy, gate HIGH/CRITICAL)
         uses: aquasecurity/trivy-action@6e7b7d1fd3e4fef0c5fa8cce1229c54b2c9bd0d8 # v0.24.0
         with:
-          image-ref: ${{ env.REGISTRY_IMAGE }}@${{ steps.bake.outputs.digest }}
+          image-ref: ${{ env.REGISTRY_IMAGE }}@${{ steps.merge.outputs.digest }}
           format: sarif
           output: trivy.sarif
           severity: HIGH,CRITICAL
@@ -120,7 +181,7 @@ jobs:
       - name: Scan (grype, gate high/critical)
         uses: anchore/scan-action@64a33b277ea7a1215a3c142735a1091341939ff5 # v4
         with:
-          image: ${{ env.REGISTRY_IMAGE }}@${{ steps.bake.outputs.digest }}
+          image: ${{ env.REGISTRY_IMAGE }}@${{ steps.merge.outputs.digest }}
           fail-build: true
           severity-cutoff: high
       - uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac # v3
@@ -128,13 +189,11 @@ jobs:
         env:
           COSIGN_YES: "true"
         run: |
-          cosign sign "${REGISTRY_IMAGE}@${{ steps.bake.outputs.digest }}"
-          cosign attest --predicate /tmp/bake-metadata.json \
-            "${REGISTRY_IMAGE}@${{ steps.bake.outputs.digest }}"
+          cosign sign "${REGISTRY_IMAGE}@${{ steps.merge.outputs.digest }}"
 
   pin-digest:
     name: pin-digest
-    needs: [build]
+    needs: [merge]
     # Only master/scheduled builds advance the committed digest; a release tag
     # must not rewrite kustomization.
     if: github.event_name != 'push' || !startsWith(github.ref, 'refs/tags/')
@@ -146,7 +205,7 @@ jobs:
           python-version: "3.x"
       - name: Pin kustomization to the built digest + strict lint
         env:
-          DIGEST: ${{ needs.build.outputs.digest }}
+          DIGEST: ${{ needs.merge.outputs.digest }}
         run: |
           pip install pyyaml
           python3 - "${DIGEST}" <<'EOF'
@@ -167,7 +226,7 @@ jobs:
 
   release:
     name: release
-    needs: [build]
+    needs: [merge]
     if: startsWith(github.ref, 'refs/tags/v')
     runs-on: ubuntu-24.04
     permissions:
@@ -177,7 +236,7 @@ jobs:
       - name: Create GitHub Release
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          DIGEST: ${{ needs.build.outputs.digest }}
+          DIGEST: ${{ needs.merge.outputs.digest }}
         run: |
           flags=()
           if [[ "${GITHUB_REF_NAME}" == *-* ]]; then

RUNBOOK.md

@@ -175,7 +175,7 @@ correlation. No step below writes a secret to a log.
 ## 6. Deploy provenance (signed image + digest record)
 
 Every rollout ships the signed `ghcr.io/syavne/d-oauth-server` image the
-`supply-chain.yml` `build` job signed (step `Keyless sign (cosign, OIDC)`)
+`supply-chain.yml` `merge` job signed (step `Keyless sign (cosign, OIDC)`)
 and the `pin-digest` job pinned into `deploy/k8s/kustomization.yaml`
 (step `Pin kustomization to the built digest + strict lint`). Nothing ships
 from an unverified local tarball. The full path is in

docs/DEPLOYMENT.md

@@ -59,8 +59,10 @@ IMAGE_TAGS="ghcr.io/syavne/d-oauth-server:v1.2.3,ghcr.io/syavne/d-oauth-server:l
   docker buildx bake image
 ```
 
-The release path in `.github/workflows/supply-chain.yml` drives this variable
-when a `v*` tag is pushed; see [Supply chain](#supply-chain).
+The CI release path does not use `IMAGE_TAGS` for its tag set: each build leg
+sets it to the image name and pushes that architecture by digest, and the
+`merge` job applies the release tags with `docker buildx imagetools create`;
+see [Supply chain](#supply-chain).
 
 Building `linux/arm64` on an amd64 host requires QEMU binfmt emulation. Docker
 Desktop bundles it; a bare Docker Engine does not. When it is missing, the arm64
@@ -513,24 +515,29 @@ on, see the TLS-terminating reverse proxy section above.
 The image pipeline lives in `.github/workflows/supply-chain.yml` and is gated
 on:
 
-- Multi-arch build with an SBOM and provenance attestation (job `build`, step
-  `Multi-arch bake with SBOM + provenance`: `docker buildx bake image` with
-  `--sbom=true --provenance=mode=max`).
-- Release publishing on a `v*` tag (job `build`, then job `release`): the same
-  signed multi-arch image is tagged `<X.Y.Z>`, `<X.Y>`, `<X>`, and `latest` (a
-  pre-release such as `v1.2.3-rc1` gets only `<X.Y.Z>` plus its sha), and a
+- Multi-arch build with SBOM and provenance (job `build`): each architecture is
+  built on a **native** runner (`linux/amd64` on `ubuntu-24.04`, `linux/arm64`
+  on `ubuntu-24.04-arm`, no QEMU) and pushed by digest with `--sbom=true
+  --provenance=mode=max`; job `merge` then assembles the multi-arch manifest
+  with `docker buildx imagetools create`. The per-arch attestations attach to
+  each platform manifest; the merged index itself carries no attestation.
+- Release publishing on a `v*` tag (jobs `build` + `merge`, then job `release`):
+  the merged multi-arch image is tagged `<X.Y.Z>`, `<X.Y>`, `<X>`, and `latest`
+  (a pre-release such as `v1.2.3-rc1` gets only `<X.Y.Z>` plus its sha), and a
   GitHub Release is created with generated notes. These tags are convenience
   references only; a production rollout still pins the digest (see
   [Deploy path](#deploy-path-use-the-signed-ghcrio-image)). `pin-digest` runs
   only on master/scheduled builds, never on a tag.
-- A vulnerability scan that fails on HIGH and CRITICAL (job `build`, steps
+- A vulnerability scan that fails on HIGH and CRITICAL (job `merge`, steps
   `Scan (trivy, gate HIGH/CRITICAL)` writing `trivy.sarif` and
   `Scan (grype, gate high/critical)`; the SBOM is committed as the
   `sbom.spdx.json` artifact by step `SBOM (syft, committed as artifact)`).
-- Keyless signing with `cosign` (job `build`, step
-  `Keyless sign (cosign, OIDC)`: `cosign sign` plus `cosign attest`).
+- Keyless signing with `cosign` (job `merge`, step
+  `Keyless sign (cosign, OIDC)`: `cosign sign` on the merged index digest; the
+  per-arch SBOM and SLSA provenance attestations are attached by the `build`
+  legs).
 - Digest pinning (job `pin-digest`, step
-  `Pin kustomization to the built digest + strict lint`): the signed build
+  `Pin kustomization to the built digest + strict lint`): the signed merge
   digest overwrites `images[].digest` in `deploy/k8s/kustomization.yaml`,
   then re-runs `python3 scripts/check_manifests.py --strict-digest`, with a
   periodic rebuild so base digests advance.
@@ -549,7 +556,7 @@ then records the exact digest it shipped:
 # 1. Read the pinned digest the `pin-digest` job committed.
 grep -A2 'name: ghcr.io/syavne/d-oauth-server' deploy/k8s/kustomization.yaml
 
-# 2. Verify the keyless cosign signature from the `build` job's
+# 2. Verify the keyless cosign signature from the `merge` job's
 #    `Keyless sign (cosign, OIDC)` step before pulling or applying.
 #    Match the identity to the signing workflow; the placeholder below
 #    stands for the real digest from step 1 (no real digest is recorded here).
@@ -662,10 +669,12 @@ d-oauth-server healthcheck --url http://127.0.0.1:8081/health/live
 - `deny.toml` for the supply-chain policy.
 - `RUNBOOK.md` for the admin bootstrap Job.
 - `.github/workflows/ci.yml` for the quality and documentation gates.
-- `.github/workflows/supply-chain.yml` for the image pipeline: multi-arch
-  bake with SBOM + provenance, trivy/grype HIGH/CRITICAL gates, keyless
-  cosign signing, digest pinning of `deploy/k8s/kustomization.yaml`, and the
-  weekly rebuild.
+- `.github/workflows/supply-chain.yml` for the image pipeline: per-arch native
+  build (linux/amd64 on `ubuntu-24.04`, linux/arm64 on `ubuntu-24.04-arm`) with
+  SBOM + provenance, a `merge` job that assembles the multi-arch manifest
+  (`docker buildx imagetools create`) and runs the trivy/grype HIGH/CRITICAL
+  gates plus keyless cosign signing, digest pinning of
+  `deploy/k8s/kustomization.yaml`, and the weekly rebuild.
 - `deploy/k8s/` for the cluster manifests: `namespace.yaml`, `configmap.yaml`
   (non-secret settings only, including the passwordless DSNs), `secret.yaml`
   (placeholder secrets — pepper, TOTP key, PII key, database and Redis

docs/SECURITY.md

@@ -69,7 +69,7 @@ class, or runtime setting).
 | Unrestricted pod egress (SSRF to metadata, exfiltration) | `networkpolicy.yaml` default-denies egress except DNS (53 to `kube-system`) and Postgres/Redis (5432/6379) scoped to the namespace hosting them — no internet-wide `0.0.0.0/0` peer; any `ipBlock` peer excludes `169.254.0.0/16`; ingress is limited to pod ports 8080/8081 | Enforcement depends on a NetworkPolicy-capable CNI; a cluster that ignores NetworkPolicy gets no benefit. A managed database outside the cluster needs a specific CIDR or a CNI FQDN policy. The default-deny also blocks the JAR `request_uri` HTTPS fetch — remote request objects are unsupported unless the deployment adds a scoped allowance (see `docs/DEPLOYMENT.md`). |
 | Operator `exec` into a running pod | No shell and a non-root, read-only-rootfs container limit what runs; every secret is a file the container user already reads, and no secret is in the environment | `pods/exec` (and `docker exec`) bypass the application's own authorization; restrict it with RBAC and audit it (`docs/OPERATIONS.md`). |
 | Kubernetes API access from the pod | `automountServiceAccountToken: false`; the binary never calls the API | A future feature that needs the API would have to re-enable it deliberately. |
-| Image/registry compromise | Digest pinning, keyless cosign signing in the `supply-chain.yml` `build` job (`Keyless sign (cosign, OIDC)`), verify-before-rollout with `cosign verify` on the pinned `ghcr.io/syavne/d-oauth-server@sha256:<digest>` (see `docs/DEPLOYMENT.md`), `pin-digest` writing the signed digest into `kustomization.yaml` with a `--strict-digest` lint, a committed digest reference with no shippable `:latest`, and a recorded digest per rollout in `RUNBOOK.md`; the local-build fallback still signs (`cosign sign`) and verifies (`cosign verify`) before loading | A registry compromise of an unpinned tag still ships if the operator skips the verify step; an unsigned `docker save \\| docker load` tarball bypasses every control, so it is forbidden except through the documented local sign/verify fallback. |
+| Image/registry compromise | Digest pinning, keyless cosign signing in the `supply-chain.yml` `merge` job (`Keyless sign (cosign, OIDC)`), verify-before-rollout with `cosign verify` on the pinned `ghcr.io/syavne/d-oauth-server@sha256:<digest>` (see `docs/DEPLOYMENT.md`), `pin-digest` writing the signed digest into `kustomization.yaml` with a `--strict-digest` lint, a committed digest reference with no shippable `:latest`, and a recorded digest per rollout in `RUNBOOK.md`; the local-build fallback still signs (`cosign sign`) and verifies (`cosign verify`) before loading | A registry compromise of an unpinned tag still ships if the operator skips the verify step; an unsigned `docker save \\| docker load` tarball bypasses every control, so it is forbidden except through the documented local sign/verify fallback. |
 | Rate-limit evasion via spoofed `X-Forwarded-For` behind an L7 proxy | Default-empty   `rate_limit.trusted_proxies` (headers ignored unless the TCP peer is an allowlisted proxy IP/CIDR); rightmost-untrusted selection; malformed headers fall back to the verified peer; attribution is bucket selection only, never authentication | An allowlist entry that is too broad (or a compromised proxy) attributes attacker-chosen IPs; keep entries to the proxy peers and never a `/0` (rejected at load). Proxies must emit bare IPs (no port/brackets/`unknown`); anything else is skipped, and only the rightmost 32 entries are considered. |
 
 Last reviewed: 2026-10-09. File-mounted DB/Redis passwords, digest pinning,