Acasă Explorează Ajutor
Înregistrare Autentificare
efren
/
testing-proxy
1
0
Bifurcare 0
Fisiere Probleme 0 Trageți solicitările 0 Wiki
Ramură: master
Ramuri Etichete
testing-proxy
/
docs
/
PLAN-V1-API.md

PLAN-V1-API.md 22 KB
Permalink Istoric Crud

v1-api-multi-provider - Work Plan

TL;DR (For humans)

What you'll get: An OpenAI-compatible /v1 API that proxies chat completion requests to three provider backends — Amazon Bedrock, Anthropic (direct API), and any generic OpenAI-compatible endpoint (vLLM, Ollama, LiteLLM, etc.). All endpoints are protected by your existing OAuth 2.1 Bearer tokens with llm scope enforcement. The architecture uses a clean provider abstraction: model IDs are prefixed (bedrock/claude-3-sonnet, anthropic/claude-3-5-sonnet, local/llama3.1), and each provider handles its own request/response translation. The minimum viable implementation covers POST /v1/chat/completions (streaming + non-streaming) and GET /v1/models.

Why this approach: Instead of hardcoding Bedrock-specific translation throughout the API layer, we abstract providers behind a unified interface (chat(messages, model, stream) → { response | stream }). This makes adding a 4th provider literally a matter of editing config.json — zero code changes. The Anthropic and generic OpenAI-compatible providers use native fetch(), keeping dependencies minimal. AWS Bedrock uses the official SDK for robustness.

What it will NOT do: It will not support legacy /v1/completions, embeddings, image generation, audio, files, or assistants. It will not auto-discover models from providers — all model mappings are explicit in config.json. It will not add CORS, JWT tokens, or client_credentials grants. It will not log message content.

Effort: Medium-High (~13 todos, 4 waves) Risk: Medium — Three different API shapes to translate; SSE streaming must exactly match OpenAI clients' expectations across all providers Decisions to sanity-check: Provider abstraction design, model ID prefix convention, config.json structure.

Your next move: Approve this plan, then start work with /start-work.

TL;DR (machine): Medium-High effort, Medium risk — OpenAI-compatible /v1 API proxy to Amazon Bedrock + Anthropic + generic OpenAI-compatible providers, OAuth 2.1 Bearer token validation against Redis, llm scope enforcement, SSE streaming support, provider abstraction architecture, SvelteKit +server.js handlers.

Scope

Must have

  • Bearer token validation helper (src/lib/oauth/bearer.js) — parses Authorization: Bearer <token>, looks up in Redis, checks expiry, returns token metadata
  • llm scope enforcement on all /v1/* endpoints — returns 403 if scope missing
  • OpenAI-compatible error format on all /v1/* endpoints — { error: { message, type, param, code } }
  • Provider abstraction router (src/lib/providers/index.js) — picks provider by model ID prefix, unified interface for all backends
  • Bedrock provider (src/lib/providers/bedrock.js) — AWS SDK, Converse/ConverseStream API, request/response translation
  • Anthropic provider (src/lib/providers/anthropic.js) — native fetch(), Messages API, request/response translation
  • Generic OpenAI-compatible provider (src/lib/providers/openai-compatible.js) — native fetch(), almost pure proxy, minimal translation
  • POST /v1/chat/completions endpoint — accepts OpenAI request format, routes to provider by model prefix, returns OpenAI response format
  • Streaming support for /v1/chat/completions — SSE data: {...} chunks with [DONE] terminator, matching OpenAI format across all providers
  • GET /v1/models endpoint — aggregates model lists from all configured providers
  • GET /v1/models/{model} endpoint — returns details for a specific model
  • Config.json updates: providers array with id, type, credentials, models mapping per provider
  • @aws-sdk/client-bedrock-runtime dependency
  • Model request/response translation layers per provider:
    • OpenAI messages[] ↔ provider-native message format
    • OpenAI usage ↔ provider-native usage/tokens
    • OpenAI choices[] ↔ provider-native response content
  • Provider health check / error propagation — Bedrock/AWS errors → OpenAI error format, Anthropic errors → OpenAI error format

Must NOT have (guardrails, anti-slop, scope boundaries)

  • Legacy /v1/completions endpoint (non-chat) — all providers are chat-native
  • /v1/embeddings — out of scope, no provider embedding config
  • /v1/images/*, /v1/audio/*, /v1/files/*, /v1/assistants/* — no provider equivalent
  • JWT access token support — existing opaque tokens only
  • CORS headers on /v1/* — not needed for server-to-server client calls
  • Logging of message content or prompts — PII guardrail
  • AWS credentials or API keys hardcoded in source files — only in config.json
  • Provider auto-discovery or model listing from provider APIs — static config only
  • Request/response transformation caching — stateless per-request translation only
  • Custom inference parameters beyond temperature, max_tokens, top_p, stop — keep translation minimal
  • Provider fallback / retry logic — single provider per request, no failover

Verification strategy

Zero human intervention - all verification is agent-executed.

  • Test decision: tests-after + agent-executed QA via curl/httpie against running dev server
  • Evidence: .omo/evidence/task-<N>-v1-api-multi-provider.<ext>

Execution strategy

Parallel execution waves

Target 5-8 todos per wave.

Wave 1: Foundation — Config, dependency, bearer helper, provider abstraction skeleton Wave 2: Provider Implementations — Bedrock, Anthropic, generic OpenAI-compatible providers Wave 3: API Endpoints — Chat completions (non-streaming), models list/retrieve Wave 4: Streaming & Final QA — SSE streaming, full integration testing, docs

Dependency matrix

Todo Depends on Blocks Can parallelize with
1 — 2,3,4,5 —
2 1 6,7,8,9 —
3 1 6,7,8,9 2
4 1 10,11 2,3
5 1 10,11 2,3,4
6 2,3,4 10,11 7,8,9
7 2,3,4 10,11 6,8,9
8 2,3,4 10,11 6,7,9
9 2,3,4 10,11 6,7,8
10 6,7,8,9 12 —
11 6,7,8,9 12 10
12 10,11 13 —
13 12 — —

Todos

  • [x] 1. Update config.json.example with multi-provider settings What to do / Must NOT do: Replace existing aws section (if any) with providers array. Each provider object has: id (string, e.g. "bedrock", "anthropic", "local"), type ("bedrock" | "anthropic" | "openai-compatible"), enabled (boolean), and provider-specific config. For Bedrock: region, credentials { accessKeyId, secretAccessKey }. For Anthropic: apiKey, baseUrl (optional, defaults to https://api.anthropic.com/v1). For generic: baseUrl. All providers have models object mapping OpenAI model IDs to provider model IDs. Must NOT include real credentials in example — use placeholders. Must NOT remove existing baseUrl/database/password sections. Parallelization: Wave 1 | Blocked by: — | Blocks: 2,3,4,5 References: config.json.example:1-32, src/lib/config.js:1-19 Acceptance criteria: cat config.json.example | grep -q providers && cat config.json.example | grep -q bedrock && cat config.json.example | grep -q anthropic QA scenarios: Happy: valid JSON parses. Failure: invalid JSON breaks parse. No commit — user handles all commits manually

  • [x] 2. Install AWS Bedrock Runtime SDK dependency What to do / Must NOT do: pnpm add @aws-sdk/client-bedrock-runtime. Must NOT install full @aws-sdk/client-bedrock or unrelated AWS packages. Must ensure package is recorded in package.json. Parallelization: Wave 1 | Blocked by: 1 | Blocks: 6,7,8,9 References: package.json, pnpm-workspace.yaml (native builds) Acceptance criteria: node -e "import('@aws-sdk/client-bedrock-runtime').then(m => console.log(typeof m.BedrockRuntimeClient))" prints "function" QA scenarios: Happy: SDK imports successfully. Failure: missing dependency throws MODULE_NOT_FOUND. No commit — user handles all commits manually

  • [x] 3. Create Bearer token validation helper What to do / Must NOT do: Create src/lib/oauth/bearer.js exporting validateBearerToken(request, requiredScope) which: (a) extracts Authorization: Bearer <token> header, (b) looks up access_token:{token} in Redis, (c) parses JSON metadata, (d) checks if expired, (e) checks if scope contains requiredScope, (f) returns { valid: true, clientId, userId, scope } or { valid: false, error, status }. Must NOT accept tokens from query strings. Must NOT accept JWT tokens differently from opaque tokens. Must use constant-time comparison for token prefix if any. Parallelization: Wave 1 | Blocked by: 1 | Blocks: 10,11 References: src/lib/redis.js:1-28, src/routes/oauth/token/+server.js:137-158 (token storage format), RFC 6750 Acceptance criteria: node -e "import('./src/lib/oauth/bearer.js').then(m => console.log(typeof m.validateBearerToken))" prints "function" QA scenarios: Happy: Valid active token with correct scope returns valid=true. Failure: Missing header returns 401. Invalid token returns 401. Expired token returns 401. Wrong scope returns 403. No commit — user handles all commits manually

  • [x] 4. Create provider abstraction router What to do / Must NOT do: Create src/lib/providers/index.js exporting: getProvider(modelId) which parses providerSlug/localModelId prefix (e.g. "bedrock/claude-3-sonnet" → bedrock provider, "anthropic/claude-3-5-sonnet" → anthropic provider), listProviders() returning enabled providers, listModels() aggregating all provider model mappings. Must validate that model ID contains exactly one / separator. Must return clear error for unknown provider or unknown model. Must NOT instantiate providers on every request — cache enabled providers. Parallelization: Wave 1 | Blocked by: 1 | Blocks: 10,11 References: src/lib/config.js:1-19, config.json provider structure Acceptance criteria: node -e "import('./src/lib/providers/index.js').then(m => m.getProvider('bedrock/test').then(p => console.log(p.id)))" prints "bedrock" QA scenarios: Happy: Valid model ID resolves to correct provider. Failure: Missing prefix returns clear error. Unknown provider returns clear error. Disabled provider returns clear error. No commit — user handles all commits manually

  • [x] 5. Create generic OpenAI-compatible provider What to do / Must NOT do: Create src/lib/providers/openai-compatible.js implementing the provider interface: chat(messages, modelId, options) and chatStream(messages, modelId, options) using native fetch() to provider's baseUrl. Request body is nearly pass-through OpenAI format. Response is nearly pass-through OpenAI format — minimal translation. Must handle baseUrl with or without trailing /v1. Must pass Authorization: Bearer <apiKey> if configured. Must set Content-Type: application/json. Must handle SSE streaming response. Must NOT modify message content. Must NOT log request body. Parallelization: Wave 1 | Blocked by: 1 | Blocks: 10,11 References: src/lib/providers/index.js, OpenAI chat completions spec, native fetch() API Acceptance criteria: node -e "import('./src/lib/providers/openai-compatible.js').then(m => console.log(typeof m.chat))" prints "function" QA scenarios: Happy: Provider initializes with valid config. Failure: Invalid baseUrl throws clear error. Missing API key (if required) throws clear error. No commit — user handles all commits manually

  • [x] 6. Create Bedrock provider module What to do / Must NOT do: Create src/lib/providers/bedrock.js implementing provider interface: chat(messages, modelId, options) and chatStream(messages, modelId, options). Use @aws-sdk/client-bedrock-runtime with ConverseCommand / ConverseStreamCommand. Translate OpenAI messages[] to Bedrock messages[] with content blocks (text only). Translate Bedrock response to OpenAI format: id (generated), object: "chat.completion", created (unix timestamp), model (OpenAI model ID), choices[].message.role/content, usage.prompt_tokens/completion_tokens/total_tokens. For streaming: translate ConverseStream events to OpenAI chat.completion.chunk SSE format. Must NOT create AWS client on every request — use singleton per provider config. Must handle AWS region and credentials from config. Must NOT expose AWS credentials in error messages. Parallelization: Wave 2 | Blocked by: 2,3,4 | Blocks: 10,11 References: src/lib/providers/index.js, @aws-sdk/client-bedrock-runtime docs, Bedrock Converse API docs, config.json Acceptance criteria: node -e "import('./src/lib/providers/bedrock.js').then(m => console.log(typeof m.chat))" prints "function" QA scenarios: Happy: Client initializes with valid config. Failure: Invalid AWS credentials throws clear error. Missing model mapping throws clear error. Invalid region throws clear error. No commit — user handles all commits manually

  • [x] 7. Create Anthropic provider module What to do / Must NOT do: Create src/lib/providers/anthropic.js implementing provider interface: chat(messages, modelId, options) and chatStream(messages, modelId, options) using native fetch() to https://api.anthropic.com/v1/messages (or configured baseUrl). Translate OpenAI request to Anthropic Messages API format: model → provider model ID, messages[] → messages[] (map role: system to top-level system param), max_tokens required (default to 4096 if missing), temperature and top_p passed through, stream boolean. Translate Anthropic response to OpenAI format: content[].text → choices[].message.content, usage.input_tokens/output_tokens → usage.prompt_tokens/completion_tokens. For streaming: read SSE from Anthropic, translate content_block_delta events to OpenAI chat.completion.chunk format. Must pass x-api-key and anthropic-version: 2023-06-01 headers. Must NOT log request body. Must handle Anthropic-specific errors (e.g. overloaded_error). Parallelization: Wave 2 | Blocked by: 2,3,4 | Blocks: 10,11 References: src/lib/providers/index.js, Anthropic Messages API docs, native fetch() API Acceptance criteria: node -e "import('./src/lib/providers/anthropic.js').then(m => console.log(typeof m.chat))" prints "function" QA scenarios: Happy: Provider initializes with valid config. Failure: Invalid API key throws clear error. Missing model mapping throws clear error. Malformed response handled gracefully. No commit — user handles all commits manually

  • [x] 8. Implement POST /v1/chat/completions (non-streaming) What to do / Must NOT do: Create src/routes/v1/chat/completions/+server.js with POST handler. Steps: (1) validate Bearer token + llm scope via bearer helper, (2) parse JSON body, (3) resolve provider from model ID prefix via getProvider(model), (4) call provider.chat() with translated params, (5) return OpenAI-format JSON response. Must NOT support stream: true in this todo — handled in todo 12. Must NOT log message content. Must return 400 for unsupported parameters with OpenAI error format. Must handle provider errors and translate to OpenAI error format (502 for provider failure). Parallelization: Wave 3 | Blocked by: 3,4,5,6,7 | Blocks: 12 References: src/lib/oauth/bearer.js, src/lib/providers/index.js, all provider modules, OpenAI chat completions spec Acceptance criteria: curl -s -X POST http://localhost:3000/v1/chat/completions -H "Authorization: Bearer VALID_TOKEN" -H "Content-Type: application/json" -d '{"model":"bedrock/claude-3-sonnet","messages":[{"role":"user","content":"Hello"}]}' | jq '.object' returns "chat.completion" QA scenarios: Happy: Valid token + valid request returns OpenAI-format response with choices and usage. Failure: Missing auth returns 401. Wrong scope returns 403. Invalid model returns 400 (unknown provider prefix or unknown model). Bad request body returns 400. Provider error returns 502 with OpenAI error format. No commit — user handles all commits manually

  • [x] 9. Implement GET /v1/models and GET /v1/models/{model} What to do / Must NOT do: Create src/routes/v1/models/+server.js with GET handler returning { object: "list", data: [...] } where each model has id (full prefixed ID), object: "model", created (static or current timestamp), owned_by: "<provider-id>". Create src/routes/v1/models/[model]/+server.js with GET handler returning single model object. Both must validate Bearer token + llm scope. Data sourced from all enabled providers' models config. Must NOT query provider APIs for model list — static config only. Must return 404 for unknown model IDs. Parallelization: Wave 3 | Blocked by: 3,4,5,6,7 | Blocks: 12 References: src/lib/oauth/bearer.js, src/lib/providers/index.js, OpenAI models spec Acceptance criteria: curl -s http://localhost:3000/v1/models -H "Authorization: Bearer VALID_TOKEN" | jq '.object' returns "list" QA scenarios: Happy: Valid token returns aggregated model list from all providers. Valid model ID returns model details. Failure: Missing auth returns 401. Unknown model returns 404. No commit — user handles all commits manually

  • [x] 10. Add SSE streaming support to /v1/chat/completions What to do / Must NOT do: Extend existing +server.js from todo 8 to handle stream: true in request body. Resolve provider, call provider.chatStream(), read async iterator, translate each chunk to OpenAI chat.completion.chunk format, send as SSE with data: {...} prefix, end with data: [DONE]. Must set Content-Type: text/event-stream and Cache-Control: no-cache. Must handle stream errors gracefully — send final error chunk or close connection. Must NOT buffer entire stream before sending. All three providers must produce identical SSE output format. Parallelization: Wave 4 | Blocked by: 8,9 | Blocks: 13 References: src/routes/v1/chat/completions/+server.js, all provider modules, OpenAI streaming spec Acceptance criteria: curl -s -N -X POST http://localhost:3000/v1/chat/completions -H "Authorization: Bearer VALID_TOKEN" -H "Content-Type: application/json" -d '{"model":"bedrock/claude-3-sonnet","messages":[{"role":"user","content":"Hello"}],"stream":true}' | grep -q 'data: ' QA scenarios: Happy: Streaming request returns SSE chunks with valid JSON deltas and [DONE]. Failure: Stream error closes connection gracefully. Invalid stream param returns 400. No commit — user handles all commits manually

  • [x] 11. Add provider error translation to OpenAI error format What to do / Must NOT do: Create src/lib/providers/errors.js exporting translateProviderError(error, providerType) which normalizes provider-specific errors into OpenAI error format: { error: { message, type, param, code } }. Handle AWS SDK errors (e.g. ThrottlingException, ValidationException), Anthropic errors (e.g. overloaded_error, invalid_request_error), and HTTP errors from generic providers. Must NOT expose provider credentials or internal URLs in error messages. Must map provider error codes to reasonable OpenAI equivalents. Parallelization: Wave 3 | Blocked by: 6,7 | Blocks: 8,9,10 References: All provider modules, OpenAI error format spec Acceptance criteria: node -e "import('./src/lib/providers/errors.js').then(m => console.log(typeof m.translateProviderError))" prints "function" QA scenarios: Happy: AWS throttling returns 429 with OpenAI error format. Anthropic overload returns 503 with OpenAI error format. Generic 401 returns 502 with OpenAI error format. No commit — user handles all commits manually

  • [x] 12. Final integration QA and documentation What to do / Must NOT do: Run dev server. Execute full flow for each provider: obtain OAuth token → call /v1/models → call /v1/chat/completions (non-streaming) → call /v1/chat/completions (streaming) → verify error formats. Update docs/ with /v1 API documentation including provider setup and model ID examples. Must NOT skip negative testing (invalid token, wrong scope, missing auth, unknown provider, unknown model). Must verify OpenAI Python SDK can connect. Parallelization: Wave 4 | Blocked by: 10,11 | Blocks: — References: All /v1 endpoint files, existing OAuth endpoints, all provider modules, OpenAI Python SDK Acceptance criteria: Full end-to-end flow executes successfully for at least one configured provider (token issuance to streaming chat completion). QA scenarios: Happy: Complete flow with valid token for each provider type. Failure: Invalid token returns 401 on all /v1 endpoints. Missing llm scope returns 403. Unknown provider prefix returns 400. Unknown model returns 404. Provider error returns 502 with OpenAI error format. No commit — user handles all commits manually

Final verification wave

Runs in parallel after ALL todos. ALL must APPROVE.

  • F1. Plan compliance audit — Verify every must-have is implemented and every must-not-have is absent
  • F2. Code quality review — Verify consistent patterns across all providers, no hardcoded credentials, no PII logging
  • F3. Real manual QA — Execute full OAuth → /v1 flow end-to-end with curl for at least one provider
  • F4. Scope fidelity — Confirm no /v1/embeddings, no /v1/completions, no CORS, no JWT support, no PII logging, no provider auto-discovery

Commit strategy

  • NO COMMITS — user handles all commits manually when ready
  • Worker makes file changes only; git commit decisions belong to user
  • User may commit everything at once, or per-todo, or not at all — their call

Success criteria

  1. POST /v1/chat/completions accepts OpenAI-format requests with prefixed model IDs (e.g. bedrock/claude-3-sonnet) and returns OpenAI-format responses (non-streaming)
  2. POST /v1/chat/completions with stream: true returns SSE with chat.completion.chunk format and [DONE] terminator
  3. GET /v1/models returns aggregated OpenAI-format model list from all configured providers
  4. GET /v1/models/{model} returns OpenAI-format model object or 404
  5. All /v1/* endpoints require valid Bearer token from Redis OAuth flow
  6. All /v1/* endpoints require llm scope — tokens without it return 403
  7. Invalid/missing Bearer tokens return 401 with OpenAI error format
  8. Provider errors (AWS, Anthropic, generic) return 502 with OpenAI error format
  9. Credentials and model mappings live only in config.json (not source code)
  10. No message content or prompts logged anywhere
  11. Full end-to-end flow works: OAuth login → authorize → token → /v1/chat/completions
  12. Adding a new provider requires only config.json changes (no code changes)