feat(routing): add LiteLLM backend to run open-weight models on the Claude Code harness

[CarlesUIPath]

Summary

This PR adds open-weight model support to coder_eval: evaluate coder_eval tasks on non-Claude models (GLM, DeepSeek, Kimi) by driving them through the Claude Code SDK harness. This way we can run any non-Claude LLM using the native Skill tool, with skill activation measured by coder_eval.

The SDK sends Anthropic messages. The target models don't, since openweight models in Bedrock use Converse messaging, or OpenRouter uses OpenAI format. For this reason, we introduce a LiteLLM proxy which functions as follows.

Claude Code SDK ──Anthropic /v1/messages──▶ LiteLLM (localhost:4000) ──▶ Bedrock Converse (eu-north-1)
  ANTHROPIC_BASE_URL=localhost:4000                                   └──▶ OpenRouter (OpenAI format)

Main changes

models/routing.py

• New LiteLLMRoute (base_url / auth_token / model / small_model), added to
  the ApiRoute union and ROUTE_NAMES ("litellm"). Model ids are passed
  verbatim — no Bedrock inference-profile qualification; the gateway maps them.
• resolve_route gains an ApiBackend.LITELLM arm that builds the route from
  LITELLM_BASE_URL / LITELLM_AUTH_TOKEN / LITELLM_MODEL (small-model falls
  back to the main model).

agents/claude_code_agent.py

• _build_sdk_env gains a LiteLLMRoute case — the key lever. It points the
  unmodified Claude Code SDK at the proxy by setting ANTHROPIC_BASE_URL +
  ANTHROPIC_AUTH_TOKEN (bearer) + ANTHROPIC_MODEL / ANTHROPIC_SMALL_FAST_MODEL.
• It neutralizes inherited credentials so the CLI can't silently bypass the
  proxy: it blanks ANTHROPIC_API_KEY (would override the bearer) and
  AWS_BEARER_TOKEN_BEDROCK + CLAUDE_CODE_USE_BEDROCK (the CLI auto-selects
  Bedrock-direct whenever the bearer token is present, skipping ANTHROPIC_BASE_URL).
• _reprice_for_litellm — the SDK returns a Claude-priced cost estimate that's wrong
  for an open-weight model, so for litellm runs coder_eval discards it and recomputes
  cost from the token buckets at the model's real rate.

Other changes

• --backend litellm route (points ANTHROPIC_BASE_URL at the proxy),
  alongside the existing direct / bedrock backends.
• Pricing registered in both src/coder_eval/pricing.py and
  evalboard/lib/pricing.ts (parity guard relaxed to subset-match so the
  frontend only mirrors what it displays).
• evalboard: the synthetic reconciliation row is now priced, so the
  per-stage Cost column sums to the turn total on sparse-stream (LiteLLM) runs.
• litellm/ — proxy config, start-litellm.sh, and a README.md (when to
  run the proxy, config format, how to add a model).
• Example models: Bedrock (GLM 5, DeepSeek V3.2, Kimi K2.5) and OpenRouter
  (Kimi K3, GLM 5.2, DeepSeek V4 Pro); OpenRouter entries pin the provider, which is picked sorting by the cheapest available.

Testing

Unit coverage for the new surface — all green (test_litellm_route.py 32,
test_routing.py 39, test_docker_litellm_env.py 8, test_pricing_registry.py 10):

• Routing — LiteLLMRoute resolution from settings, model passed verbatim
  (no inference-profile), small-model fallback, union/ROUTE_NAMES registration,
  validate_api_keys for the litellm backend, and the LITELLM backend enum.
• Agent _build_sdk_env — sets ANTHROPIC_BASE_URL/AUTH_TOKEN/MODEL,
  blanks inherited ANTHROPIC_API_KEY + Bedrock creds
  (CLAUDE_CODE_USE_BEDROCK/AWS_BEARER_TOKEN_BEDROCK), PATH forwarding, and the
  container loopback-URL rewrite/forwarding.
• Cost — open-weight rates + converse-prefix normalization, and
  _reprice_for_litellm overriding the SDK's Claude-priced estimate with the
  model's real rate.
• Preflight — proxy-down vs reachable vs HTTP-error-means-up.
• evalboard — reconciliation-row cost so the per-stage Cost column sums to the
  turn total (parseMessages.test.ts), and Python↔TS pricing parity covering the
  OpenRouter models (pricing-parity.test.ts).

Full suite green except pre-existing live-integration tests that need real API
credentials/network (unrelated to this change).

Limitations

• Not blocking: on the litellm path the per-message token
  stream is sparse — the provider reports only the turn total, so most tokens land
  in the synthetic reconciliation row. This PR prices that row so the per-stage Cost
  column still sums to the turn total in evalboard; the turn total is correct, only
  the per-stage split is coarse. Full per-stage granularity would need per-chunk
  usage from LiteLLM (stream_options: {include_usage: true}).

Remaining TODOs

• (feat) Accurate OpenRouter cost. coder_eval prices litellm runs from a static
  table, but OpenRouter picks a provider per request (further filtered by the
  account's data policy), which can differ from the model page's headline rate the
  table uses (observed: a DeepSeek V4 Pro
  run billed at StreamLake/Novita rates, not the $0.435 headline). Token counts are
  exact; only the per-token rate is affected.
  • Fix: source the real cost dynamically: read OpenRouter's actual per-run cost
    (account usage-delta) rather than a static rate.
  • Scope: benefits any OpenRouter-routed model, beyond this PR.

[github-actions]

Claude finished @CarlesUIPath's task in 1m 48s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review guidelines
• Read CLAUDE.md for project conventions
• Examine full diff (git diff origin/main...HEAD)
• Review each changed file with full context
• Perform cross-file consistency checks
• Conduct "what's missing" analysis
• Provide design-level scrutiny
• Format and submit final review

[uipreliga]

Review: coder_eval — pr:54 (24 files) axis:1,2,3,4,5,6,7,8

Scope: pr:54 (24 files) axis:1,2,3,4,5,6,7,8 · branch pr-54 (head: dev_create_openweight_support) · cdc48f4 · 2026-07-27T17:07Z · workflow variant

Change class: complex — adds a new ApiBackend value (LITELLM) plus a new ApiRoute variant, threads it through config/CLI/orchestrator/docker env/pricing, and adds a network preflight; correctness requires reasoning about route-seam exhaustiveness, env-var precedence, and credential handling

The LiteLLM backend lands on a genuinely healthy base — architecture 10/10, security 9.8/10, and a clean route-seam design — but the same PR widens ApiRoute without carrying the new member through the judge dispatch or the evalboard pricing mirror, so on --backend litellm an identical agent trajectory can silently score 0.0 on every llm_judge criterion, lose its cost (which quietly disables the max_usd gate), and fail unhelpfully under --driver docker on Linux; the real risks are all narrow, well-localized seam omissions plus untested wiring and completely absent user-facing docs, so the bottom line is: fix the six seams below and this is mergeable, not a redesign.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.4 / 10	0	0	2	6	_reprice_for_litellm duplicates _backfill_cost's calculate_cost block, declares a -> TokenUsage return that all 5 call sites discard, and nulls turn cost on a rate-card miss with no warning (unlike its sibling)
2. Type Safety	7.9 / 10	0	2	0	1	New LiteLLMRoute union member is unhandled at the judge/route dispatch seams — llm_judge falls through its "unreachable" case _: and silently scores every judged criterion 0.0 on --backend litellm (agent_judge/simulator inherit an unservable model; the new exhaustiveness guard does not cover this seam)
3. Test Health	8.5 / 10	0	1	1	0	New LiteLLM code paths lack test coverage: repricing wiring, two _validate_litellm_settings branches, the preflight abort branch, the IPv6 loopback rewrite, and the missing attribution-header assertion
4. Security	9.8 / 10	0	0	0	2	LiteLLM endpoint + host alias are forwarded into task containers with no api_backend == LITELLM gate, so a --backend direct docker run still hands the sandboxed agent a working paid gateway
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	9.2 / 10	0	0	1	3	On Linux hosts, --driver docker --backend litellm cannot connect: start-litellm.sh:88 hardcodes --host 127.0.0.1 (no override) while docker_runner.py:1131 points containers at host.docker.internal (bridge gateway), and the host-side preflight (run_command.py:89 probes the un-rewritten URL) reports green — so failures surface per task after N containers start
7. API Surface & Maintainability	9.4 / 10	0	0	1	1	New litellm backend ships with no docs/.env.example updates: USER_GUIDE and .env.example still say "direct or bedrock", the four LITELLM_* settings (LITELLM_SMALL_MODEL entirely) are undocumented, and the new top-level litellm/ dir is outside the documented repo tree and docs SSOT
8. Evaluation Harness Quality	8.5 / 10	0	1	1	0	pricing.py↔pricing.ts parity guard relaxed to a one-way subset in the same PR that leaves the 3 new Bedrock open-weight ids unmirrored and resolvePricing without the converse/ prefix strip — Cost column renders "—" for Bedrock-routed LiteLLM runs

Overall Score: 9 / 10 · Weakest Axis: Type Safety at 7.9 / 10
Totals: 🔴 0 · 🟠 4 · 🟡 6 · 🔵 13 across 8 axes.

Blockers

1. [Axis 2] New LiteLLMRoute union member is unhandled at the judge/route dispatch seams — llm_judge falls through its "unreachable" case _: and silently scores every judged criterion 0.0 on --backend litellm (agent_judge/simulator inherit an unservable model; the new exhaustiveness guard does not cover this seam) (src/coder_eval/models/routing.py:121) — routing.py:121 extends the union: ApiRoute = DirectRoute | BedrockRoute | LiteLLMRoute, but the judge-dispatch seam was not extended. criteria/llm_judge.py:182-209 still matches only two arms and ends with:

        case _:
            # route is None or an unexpected type — the unconfigured-arm guard in
            # _check_impl handles None before dispatch, so this is defensive only.
            return None, "llm_judge: no usable API route", "(no route)", None

That comment is now false: LiteLLMRoute is a supported production backend that reaches it. The upstream short-circuit at llm_judge.py:99 also does not cover it — if route is None or (isinstance(route, DirectRoute) and route.judge_transport is None): — and its comment at line 98 ("The Bedrock backend always has a usable judge transport") is now incomplete, so the run pays to build the full judge context and then returns JudgeCriterionResult(score=0.0, error="llm_judge: no usable API route"). grep -rln llm_judge tasks/ returns 3 task files, so any of them run with --backend litellm silently loses that criterion's score under a message that misreports the cause (a route exists; it is just unhandled).

Fix: add a case LiteLLMRoute(): arm invoking the Anthropic-compatible endpoint via route.base_url / route.auth_token (the gateway speaks Messages, which is the whole premise of the PR), or — if judging on an open-weight model is deliberately out of scope — reject it in the early guard at llm_judge.py:99 so the operator gets that arm's actionable error instead of a 0.0. Then replace case _: with case None: plus assert_never(route) so pyright flags the next union member at the seam. Finally extend tests/test_route_seam_exhaustiveness.py — its docstring claims "every route-matching seam" but it enumerates only _build_sdk_env, _format_routing and ROUTE_NAMES (lines 34-50); the judge seam and Orchestrator._record_route_environment_info are both unguarded.
2. [Axis 2] Scheme-less LITELLM_BASE_URL makes the preflight raise a bare, uncaught ValueError instead of a clean typer.Exit(1) (src/coder_eval/config.py:112) — config.py:112 declares litellm_base_url: str | None = None with no URL type or validator, and the only check is truthiness — config.py:190: if not self.litellm_base_url: missing.append("LITELLM_BASE_URL"). Two consumers then assume a well-formed absolute URL.

cli/run_command.py:85-89 builds and opens it:

    url = f"{current_settings.litellm_base_url.rstrip('/')}/health/liveliness"
    try:
        urllib.request.urlopen(url, timeout=5).close()  # nosec B310
    except urllib.error.HTTPError:
        return None
    except (urllib.error.URLError, OSError) as exc:

Verified by execution: LITELLM_BASE_URL=my-proxy.internal (scheme omitted, no colon) makes urlopen raise ValueError: unknown url type: 'my-proxy.internal/health/liveliness' — ValueError is neither URLError nor OSError, so it escapes _litellm_preflight_error as a raw traceback out of _run_all_tasks instead of the clean typer.Exit(1) the surrounding code is built to produce. LITELLM_BASE_URL=localhost:4000 is caught, but only by accident, and reports the misleading "proxy not reachable ... unknown url type: localhost" rather than "missing scheme". The same missing validation makes orchestrator.py:1097 degrade silently — urlparse(self.route.base_url).hostname or "" records an empty litellm_base_url_host in environment_info for any scheme-less value (reachable when the CLI preflight is bypassed, e.g. library use of Orchestrator), losing the one routing dimension the PR added for cross-run comparison.

Fix: tighten the field to a validated URL — e.g. litellm_base_url: AnyHttpUrl | None = None (pydantic already ships it, and str(...) at the two use sites keeps the SDK env a plain string), or add a @field_validator that requires an http/https scheme and a non-empty host and raises naming LITELLM_BASE_URL. Either way the operator error surfaces at settings load with a field name, and the two downstream urlopen/urlparse assumptions become type-guaranteed. Add a test for the scheme-less input — tests/test_litellm_route.py::TestLitellmPreflight only exercises well-formed http://127.0.0.1:* values (lines 261-286).
3. [Axis 3] New LiteLLM code paths lack test coverage: repricing wiring, two _validate_litellm_settings branches, the preflight abort branch, the IPv6 loopback rewrite, and the missing attribution-header assertion (src/coder_eval/agents/claude_code_agent.py:610) — Coverage confirms line 610 is in the miss list for agents/claude_code_agent.py (95.65%: missing 151, 298, 454, 559, 610, 871, …). The guard executes but the body never does:

609:        if isinstance(self._agent.route, LiteLLMRoute):
610:            self._agent._reprice_for_litellm(usage, self.effective_model)

tests/test_litellm_route.py::TestRepriceForLitellm only calls the static method directly with a hand-built TokenUsage; nothing constructs a _ClaudeTurnState/TurnRecord under a LiteLLMRoute, so the only line that makes repricing affect a real run is unexercised. Failure scenario: delete or invert line 609 and the whole suite still passes — every LiteLLM turn then persists the SDK's Claude-priced total_cost_usd into task.json/reports, and (for an unpriced id) total_cost_usd=None makes orchestrator.py:803 log "max_usd budget configured but no turn reported cost; skipping cost check", silently disabling the cost gate. Add a test that drives _ClaudeTurnState/build_turn_record (or ClaudeCodeAgent with a stubbed SDK stream) under LiteLLMRoute(model="zai.glm-5") and asserts the resulting TurnRecord.token_usage.total_cost_usd equals the rate-table figure, plus one with an unpriced model asserting None and that the max_usd gate is skipped. n/a
4. [Axis 8] pricing.py↔pricing.ts parity guard relaxed to a one-way subset in the same PR that leaves the 3 new Bedrock open-weight ids unmirrored and resolvePricing without the converse/ prefix strip — Cost column renders "—" for Bedrock-routed LiteLLM runs (evalboard/lib/__tests__/pricing-parity.test.ts:52) — The bidirectional assertion expect(Object.keys(PRICING).sort()).toEqual(Object.keys(py).sort()) was replaced by const orphans = Object.keys(PRICING).filter((m) => !(m in py)); (line 53) and the rate loop inverted to for (const [model, ts] of Object.entries(PRICING)) (line 61) — nothing now iterates the Python keys, so a model priced in pricing.py but missing from pricing.ts no longer fails the build. This PR then exercises exactly that hole: 6 keys added at src/coder_eval/pricing.py:95-105, only 3 at evalboard/lib/pricing.ts:58-60. deepseek.v3.2, zai.glm-5 and moonshotai.kimi-k2.5 — the three Bedrock open-weight models the litellm config actually ships (litellm/litellm-config.yaml:26-39) — are absent from the TS table (grep -n 'deepseek\|glm\|kimi' evalboard/lib/pricing.ts returns only lines 58-60, the OpenRouter ids). Compounding it, resolvePricing (evalboard/lib/pricing.ts:86-92) does exact match plus a -YYYYMMDD strip only, and was NOT given the bedrock/converse/ / bedrock/ / converse/ stripping that _normalize_model gained at src/coder_eval/pricing.py:162-167 — and the PR's own Python test documents that the recorded id arrives prefixed: tests/test_litellm_route.py:229 says "The SDK reports model_used as e.g. 'converse/zai.glm-5'". So on a Bedrock-open-weight litellm run messageCostUsd returns null and the newly-priced reconciliation row plus the whole per-message Cost column render "—", which is both the symptom the PR's own doc names (litellm/README.md:193: "evalboard cost column blank for a model | Model missing from evalboard/lib/pricing.ts") and the exact bug evalboard/lib/runs.ts:1722-1739 was added to fix. Fix: add the three Bedrock ids to pricing.ts, mirror _normalize_model's prefix stripping inside resolvePricing (and assert that parity too, since the tables matching is not sufficient when the lookups differ), and either restore the Python→TS direction of the key assertion or replace it with an explicit, commented allowlist of deliberately-unmirrored keys so a real omission still breaks the build.

Non-blocking, but please consider before merge

1. [Axis 1] _reprice_for_litellm duplicates _backfill_cost's calculate_cost block, declares a -> TokenUsage return that all 5 call sites discard, and nulls turn cost on a rate-card miss with no warning (unlike its sibling) (src/coder_eval/agents/claude_code_agent.py:1450) — _reprice_for_litellm (line 1450) repeats the identical 6-line invocation already in its immediate neighbour _backfill_cost (line 1421): calculate_cost(model, uncached_input_tokens=usage.uncached_input_tokens, output_tokens=usage.output_tokens, cache_creation_tokens=usage.cache_creation_input_tokens, cache_read_tokens=usage.cache_read_input_tokens). Extract one _price_from_buckets(usage, model) helper and let the two policies (backfill-if-absent vs. always-reprice) be one-liners over it. Two further inconsistencies with the neighbour it mirrors: (a) _backfill_cost is always consumed as a value (return ClaudeCodeAgent._backfill_cost(...) at lines 1385/1397/1409) whereas the new method's -> TokenUsage return is thrown away at its only call site — self._agent._reprice_for_litellm(usage, self.effective_model) (line 610) — so it silently relies on in-place mutation; make it -> None or consume the result. (b) _backfill_cost emits logger.warning("No pricing for model %r; ...") when the rate card misses (line 1445), while the new method sets total_cost_usd = None with no diagnostic, so an open-weight model missing from _PRICING loses its cost with nothing in the task log.
2. [Axis 1] start-litellm.sh's AWS_REGION export/banner is shadowed for every Bedrock entry in the shipped config (all three pin aws_region_name), so the region is stated in three places and the banner can misreport the region actually used (litellm/start-litellm.sh:47) — The script resolves, defaults and advertises a region — export AWS_REGION="${AWS_REGION:-$(read_env AWS_REGION)}" (line 47), export AWS_REGION="${AWS_REGION:-eu-north-1}" (line 48), echo "region : $AWS_REGION" (line 64) — and litellm/README.md:47 documents it as a prerequisite ("defaults to eu-north-1"). But all three Bedrock entries in litellm/litellm-config.yaml hardcode aws_region_name: eu-north-1 (lines 29, 34, 39), and per-model litellm_params win over the process env, so setting AWS_REGION=us-east-1 changes the printed banner and nothing else — a second source of truth that reads as load-bearing. Pick one: either template the region into the config (aws_region_name: os.environ/AWS_REGION) or delete the export/echo/README row and state that the region is pinned in the yaml.
3. [Axis 3] No mechanical guard ties litellm-config.yaml's six model_name ids to the _PRICING keys they must match, so a proxy-config rename silently nulls turn cost and disables the max_usd gate (litellm/litellm-config.yaml:26) — The six model_name: values operators put in LITELLM_MODEL (deepseek.v3.2 :26, zai.glm-5 :31, moonshotai.kimi-k2.5 :36, moonshotai/kimi-k3 :50, z-ai/glm-5.2 :58, deepseek/deepseek-v4-pro :66) must match _PRICING keys exactly; they do today, but nothing enforces it, and no runner reads this YAML at all. Failure scenario: add or rename a proxy model without touching pricing.py → _reprice_for_litellm takes the else None arm (claude_code_agent.py:1463-1470) and, unlike _build_token_usage, logs no warning, so total_cost_usd is None for every turn; orchestrator.py:798-806 then logs "max_usd budget configured but no turn reported cost; skipping cost check" and the cost gate never fires. The repo already has precedent for this shape of guard (evalboard/lib/__tests__/pricing-parity.test.ts for the TS mirror, the CONTAINER_ENTRYPOINT drift-guard test in isolation/docker_runner.py:59). Add a pytest that parses litellm/litellm-config.yaml and asserts every model_name is a key in coder_eval.pricing._PRICING. n/a
4. [Axis 6] On Linux hosts, --driver docker --backend litellm cannot connect: start-litellm.sh:88 hardcodes --host 127.0.0.1 (no override) while docker_runner.py:1131 points containers at host.docker.internal (bridge gateway), and the host-side preflight (run_command.py:89 probes the un-rewritten URL) reports green — so failures surface per task after N containers start (src/coder_eval/isolation/docker_runner.py:1131) — docker_runner.py:1120-1131 rewrites the loopback URL and publishes the alias — argv += ["--env", f"LITELLM_BASE_URL={rewritten}", "--add-host", f"{_DOCKER_HOST_ALIAS}:host-gateway"] — with the comment at line 1122 "publish that alias (--add-host) for Linux parity". But the launcher this PR ships binds loopback only: litellm/start-litellm.sh:88 exec uvx --from 'litellm[proxy]' litellm --config "$CONFIG" --host 127.0.0.1 --port "$PORT". On Linux host.docker.internal:host-gateway resolves to the bridge gateway address (e.g. 172.17.0.1); a process bound to 127.0.0.1 refuses connections to that address, so --driver docker --backend litellm can never work on Linux.
   Worse for diagnosis: the new host-side preflight (run_command.py:85-89, probing LITELLM_BASE_URL from the host) passes green because the host can reach 127.0.0.1:4000 — the failure then surfaces as N per-task connection errors after N containers have been built/started, which is exactly the late, misattributed failure the preflight was added to prevent.
   Fix: make the bind address overridable and non-loopback for the container path — e.g. --host "${LITELLM_HOST:-127.0.0.1}" in start-litellm.sh with litellm/README.md instructing LITELLM_HOST=0.0.0.0 (noting the exposure trade-off) when using --driver docker; and/or have the preflight probe the rewritten URL when the resolved driver is docker so the mismatch fails at startup rather than per task.
5. [Axis 7] New litellm backend ships with no docs/.env.example updates: USER_GUIDE and .env.example still say "direct or bedrock", the four LITELLM_ settings (LITELLM_SMALL_MODEL entirely) are undocumented, and the new top-level litellm/ dir is outside the documented repo tree and docs SSOT* (src/coder_eval/cli/run_command.py:259) — click_type=click.Choice(["direct", "bedrock", "litellm"], case_sensitive=False) (run_command.py:259) widens a documented flag, but no docs/ or .env.example change ships with it. Verified stale locations: docs/USER_GUIDE.md:52 (| --backend, -b | API backend: 'direct' or 'bedrock' …|), USER_GUIDE.md:142-143 ("supports two API routing modes"), USER_GUIDE.md:220 (API_BACKEND … 'direct' or 'bedrock') plus its Environment Variables table which lists BEDROCK_/CODEX_/GEMINI_* but none of the four new LITELLM_BASE_URL / LITELLM_AUTH_TOKEN / LITELLM_MODEL / LITELLM_SMALL_MODEL settings added at config.py:112-115 (three of which are hard-required by _validate_litellm_settings, config.py:190-197); .env.example:16 ("# API Backend (direct or bedrock — default: direct)") and .env.example:22-23 (the per-backend judge-transport list). Separately, docker_runner.py:1131 now emits --env LITELLM_BASE_URL=<rewritten>, which contradicts docs/DOCKER_ISOLATION.md:246 ("Credentials are forwarded via --env VAR (name-only, never embedded in argv)"), and that same doc line does not list the new LITELLM_* vars added to the default allowlist at models/sandbox.py:230-236. Failure scenario: a user reading the shipped docs cannot discover or configure the backend (litellm/README.md is not in mkdocs.yml nav, so it is invisible on the docs site), and a docker-driver reader is told secrets are never rendered in argv while a value now is. Fix: update USER_GUIDE.md's flag row, routing-modes section, and env table; add the LITELLM_* block to .env.example; amend DOCKER_ISOLATION.md:246 to list the new vars and to carve out the URL-value exception; and add litellm/README.md (or a docs/ page linking it) to mkdocs.yml nav + extra.docs_index and re-run make docs-indexes.
6. [Axis 8] --resume config fingerprint omits settings.litellm_model (bedrock_model is threaded), so changing LITELLM_MODEL across a resume mixes two models with no drift warning (src/coder_eval/orchestrator.py:1094) — orchestrator.py:1094-1099 records only self.result.environment_info["litellm_base_url_host"] = urlparse(self.route.base_url).hostname or "" and environment_info["litellm_model"] = self.route.model. Unlike Bedrock, where bedrock_model fully determines the weights, the litellm alias is resolved by an external, mutable, uncommitted mapping: litellm/litellm-config.yaml:26-73 maps model_name → litellm_params.model, and litellm/README.md:112-132 documents editing it as the normal workflow. Two runs whose records are byte-identical (api_routing=litellm, litellm_base_url_host=localhost, litellm_model=zai.glm-5) can therefore have run different weights — the port is dropped, so even two proxies on the same host are indistinguishable, and nothing captures the config's content or version. For the OpenRouter entries it is worse: extra_body.provider: {sort: price, allow_fallbacks: false} pins the cheapest upstream, which changes as OpenRouter's marketplace changes, and different upstreams serve different quantizations of the same open-weight model — yet nothing records provider_name. litellm/README.md:181-182 frames this as cost-only ("Token counts are exact and provider-independent; only the per-token rate is subject to this"), which understates it: the served weights, hence agent output and score, can differ run to run for the same recorded config. Second site, same theme: src/coder_eval/cli/run_command.py:672-674 calls compute_run_fingerprint(config, experiment.experiment_id, settings.api_backend.value, settings.bedrock_model) — bedrock_model is threaded in precisely because it is the route-level model source living outside BatchRunConfig (orchestration/batch.py:426-445), but the exactly-parallel settings.litellm_model was not added, so --resume into a run dir after changing LITELLM_MODEL produces no drift warning and silently mixes two models in one run.json. Fix: record the resolved effective model (not just the route fallback — note _format_routing was updated to prefer effective_model at orchestrator.py:120-134 while this recording block was not), the full base_url host:port, and a fingerprint of the proxy's resolved model list (the preflight at cli/run_command.py:71-96 already makes an HTTP call, so GET /v1/model/info is nearly free); and add settings.litellm_model to the compute_run_fingerprint call.

Nits

1. [Axis 1] Pre-rename "custom" vocabulary left throughout the new route code, tests and docstrings (including one naming a backend that does not exist) (tests/test_litellm_route.py:35) — tests/test_litellm_route.py:34-35 reads class TestResolveRouteCustom: / """resolve_route() builds a LiteLLMRoute for the CUSTOM backend.""" — there is no CUSTOM member of ApiBackend (only DIRECT/BEDROCK/LITELLM, enums.py:68-70), so a reader greps for a backend that was renamed away. Same leftovers at test lines 89, 98-99, 120, 131, 134, 153, 160, 168, 177 (test_custom_route_*, "missing custom settings"), plus src: _validate_litellm_settings's docstring "Validate that required custom Anthropic-endpoint settings are present" / "If required custom settings are missing" (config.py:184, 187), resolve_route's "the Bedrock/custom credential checks" (routing.py:139) and case LiteLLMRoute() as cr: (claude_code_agent.py:773, cr = custom route). Rename to litellm/lr for one consistent vocabulary.
2. [Axis 1] Unnecessary function-local import of ApiBackend where the module already imports from ..models at top level (src/coder_eval/cli/run_command.py:81) — _litellm_preflight_error opens with from ..models import ApiBackend (line 81) although the module already has from ..models import PreservationMode, ResolvedTask, RunSummary, TaskResult at line 19 — there is no import cycle to dodge, so the deferred import is noise that suggests one exists. Add ApiBackend to the line-19 import and drop the local one (the pre-existing twin at line 368 inside the --backend block can go the same way).
3. [Axis 1] New infra comments document an "autostart"/"sidecar" proxy mode that the code does not implement (litellm/litellm-config.yaml:13) — litellm-config.yaml:10-14 lists "Two ways to run the proxy" including "Autostart: coder_eval spawns the proxy against this same file, or an always-on docker sidecar mounts it", and run_command.py:75-76 frames the preflight as covering "the manual proxy / always-on-sidecar path". Neither exists: nothing in src/ spawns or supervises a proxy, and litellm/README.md:32-34 states the opposite ("coder_eval does not own the proxy lifecycle: it expects an already-running proxy"). Delete the speculative mode from both comments so the only documented contract is the one the code implements.
4. [Axis 1] README's "find the OpenRouter slug" snippet filters on a leftover 'SEARCH' substring and returns nothing (litellm/README.md:137) — Under "Find the exact OpenRouter slug + rates" the snippet ends ... for m in json.load(sys.stdin)['data'] if 'SEARCH' in m['id'].lower()] (line 137) — a copy-paste leftover from a specific lookup: since m['id'] is lower-cased, the literal 'SEARCH' can never match, so a reader following the doc gets []. Replace with a parameterised filter (e.g. if (q := 'deepseek') in m['id'].lower()) or drop the filter. Same file, line 49: "It can be customly designed." should be "it can be any value you choose".
5. [Axis 1] _resolve_effective_model's LiteLLM branch duplicates the Bedrock branch verbatim minus one transform (src/coder_eval/agents/claude_code_agent.py:826) — Lines 826-831 (if isinstance(self.route, LiteLLMRoute): effective = config_model or route_model; if effective: env["ANTHROPIC_MODEL"] = effective; return effective) are byte-identical to the tail of the Bedrock branch at 819-825, whose only extra step is the to_bedrock_inference_profile call. Collapse to one path: apply the profile qualification only if isinstance(self.route, BedrockRoute), then effective = config_model or route_model, and write env["ANTHROPIC_MODEL"] for both route types — removing the third copy of the same four lines from a hot module.
6. [Axis 1] ApiBackend values are re-enumerated at unguarded seams (CLI --backend choices, credential-validation if-chain) with no exhaustiveness check (src/coder_eval/cli/run_command.py:259) — Line 259 now reads click_type=click.Choice(["direct", "bedrock", "litellm"], case_sensitive=False), a hand-maintained mirror of ApiBackend (enums.py:68-70). tests/test_config_precedence.py:287 asserts the enum has exactly three members but nothing asserts the CLI accepts them, so a fourth backend is rejected at the flag with "invalid choice" until someone remembers this line. Derive it: click.Choice([b.value for b in ApiBackend], case_sensitive=False) (the sibling ["tempdir", "docker"] / ["full", "minimal"] choices at lines 252/301 have the same shape and could follow).
7. [Axis 2] New test files defeat the shape checking they exist to provide (list[object] forcing type-ignores; unspecced MagicMock for ResolvedTask) (tests/test_route_seam_exhaustiveness.py:22) — pyright's include is coder_eval/ only, so nothing type-checks tests/ — which makes loose annotations in a guard test costly. tests/test_route_seam_exhaustiveness.py:22 declares _INSTANCES: list[object] = [ and that choice then forces two suppressions on the very calls the file exists to police: line 42 env, _model = ClaudeCodeAgent._build_sdk_env(r) # type: ignore[arg-type] and line 49 out = _format_routing(r) # type: ignore[arg-type]. Annotating _INSTANCES: list[ApiRoute] makes both calls type-correct, removes both ignores, and makes a fixture for a non-member a static error rather than only a runtime set-equality failure. Separately, tests/test_docker_litellm_env.py:57 (and :112) substitutes rt = MagicMock() for the ResolvedTask passed to DockerRunner(rt), setting only .task, .run_dir, .task_file; every other attribute _build_argv might read resolves to an auto-created Mock, so a future _build_argv read of a real ResolvedTask field would be silently satisfied instead of failing (rubric item 17). Use MagicMock(spec=ResolvedTask) there.
8. [Axis 4] LiteLLM endpoint + host alias are forwarded into task containers with no api_backend == LITELLM gate, so a --backend direct docker run still hands the sandboxed agent a working paid gateway (src/coder_eval/isolation/docker_runner.py:1128) — The forwarding condition keys only off the variable being present in os.environ, never off the run's actual backend:

1128: if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist and cfg.network != "none":
1131:     argv += ["--env", f"LITELLM_BASE_URL={rewritten}", "--add-host", f"{_DOCKER_HOST_ALIAS}:host-gateway"]

config.py's module-level export loop republishes every .env value into os.environ unconditionally, so a leftover LITELLM_BASE_URL=http://localhost:4000 means every docker task — including --backend direct and --backend bedrock runs — gets the rewritten, reachable proxy URL plus --add-host host.docker.internal:host-gateway. Paired with LITELLM_AUTH_TOKEN now sitting in the default passthrough allowlist (src/coder_eval/models/sandbox.py:236), the untrusted agent inside the container receives both a reachable endpoint and a valid key for the operator's Bedrock/OpenRouter spend on runs that have nothing to do with the litellm backend — broader than least privilege requires (CWE-668). Fix: gate the whole block (and ideally the LITELLM_* allowlist entries) on settings.api_backend == ApiBackend.LITELLM, mirroring how the litellm preflight in cli/run_command.py:83 already checks current_settings.api_backend != ApiBackend.LITELLM before acting.

Secondary, same block: LITELLM_BASE_URL is the only allowlisted variable whose value is rendered into the argv, which is logged verbatim at docker_runner.py:566 (logger.info("Running task '%s' in docker: %s", ..., " ".join(argv))) — breaking the invariant the surrounding comment states at line 1105 ("secrets stay out of the rendered argv list that we log"). The full URL is likewise printed to the console at cli/run_command.py:94. Harmless for a bare http://host:port, but a base URL carrying userinfo (http://user:pass@host:4000) would be logged in clear text; strip userinfo before rendering/printing (the orchestrator already does the right thing at orchestrator.py:1097, recording only urlparse(...).hostname). CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:L/I:N/A:N
9. [Axis 4] # nosec B310 on the preflight urlopen without constraining the URL scheme (src/coder_eval/cli/run_command.py:89) — The suppression is documented but the stated constraint is not actually enforced in code:

87:        # B310: url is built from the operator-configured LITELLM_BASE_URL (not
88:        # untrusted input); this only probes reachability of that proxy endpoint.
89:        urllib.request.urlopen(url, timeout=5).close()  # nosec B310

url is f"{current_settings.litellm_base_url.rstrip('/')}/health/liveliness" (line 85) and litellm_base_url is a free-form str | None on Settings (config.py:112) with no scheme validation anywhere — _validate_litellm_settings (config.py:183-201) only checks truthiness. B310 exists precisely to catch the file:// / ftp:// schemes that urlopen also accepts, so as written the suppression asserts a property the code never checks. Reachability is the only thing consumed here (the response is immediately .close()d) and the value is operator-owned .env config, so real impact is confined to the error string at line 94 revealing whether a local path exists — hence Low, not an audit-scope escalation.

Fix: make the comment true with a two-line guard before the call, e.g. if urlsplit(url).scheme not in {"http", "https"}: return f"LITELLM_BASE_URL must be http(s), got {...}", then the # nosec B310 is airtight. Better still, add a Pydantic validator on Settings.litellm_base_url rejecting non-http(s) schemes so the constraint holds for the SDK env path (ANTHROPIC_BASE_URL at agents/claude_code_agent.py:779) and the docker rewrite (isolation/docker_runner.py:1129) too, not just the preflight. CVSS:3.1/AV:L/AC:H/PR:H/UI:N/S:U/C:L/I:N/A:N
10. [Axis 6] Preflight abort happens after the run dir is created and runs/latest is repointed, leaving an empty run dir and a clobbered latest (src/coder_eval/cli/run_command.py:496) — _run_all_tasks creates the run dir and repoints the symlink first — run_command.py:452 run_dir = prepare_run_directory(run_dir) (which does run_dir.mkdir(parents=True, exist_ok=True), run_helpers.py:63) and line 456 create_latest_symlink(settings.runs_dir, run_dir.name) — and only then runs the preflight at line 496-499 (raise typer.Exit(1)). A dead proxy therefore leaves an empty runs/<run_id>/ behind on every attempt and leaves runs/latest pointing at that empty dir, so a follow-up coder-eval report/CI step reading runs/latest sees an empty run instead of the last good one. Move the preflight above the prepare_run_directory call (it depends only on settings), or delete the freshly created dir / restore the symlink before exiting.
11. [Axis 6] New LITELLM arm of resolve_route guards required credentials with assert, on a path that is not always preceded by validate_api_keys (src/coder_eval/models/routing.py:169) — routing.py:169-170 uses assert settings.litellm_base_url is not None, "LiteLLM backend requires litellm_base_url" / assert settings.litellm_auth_token is not None, ..., justified by the docstring's "Called after validate_api_keys() has verified credentials". That precondition does not hold on the evaluate-only path: orchestrator.py:899 calls resolve_route(settings) inside the if self.sandbox is not None: branch and returns at line 905 — settings.validate_api_keys(...) is only reached at line 911, after that early return. So an API_BACKEND=litellm re-grade with no LITELLM_BASE_URL raises a bare AssertionError instead of the friendly _validate_litellm_settings message (config.py:198-202), and under python -O the asserts vanish and a LiteLLMRoute(base_url=None, auth_token=None) is constructed and pushed into _build_sdk_env. The repo already codifies the opposite convention for exactly this reason — criteria/agent_judge.py:125 ("Explicit raise (not assert) so python -O cannot strip this guard") and evaluation/sub_agent.py:83 ("Not assert because the check must survive python -O"). Raise ValueError (reusing _validate_litellm_settings) in the new arm, or call _validate_litellm_settings() on the evaluate-only path.
12. [Axis 6] start-litellm.sh kills whatever holds the port without verifying it is a LiteLLM proxy, then rebinds after a fixed 1s sleep (litellm/start-litellm.sh:73) — Lines 69-75 do existing=$(lsof -tiTCP:"$PORT" -sTCP:LISTEN 2>/dev/null || true) then kill $existing 2>/dev/null || true followed by sleep 1. Two resilience gaps: (1) the pid list is not checked to be a litellm process, so LITELLM_PORT colliding with any other local service (or the default 4000 already used by another tool) silently kills the developer's unrelated listener; (2) the fixed sleep 1 doesn't verify the port was actually released, so a slow shutdown makes the following exec ... --port "$PORT" fail on bind. Suggest confirming the command name for each pid (e.g. ps -o command= -p "$pid" | grep -q litellm) before killing, and replacing sleep 1 with a bounded poll on lsof -tiTCP:"$PORT" becoming empty (exiting with a clear error if it doesn't).
13. [Axis 7] Preflight error tells the user to "unset LITELLM_BASE_URL", which makes the run fail with a different hard error (src/coder_eval/cli/run_command.py:95) — run_command.py:94-95 emits "LiteLLM proxy not reachable at … Start it (e.g. litellm/start-litellm.sh) or unset LITELLM_BASE_URL.", and litellm/README.md:190 repeats it ("Proxy not running — start it, or unset LITELLM_BASE_URL"). Failure scenario: with API_BACKEND=litellm the user follows the advice, unsets LITELLM_BASE_URL, and the preflight now short-circuits to None (run_command.py:83) only for the run to abort per-task in _validate_litellm_settings with "LiteLLM-endpoint routing is enabled but missing required settings: LITELLM_BASE_URL. Please set them in your .env file." (config.py:190-202) — the opposite instruction. Reword to "start the proxy, or switch backends (--backend direct / --backend bedrock)" in both places.

What's Missing

Parallel paths:

• 🟠 The route union grew a third member (models/routing.py:121) but the judge dispatch seams were not extended: criteria/llm_judge.py:182-209 still matches only BedrockRoute/DirectRoute and falls into the case _: arm (whose comment still claims it is unreachable), and the upstream short-circuit at llm_judge.py:99 only special-cases DirectRoute. Also un-updated in the same family: the judge-transport mapping table in docs/TASK_DEFINITION_GUIDE.md:841-845, which still lists only direct/bedrock rows, and criteria/agent_judge.py's error text at line 130 ("must construct one (DirectRoute / BedrockRoute)"). (trigger: src/coder_eval/models/routing.py) (restates: Axis 2: LiteLLMRoute unhandled at the judge/route dispatch seams)
• 🟡 simulation/user_simulator.py:193-196 deliberately passes model=None so the route supplies the simulator's model, with a comment reasoning only about BedrockRoute. The new _resolve_effective_model LiteLLM branch (claude_code_agent.py:826-831) makes that fall through to route.model, so under --backend litellm the simulated user silently runs on the same open-weight model as the agent under test (e.g. zai.glm-5) — dialog-mode runs are no longer comparable across backends, and the simulator's stop-token/termination contract was never validated on these models. Neither the comment, the simulator, nor litellm/README.md mentions dialog mode. (trigger: src/coder_eval/agents/claude_code_agent.py)
• 🟡 The new fail-fast preflight was added only to the host path (cli/run_command.py:496, probing the un-rewritten URL); the docker path got the URL rewrite (isolation/docker_runner.py:1120-1131) but no equivalent guard. Two per-task-late failures result: on Linux the bridge gateway can't reach the 127.0.0.1-bound proxy, and docker.network: "none" + api_backend=litellm skips LITELLM_BASE_URL entirely so the in-container Settings aborts with missing required settings: LITELLM_BASE_URL — an error that misattributes an operator network-config conflict to a missing env var. (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 6: --driver docker --backend litellm cannot connect on Linux hosts)
• 🔵 Only ClaudeCodeAgent implements the new route; codex_agent.py:648, antigravity_agent.py:199 and noop_agent.py:47 accept route and ignore it. Because Settings.validate_api_keys gates on the backend only (config.py:218-221, agent-type-agnostic), --backend litellm --type codex demands the three LITELLM_* settings, records api_routing=litellm + litellm_model in environment_info, and then calls Codex's own endpoint — a run record that misdescribes the route. Same pre-existing shape as bedrock, but the new backend exists purely to redirect model traffic, so the mismatch is more misleading. (trigger: src/coder_eval/models/routing.py)

Tests:

• 🟠 No test drives _ClaudeTurnState.finalize()/build_turn_record under a LiteLLMRoute, so claude_code_agent.py:609-610 — the only line that makes repricing affect a real run — is uncovered and can be deleted with the suite still green; TestRepriceForLitellm only calls the static helper on a hand-built TokenUsage. Same gap for the two remaining _validate_litellm_settings branches (auth-token, model) and the preflight-abort branch in _run_all_tasks. (trigger: src/coder_eval/agents/claude_code_agent.py) (restates: Axis 3: New LiteLLM code paths lack test coverage (repricing wiring, validation branches, preflight abort))
• 🟡 TestBuildSdkEnvCustom (tests/test_litellm_route.py:134-149) asserts the ANTHROPIC_* vars and the Bedrock-cred blanking but never CLAUDE_CODE_ATTRIBUTION_HEADER == "0" — the workaround for the HTTP 400 Bedrock returns on metadata.user_id, i.e. the single line whose removal breaks every LiteLLM→Bedrock run. _rewrite_loopback_for_container's ::1 member of _LOOPBACK_HOSTS is likewise untested (tests/test_docker_litellm_env.py:30-42 covers only localhost/127.0.0.1/no-port/non-loopback). (trigger: tests/test_litellm_route.py) (restates: Axis 3: New LiteLLM code paths lack test coverage (repricing wiring, validation branches, preflight abort))
• 🟡 The new 88-line launcher ships with zero mechanical coverage: no shellcheck target in make verify/make lint and no shellcheck step in .github/workflows/pr-checks.yml (grep finds none anywhere), and no test over its env resolution, lsof-based port reclaim, or the hardcoded --host 127.0.0.1 bind that the docker path depends on. Its sibling infra file docker/coder_eval_entrypoint.sh at least has the CONTAINER_ENTRYPOINT drift-guard test (isolation/docker_runner.py:57-59); the litellm launcher has no equivalent, so the script/docker_runner/README contract can drift silently. (trigger: litellm/start-litellm.sh)
• 🟡 Nothing in src/ or tests/ parses litellm/litellm-config.yaml (grep for it hits only the README, the script and the file itself), so the invariant its own README states at lines 103-104 — every model_name must equal a pricing.py key — is documented but unenforced; a proxy-side rename nulls turn cost and silently skips the max_usd gate (orchestrator.py:798-806). (trigger: litellm/litellm-config.yaml) _(restates: Axis 3: No mechanical guard ties litellm-config.yaml model_name ids to PRICING keys)
• 🟡 The new reconciliation-row pricing has unit coverage for parseMessages (parseMessages.test.ts:772-830) but nothing asserts the property it was added for: that the per-message Cost column now sums to the task's authoritative total_cost_usd. There is a token analogue (selectTokenTotals, runs.ts:1957-1972, with the stream-sums-exactly invariant) but no cost analogue and no test, so the column can drift from the backend figure undetected. The USD-mode bucket cells on that row (_sections.tsx:1070, tokenBucketUsd(m.model, …)) also flip from "—" to real numbers now that m.model is set — an untested, undocumented side effect of the same two-line change. (trigger: evalboard/lib/runs.ts)

Downstream consumers:

• 🟡 runs.ts:1731-1739 prices the reconciliation row for every run, not just LiteLLM ones. On Claude runs that residual is by construction the tokens no message reported — including sub-agent input/cache that partially bubbles up — yet it is now priced entirely at turn.model_used (the turn's last generation model). A parent-Sonnet turn with Haiku sub-agents therefore over-states that row, and vice-versa; the change silently re-renders the Cost column of every historical run in the dashboard. No consumer was updated to mark the figure as an approximation, and the code comment's "the authoritative task total_cost_usd is unaffected" doesn't cover the visible per-row sum that operators actually read. (trigger: evalboard/lib/runs.ts)
• 🟠 Six ids were added to src/coder_eval/pricing.py:95-105 but only the three OpenRouter ones to evalboard/lib/pricing.ts:58-60, and resolvePricing (pricing.ts:86-93) was not given the bedrock/converse//bedrock//converse/ stripping that _normalize_model gained at pricing.py:162-167 — nor does it have the pre-existing eu.anthropic./region-prefix handling. So the newly-priced reconciliation row and the whole Cost column render "—" for exactly the Bedrock open-weight path the shipped proxy config serves, and the loosened subset-only parity test can no longer catch it. (trigger: evalboard/lib/pricing.ts) (restates: Axis 8: pricing.py↔pricing.ts parity guard relaxed while the 3 Bedrock open-weight ids stay unmirrored)
• 🟡 compute_run_fingerprint threads settings.bedrock_model precisely because it is a route-level model source living outside BatchRunConfig (orchestration/batch.py:426-445); the exactly-parallel settings.litellm_model was not added at the call site (cli/run_command.py:672-673), so --resume after changing LITELLM_MODEL mixes two models into one run.json with no drift warning. (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 8: --resume config fingerprint omits settings.litellm_model)

Display & mapping dicts:

• 🔵 ROUTE_NAMES gained its member and is now guarded by tests/test_route_seam_exhaustiveness.py, but the ApiBackend value has to be hand-added at three still-unguarded enumerations: the CLI click.Choice(["direct", "bedrock", "litellm"]) (run_command.py:259), the validate_api_keys if-chain (config.py:218-221), and the docs mappings (docs/USER_GUIDE.md:52 and :220, docs/agents/CLAUDE_CODE.md:37, docs/TASK_DEFINITION_GUIDE.md:841). A fourth backend is rejected at the flag with "invalid choice" until someone remembers line 259; derive it from the enum instead. (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 1: ApiBackend values re-enumerated at unguarded seams)
• 🔵 Prefix normalization was added for pricing only (pricing.py:162-167), so report surfaces still render the raw SDK-reported id. Since the SDK reports model_used as e.g. converse/zai.glm-5 (per the PR's own test comment at tests/test_litellm_route.py:229), the Models line in reports.py:267, the per-task Model cell (reports.py:376, reports_html.py:333, orchestrator.py:257) and the variant grouping in reports_experiment.py:115 show a differently-spelled id than the configured LITELLM_MODEL — and the same model can split into two labels across runs depending on whether the id arrived prefixed. (trigger: src/coder_eval/pricing.py)

Daily/nightly:

• 🟡 The PR touches three shared production paths without stating the blast radius: _normalize_model now strips routing prefixes for every route (pricing.py:159-167), the evalboard reconciliation row is now priced on every run, and four LITELLM_* vars were added to the default docker env-passthrough allowlist (models/sandbox.py:235-238) so every docker task — including nightly --backend bedrock runs — now forwards them plus --add-host host.docker.internal:host-gateway whenever the vars are set. Meanwhile nothing exercises the new backend in CI (every smoke step in .github/workflows/pr-checks.yml pins API_BACKEND: bedrock), and the run-record contract consumed by the evalboard and the external eval-runner gains a third api_routing value plus litellm_base_url_host/litellm_model keys. The PR should state explicitly that the nightly stays on bedrock and that the litellm path ships CI-unverified. (trigger: src/coder_eval/models/sandbox.py)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE032 — exhaustive match in src/: forbid a wildcard case _: arm in any match statement under src/coder_eval/ unless its body's sole statement is assert_never(<subject>) (or a raise). New rule file tests/lint/rules/ce032_exhaustive_match.py (a plain BaseRule AST visitor on ast.Match), added to ALL_RULES in tests/lint/runner.py and given a TestCE032… class in tests/test_custom_lint.py. Measured cost on main: src/ contains exactly 4 match statements (criteria/llm_judge.py:182, agents/claude_code_agent.py:747, models/routing.py:124, models/mutations.py:82) and exactly 1 wildcard arm — criteria/llm_judge.py:206, i.e. the confirmed high-severity defect itself. Zero-noise after that one fix. Verified mechanically that the replacement pattern bites: a case _: assert_never(v) over a 3-member union with only 2 arms handled produces a hard pyright reportArgumentType error ('Argument of type "C" cannot be assigned to parameter "arg" of type "Never"'), so the seam becomes statically self-guarding for the next union member too. Prevents: The A2/A1/A3/A5/A6/A7/A8 cross-axis high: LiteLLMRoute falling through criteria/llm_judge.py:206's 'defensive only' case _: and silently scoring every llm_judge/agent_judge criterion 0.0 under --backend litellm. models/routing.py's own match settings.api_backend was updated precisely because it has no wildcard (pyright would have flagged the implicit None return); the wildcard is what disabled static protection at the one seam that broke.
• [pyright] Flip reportMatchNotExhaustive = "error" in [tool.pyright] (pyproject.toml, alongside the existing reportUnboundVariable/reportMissingTypeArgument block). Verified empirically: under the current standard config this check is OFF (a match over A | B | C handling only A/B yields zero diagnostics), and enabling it produces zero new diagnostics across all 118 analyzed files on main (same 3 pre-existing reportMissingImports errors in agents/codex_agent.py), so it is free to turn on today. Must be paired with CE032 — pyright suppresses this check whenever a wildcard case _: is present, which is exactly why the judge seam was invisible. Prevents: Same high-severity judge-dispatch finding, plus every future ApiBackend / ApiRoute / FinalStatus / criterion-discriminator widening that reaches a match seam (the Technique-3 exhaustiveness class generally).
• [ce-lint] CE033 — route/backend consumer coverage (whole-tree rule, wired like CE031 as a @pytest.mark.lint class rather than a BaseRule, since it reasons over the whole src/ tree): derive the member set of the ApiRoute union and of ApiBackend, then assert every member name is referenced at least once in each file of a declared consumer list — criteria/llm_judge.py, agents/claude_code_agent.py, orchestrator.py, config.py (validate_api_keys / _validate_*_settings), models/routing.py (ROUTE_NAMES), isolation/docker_runner.py, cli/run_command.py. This is additive to CE032/pyright because the two seams that actually broke are not match statements: the early guard if route is None or (isinstance(route, DirectRoute) and route.judge_transport is None): (criteria/llm_judge.py:99) and the hand-maintained click.Choice(["direct", "bedrock", …]). It also generalizes the PR's own tests/test_route_seam_exhaustiveness.py, whose docstring claims 'every route-matching seam' while enumerating only 3 (ROUTE_NAMES, _build_sdk_env, _format_routing) — the declared list moves from a test's local literal into a lint-enforced contract. Prevents: The judge-seam high (unhandled route in an isinstance chain, not a match); the --backend click.Choice re-enumeration finding; and the credential-validation if-chain finding — i.e. the whole 'adding a backend requires editing N open, unchecked enumerations' theme.
• [ce-lint] Extend CE030 (tests/lint/doc_schema_parity.py::DOCUMENTED_MODELS) to cover coder_eval.config.Settings, paired with docs/USER_GUIDE.md and .env.example (a field must be mentioned in both, or carry an EXEMPT entry naming why it is not operator-authored). This is the missing inverse of CE027, which only checks docs→Settings (a documented var must have a consumer) and therefore cannot see a new setting nobody documented. Measured cost on main: of 15 Settings fields, 14 already appear in both surfaces; only RUNS_DIR would need a doc line or one EXEMPT entry — so the rule lands green after a single-line change. Prevents: The A7-medium doc-drift finding: LITELLM_BASE_URL / LITELLM_AUTH_TOKEN / LITELLM_MODEL / LITELLM_SMALL_MODEL added at config.py:112-115 (three hard-required by _validate_litellm_settings) with zero grep hits in docs/, .env.example, mkdocs.yml or README.md, while USER_GUIDE.md:52/142/220 and .env.example:16 still state the two-value 'direct or bedrock' universe. Consider also extending CE028 so an operator-facing README shipped outside docs/ (litellm/README.md) must be linked from a nav'd page, since it is currently invisible on the docs site.
• [ce-lint] CE034 — no assert on Settings-derived state in src/: flag an ast.Assert whose test reads an attribute of a value named settings / annotated Settings (extendable to any function parameter typed Settings), directing the author to raise ValueError (or to reuse _validate_*_settings). File tests/lint/rules/ce034_no_assert_on_settings.py + ALL_RULES. This codifies a convention the repo already states in prose twice — criteria/agent_judge.py:125 ('Explicit raise (not assert) so python -O cannot strip this guard') and evaluation/sub_agent.py:83 ('Not assert because the check must survive python -O'). Measured cost: 2 offenders on main (models/routing.py:126-127, the Bedrock arm — the same latent bug, since the evaluate-only path at orchestrator.py:899 reaches resolve_route before validate_api_keys), plus the 2 the PR adds. Prevents: The A6-low finding on the new LITELLM arm (models/routing.py:169-170): a bare AssertionError instead of the friendly _validate_litellm_settings message on the re-grade path, and under python -O a LiteLLMRoute(base_url=None, auth_token=None) constructed and pushed into _build_sdk_env.
• [ce-lint] CE035 — shipped proxy config ↔ pricing parity (YAML-surface rule, wired as a @pytest.mark.lint class like CE029/CE031): yaml.safe_load('litellm/litellm-config.yaml') and assert (a) every model_list[*].model_name is a key of coder_eval.pricing._PRICING, (b) every litellm_params.model normalizes via _normalize_model to a priced key, and (c) no entry hardcodes aws_region_name: — it must read os.environ/AWS_REGION so the launcher's exported region is the single source of truth. Precedent for this exact shape already exists (evalboard/lib/__tests__/pricing-parity.test.ts, and the CONTAINER_ENTRYPOINT drift guard referenced in isolation/docker_runner.py). Prevents: (a)+(b) the A3-medium: renaming/adding a proxy model without touching pricing.py makes _reprice_for_litellm return None with no warning, so total_cost_usd is null for every turn and orchestrator.py:798-806 logs 'max_usd budget configured but no turn reported cost; skipping cost check' — the cost gate silently stops existing. (c) the A1-medium AWS_REGION triple-source finding (start-litellm.sh:47/48/64 + README.md:47 + the yaml header at :18 all claim a knob that the three aws_region_name: eu-north-1 pins shadow, so the preflight banner can misreport the region actually used).
• [ce-lint] CE036 — validated URLs at the settings boundary, one probe helper: two mechanical halves in one rule. (1) Any Settings field whose name ends in _url/_base_url must be annotated AnyHttpUrl | None or carry a @field_validator (today litellm_base_url: str | None = None, config.py:112, has neither and is guarded only by a truthiness check at config.py:190). (2) Ban urllib.request.urlopen / urllib.request.urlretrieve in src/coder_eval/, requiring one shared helper that validates the scheme is http/https, normalizes ValueError/URLError/OSError into a single error type, and scrubs userinfo before the URL reaches a log or console line. Measured: zero urlopen call sites in src/ on main, so this is a zero-noise, forward-only ban. Prevents: The A2-high ValueError escape (reproduced by execution: LITELLM_BASE_URL=my-proxy.internal → ValueError: unknown url type propagating past _litellm_preflight_error's (URLError, OSError) handlers, through the un-try'd await asyncio.to_thread(...) at run_command.py:496, out of the bare asyncio.run at :386 as a raw traceback instead of typer.Exit(1)); the A4-low # nosec B310 whose stated scheme constraint is never enforced; the empty litellm_base_url_host recorded at orchestrator.py:1097 on the library path; and the userinfo-in-logged-argv/console concern at docker_runner.py:566 / run_command.py:94.
• [ce-lint] CE037 — no redundant function-local import: flag a function-local from X import … / import X when the same module is already imported unconditionally at module level in that file (imports under if TYPE_CHECKING: deliberately excluded, so the deferred-runtime-import pattern still passes). Narrower and quieter than ruff's PLC0415, which would flag every deliberate cycle-avoiding lazy import this codebase mandates, and it does not collide with CE017 (which requires lazy imports in models/ precisely where no top-level import exists). Measured cost on main: 14 sites, every one trivially fixable by adding the name to the existing top-level import — e.g. models/sandbox.py:84 re-imports RepoSource from coder_eval.models.templates, already imported at line 12 for TemplateSource; also reports.py:603/653, reports_experiment.py:68/69/539, orchestrator.py:699, cli/__init__.py:64, cli/run_command.py:470, isolation/docker_runner.py:1177, simulation/user_simulator.py:204, evaluation/judge_context.py:479. Prevents: The A1-low finding at cli/run_command.py:81 (from ..models import ApiBackend inside _litellm_preflight_error while line 19 already imports from ..models), plus its pre-existing twin at :368 — deferred imports that falsely signal an import cycle to the next reader.
• [ce-lint] CE038 — dead return annotation on a private helper (whole-tree rule): flag a _-prefixed function/method defined in src/ whose return annotation is not None/NoReturn/Never when every call site across src/ + tests/ is a bare expression statement (value discarded). Measured on main: 316 annotated private helpers scanned, 2 flagged, and both are annotated NoReturn (agent.py:116 _finalize_and_raise_timeout, agent.py:132 _finalize_and_raise_crash) — so with NoReturn/Never excluded the rule is zero-noise today. Name-based call-site matching means a # noqa: CE038 escape is needed for reflection/override cases; keep the rule scoped to private names for that reason. Prevents: The A1/A2/A5/A6/A8-merged medium on agents/claude_code_agent.py:1450: _reprice_for_litellm(...) -> TokenUsage whose value is discarded at all five call sites (production :610 plus tests/test_litellm_route.py:238,243,249,254), so its declared contract is unconsumed and the method silently relies on in-place mutation — unlike its neighbour _backfill_cost, whose value is consumed at :1385/:1397/:1409.
• [ce-lint] CE039 — CLI choice sets must derive from their model SSOT: flag a click.Choice([<string literals>]) in src/coder_eval/cli/ when that literal set equals, or is a strict subset of, either a StrEnum value set in models/enums.py or the args of a Literal[…] model field — requiring click.Choice([b.value for b in ApiBackend], …) instead. Subset detection (not just equality) is what catches the actual drift shape: an enum member that exists but is missing from the flag. Genuinely CLI-only sets (["full", "minimal"] at run_command.py:221) match no SSOT and stay untouched; ["tempdir", "docker"] (:270) mirrors SandboxConfig.driver: Literal["tempdir", "docker"] (models/sandbox.py:310) and ["direct", "bedrock", …] (:228/:259) mirrors ApiBackend. Prevents: The A1/A2/A7 finding: the hand-maintained --backend choice list, where tests/test_config_precedence.py:287 asserts the enum has N members but nothing asserts the CLI accepts them — so a new backend is rejected at the flag with 'invalid choice' until someone remembers that line.
• [bandit-codeql] Re-enable bandit (and pip-audit) in make verify — both lines are currently commented out (Makefile:55-56), so the security gate exists only in .github/workflows/pr-checks.yml:92/125 and a developer running the documented make verify never sees a new bandit hit or a new # nosec suppression before pushing. Restore uv run bandit -r src/ -ll to verify so the B310-class decision is surfaced locally at authoring time, and keep the CI invocation as the backstop. Prevents: The A4-low # nosec B310 finding — the suppression (and the fact that its stated scheme precondition is unenforced) was never surfaced by the local gate the contributor workflow prescribes. Pairs with CE036, which removes the need for the suppression entirely.

Harness improvements (not statically reachable):

• Backend-parametrized route-behavior matrix test, parametrized from list(ApiBackend) (so adding a member auto-creates failing cases rather than requiring someone to remember a new test): for each backend, drive one stubbed SDK turn end-to-end through _ClaudeTurnState.finalize() / EventCollector.build_turn_record() and assert (a) the persisted TurnRecord.token_usage.total_cost_usd equals the rate-table figure for a priced id and is None for an unpriced one, (b) the max_usd gate outcome for the None case, (c) _build_sdk_env exports, and (d) that a task carrying llm_judge reaches a real judge transport (or fails with the actionable llm_judge.py:99 error, never a silent score=0.0). Why not static: No static rule can tell that the wiring is unreached: claude_code_agent.py:609-610 (if isinstance(self._agent.route, LiteLLMRoute): self._agent._reprice_for_litellm(...)) is syntactically fine and locally correct. Proving the gap required executing the suite with the line mutated to if False: — 3487 passed, 13 skipped, i.e. total blindness. Detecting it needs a live event stream and an assembled TurnRecord. Prevents: A3-high (repricing wiring uncovered — line 610 in the coverage miss list at 95.65%); the judge-seam high (a 0.0 score under --backend litellm that no test would notice); and the uncovered _validate_litellm_settings branches / preflight abort branch grouped under the same A3 finding.
• Diff-coverage gate: add diff-cover coverage.xml --compare-branch=origin/main --fail-under=90 (or pytest-cov's per-file --cov-fail-under on changed files) to make verify and pr-checks.yml, downstream of the existing --cov-report=xml step. Why not static: Coverage is a runtime measurement, not a syntactic property — and the existing repo-wide --cov-fail-under=80 is structurally insensitive to a handful of new uncovered lines: agents/claude_code_agent.py sits at 95.65% with the new :610 wiring, both new _validate_litellm_settings branches, the preflight abort branch and the IPv6 loopback rewrite unexecuted. Prevents: The whole A3-high cluster (five uncovered new LiteLLM paths reported as one theme), and the recurring 'new public behavior shipped untested' block class generally.
• Startup-validation-before-side-effects test: assert that when _litellm_preflight_error returns non-None, _run_all_tasks creates no runs/<run_id>/ directory and leaves runs/latest pointing at the previous run. Cheapest fix is to move the preflight above prepare_run_directory (run_command.py:452) / create_latest_symlink (:456) — the check depends only on settings — and the test locks the ordering in. Why not static: The defect is an ordering/side-effect property of one code path, not a syntactic pattern: both statements are individually valid and a linter cannot know that prepare_run_directory mutates shared state that the later raise typer.Exit(1) at :496-499 does not roll back. Verifying it needs the CLI path executed and the filesystem inspected. Prevents: A6-low: a dead proxy leaves an empty run dir on every attempt and repoints runs/latest at it, so a follow-up coder-eval report / CI step reading runs/latest sees an empty run instead of the last good one.
• Container→host proxy reachability check: (a) make _litellm_preflight_error probe the rewritten, container-visible URL when the resolved driver is docker (it currently probes the host-side LITELLM_BASE_URL and returns green); (b) make the launcher's bind address overridable — --host "${LITELLM_HOST:-127.0.0.1}" in litellm/start-litellm.sh:88, documented in litellm/README.md with the exposure trade-off; (c) add one ubuntu CI smoke job that starts the proxy and runs a single --driver docker --backend litellm task. Why not static: The mismatch is a live-network property of Linux bridge networking: host.docker.internal:host-gateway resolves to the docker0 gateway (e.g. 172.17.0.1) while the shipped launcher binds 127.0.0.1, so connections are refused. No AST or grep rule can see that — and the existing tests/test_docker_litellm_env.py locks in only the half that works (argv contains the rewritten env + --add-host). Prevents: A6-medium: on Linux hosts --driver docker --backend litellm can never connect, yet the host-side preflight reports green, so the failure surfaces as N per-task connection errors after N containers have been built — precisely the late, misattributed failure the preflight was added to prevent (macOS/Windows Docker Desktop masks it).
• Cross-language pricing parity, both directions plus lookup behavior: in evalboard/lib/__tests__/pricing-parity.test.ts, restore the Python→TS key direction with an explicit, commented allowlist of deliberately-unmirrored ids (so a real omission still breaks the build under the new, intentional subset semantics), and add a behavior case asserting resolvePricing resolves the same inputs _normalize_model does — including prefixed ids such as converse/zai.glm-5 (tests/test_litellm_route.py:229 documents that shape as what the SDK reports). Why not static: It crosses the Python/TypeScript boundary and asserts lookup behavior, not text: the Python CEnnn rules only walk src/'s AST and cannot see into the jest suite or evalboard/lib/pricing.ts, and the defect is a missing normalization step in TS, not a missing key alone. Prevents: A8-high: the parity guard was relaxed to a one-way subset in the same PR that leaves the three shipped Bedrock open-weight ids (deepseek.v3.2, zai.glm-5, moonshotai.kimi-k2.5) unmirrored and resolvePricing without the prefix strip — so the per-message and reconciliation Cost columns render '—' for exactly the runs whose cost rendering the PR added.
• Fingerprint-completeness test, enumeration-driven: parametrize over the route-level model sources (api_backend, bedrock_model, litellm_model, …) and assert that changing each one produces a fingerprint_diff warning on --resume. Better still, remove the hand-threading entirely — derive the fingerprint's model input from the resolved ApiRoute instead of the current positional settings.bedrock_model argument at cli/run_command.py:672-674 / orchestration/batch.py:426-445, so a new backend's model source cannot be forgotten. Why not static: Detecting it requires running the resume flow and diffing two stamped run.json fingerprints — and the failure mode is silence by omission (fingerprint_diff only compares keys present in both stamps), which has no syntactic signature. Prevents: A8-medium: --resume into a run dir after changing LITELLM_MODEL via the env-default path emits no drift warning and mixes two models in one run.json, exactly the case bedrock_model was threaded in to prevent.
• Backend-gated container env forwarding, with a docker-argv matrix test: gate the LITELLM_BASE_URL / --add-host block (isolation/docker_runner.py:1128-1131) and ideally the LITELLM_* default-allowlist entries (models/sandbox.py:235-238) on settings.api_backend == ApiBackend.LITELLM, mirroring the preflight's own check at cli/run_command.py:83; then add a docker-argv test parametrized over ApiBackend asserting the LiteLLM env/alias appear only for the litellm backend, and that userinfo in the base URL is scrubbed before argv rendering and console print. Why not static: The condition to enforce is a runtime relationship between the resolved backend and what gets forwarded; a linter sees only if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist, which is syntactically unremarkable. The leak also depends on .env contents republished into os.environ, i.e. process state. Prevents: A4-low (CWE-668): a leftover LITELLM_BASE_URL means every docker task — including --backend direct/--backend bedrock — hands the sandboxed agent a reachable paid gateway plus (via the default passthrough allowlist) a valid LITELLM_AUTH_TOKEN, broader than least privilege requires; and the logged-argv/console rendering of the URL value.
• Shell-script gate for the newly shipped operator scripts: add shellcheck over litellm/*.sh (and the repo's other scripts) to make check/pr-checks.yml, and fix the two semantic gaps in litellm/start-litellm.sh:69-75 by hand — confirm each pid actually belongs to a litellm process before kill, and replace the fixed sleep 1 with a bounded poll on lsof -tiTCP:$PORT becoming empty (clear error if it never releases). Why not static: shellcheck is static but lives entirely outside the Python lint/pyright/CEnnn surface this repo gates on, so it needs its own tool + CI wiring; and the two substantive gaps — killing an unverified pid, and rebinding after an unverified fixed sleep — are semantic, so no linter of any kind will flag them. Prevents: A6-low: LITELLM_PORT colliding with an unrelated local listener silently kills the developer's service, and a slow shutdown makes the following exec … --port "$PORT" fail on bind. Related to the A1-medium AWS_REGION duplication in the same script (whose config half CE035 covers).

Top 5 Priority Actions

1. Add a case LiteLLMRoute(): arm (or an explicit reject in the early guard at llm_judge.py:99) to the judge dispatch at src/coder_eval/criteria/llm_judge.py:206, which today falls through its "unreachable" case _: and returns score=0.0 with "no usable API route" for every judged criterion on --backend litellm — a scored-to-zero verdict for identical agent output — and swap case _: for case None: + assert_never(route) so pyright catches the next union member.
2. Test and harden the repricing wiring at src/coder_eval/agents/claude_code_agent.py:610 (uncovered today — inverting the guard on line 609 leaves the whole suite green) and add the rate-card-miss warning _reprice_for_litellm (:1450) lacks versus its sibling _backfill_cost (:1446), because an unpriced model id silently yields total_cost_usd=None and orchestrator.py:803 then skips the max_usd check, turning a COST_BUDGET_EXCEEDED into a normal completion.
3. Make the proxy bind address overridable in litellm/start-litellm.sh:88 (hardcoded --host 127.0.0.1) and have the preflight probe the rewritten URL when the resolved driver is docker, so --driver docker --backend litellm on a Linux host fails at startup instead of passing a green preflight and then failing every task against host.docker.internal (src/coder_eval/isolation/docker_runner.py:1131).
4. Mirror the three shipped Bedrock ids (deepseek.v3.2, zai.glm-5, moonshotai.kimi-k2.5) into evalboard/lib/pricing.ts:58 and port _normalize_model's converse//bedrock/ prefix strip into resolvePricing (:87), then restore a Python→TS key direction — or an explicit commented allowlist — in the parity guard this PR loosened at evalboard/lib/tests/pricing-parity.test.ts:52, plus a pytest tying litellm/litellm-config.yaml's model_names to _PRICING keys.
5. Tighten litellm_base_url to a validated http(s) URL at src/coder_eval/config.py:112 and replace the two asserts in the LITELLM arm at src/coder_eval/models/routing.py:169 with a ValueError: one change removes the bare uncaught ValueError: unknown url type traceback from the un-try'd preflight (run_command.py:89), makes the # nosec B310 justification true, keeps a scheme-less value from recording an empty litellm_base_url_host, and survives python -O per the repo's own convention.
6. Close the operator-facing gaps the backend shipped without: update docs/USER_GUIDE.md:52/142/220 and .env.example:16 (both still say "direct or bedrock"), document the four LITELLM_* settings, add litellm/README.md to mkdocs.yml nav + extra.docs_index, reword the "unset LITELLM_BASE_URL" advice at run_command.py:95 that leads straight into a hard validation error, gate the container LiteLLM env forwarding on api_backend == LITELLM (docker_runner.py:1128), move the preflight above prepare_run_directory (run_command.py:452/496) so a dead proxy stops clobbering runs/latest, and add settings.litellm_model to the resume fingerprint (run_command.py:672).

---

Stats: 0 🔴 · 4 🟠 · 6 🟡 · 13 🔵 across 8 axes reviewed.

[akshaylive]

Full per-stage granularity would need per-chunk usage from LiteLLM

Let's add that?

[bai-uipath]

Right approach, one blocker. Driving open-weight models through the unmodified Claude Code SDK behind a translation shim — rather than writing a fourth agent adapter — is the right answer to the harness-variance problem that made the codex/antigravity comparisons untrustworthy: skill tooling, progressive disclosure, and activation detection stay constant, only the wire protocol changes. Credential handling is careful, and keeping the new settings out of the ANTHROPIC_* namespace so they can't leak process-wide is exactly right.

The blocker: the backend selection reaches two things it shouldn't — the judge and the simulated user — so a run completes and reports a wrong score rather than failing.

What I verified

End to end on the tempdir driver, both arms of the proxy, on a task exercising file writes, shell, and three checkers.

	DeepSeek V4 Pro (OpenRouter)	GLM 5 (Bedrock)
Result	success, 3/3 criteria, 17.3s	success, 3/3 criteria, 18.6s
uncached input / cache read	23,046 / 43,264	63,017 / 0
cost	$0.010606	$0.076312
cost vs. rate card	exact	exact

• ✅ Routing, model selection, cost repricing, and run-record capture all correct; no credential in the persisted artifacts. The small-model fallback is load-bearing — without it the SDK's background calls ask the gateway for a Claude model it doesn't serve.
• ✅ Open-weight models are enabled for our Bedrock account in eu-north-1.
• ✅ Python suite green apart from three pre-existing failures and three live tests that only break because my test environment selected the new backend.

The ordinary non-container path works. The problems are at the edges.

Before merge

Pin the judge and the simulated user to Bedrock — don't route them to the gateway. One backend selection governs the whole run, so picking an open-weight model also swaps out the thing doing the grading and the thing playing the human. Those are instruments, not subjects: if they move when the model under test moves, the arm can't be compared to the Claude baseline it exists to be compared against. One change at one seam — let them resolve their own route instead of inheriting the agent's. Their Bedrock credentials are still present during these runs, so nothing new needs plumbing.

• The judge doesn't recognise the new backend, so every judged criterion scores zero. It dispatches on backend type, has no arm for this one, and falls through to a failing score with an error claiming no route is configured — after paying to build the full judge context. 327 of 1120 skills-suite tasks, 281 of them uipath-troubleshoot: a suite run comes back looking like a capability gap when it's a dispatch gap. Teaching it the new backend instead would be wrong twice over — the judge model every task inherits is a Claude id the gateway doesn't serve, and a model grading its own work isn't a measurement.
• A quarter of the suite silently changes character, because the simulated user runs on the open-weight model too. Multi-turn tasks hand the simulator the same backend by design. Nothing errors, which is why this is worth fixing now: a weaker simulated user asks different follow-ups, so the task itself changes, and the result reads as model weakness. ~295 of 1120 tasks. Same seam as the judge, so nearly free alongside it.
• The new guard test misses the paths where this happens. It claims to cover every backend-dispatch site and enumerates three, neither grading nor simulation among them — confidence it hasn't earned, which is how this got this far. Fix: extend it, or use static exhaustiveness so the seams can't be enumerated incompletely.

Follow-ups

• High — the container path can't connect anywhere but Docker Desktop, and reports itself healthy while broken. The launcher binds loopback while containers are pointed at the docker host alias, which resolves to the bridge gateway; a loopback-bound process refuses that. It also breaks under colima, which our local container runs use, since the explicit host mapping overrides colima's own. Worse, the startup check probes the host-side address rather than the containers', so it passes on exactly the configuration that fails and the real failure surfaces per-task. Fix: make the bind address configurable and probe the address containers will actually use. Every container-driver run is affected, and the healthy-looking check means whoever hits it debugs their own setup first.
• Minor — the startup check could confirm the requested model is actually served, not just that something answers. Editing the gateway's model list without restarting is the documented foot-gun, and it currently surfaces as an opaque mid-run error. The check already makes a call, and the same response yields the gateway's real model list — useful for reproducibility, since the run record names an alias resolved by an external, mutable, uncommitted mapping.

Cost reporting — noted, not for this PR

The accounting is correct: both arms priced to the cent against the rate card. One thing to be aware of rather than act on now — OpenRouter's figure is a rate-card estimate sitting in the same column as a billed one. OpenRouter picks a provider per request and the served rate can differ substantially from the headline rate we price from, as the change itself notes. So the two arms' cost values aren't quite the same kind of number.

Not worth solving until something depends on OpenRouter spend. When it does, the options run roughly:

Option	Fidelity	Needs
Snapshot the pinned provider's real rate at startup, record which upstream served	Most of the error gone; still an estimate — provider can shift mid-run, quantization varies	One API call
Gateway writes per-request spend to a file the harness reads back	Close to actual	Plumbing, file lifecycle
Per-run virtual keys + the gateway's spend log	Authoritative; the design the gateway intends	Postgres

Leaving the call to you — flagging it so the estimate/billed distinction is on record before any number gets quoted downstream.

Lower priority

• Nothing can run this in CI yet. The harness owns no proxy lifecycle and expects one already running, so there's no path for the nightly or ADO today. Not urgent — the comparison work happens by hand first, and that path works. When it comes up, don't have the harness spawn the proxy: that means owning reaping across every abnormal exit, and a spawned grandchild has worse observability. Long-lived sidecar with its own restart policy and logs, harness stays a pure client.

Everything smaller I found is cosmetic or narrow enough to leave out.

[CarlesUIPath]

Full per-stage granularity would need per-chunk usage from LiteLLM

Let's add that?

Yes. This requires a bit more work on LiteLLM, and it would be nice to have. I would be inclined to push the current changes first since the model which has issues with real pricing vs. evalboard is only DeepSeek.