Judge criteria (llm_judge/agent_judge) serialize and pin threads during check_all

[akshaylive]

Summary

_check_impl on BaseCriterion is sync by design (src/coder_eval/criteria/base.py), and the orchestrator offloads the entire SuccessChecker.check_all() call into a single asyncio.to_thread (src/coder_eval/orchestrator.py:1349/1408/1501). This is the right call for the 12 file/command/trajectory criteria, which are pure CPU/file-bound and need no I/O.

The two judge criteria are where this gets more expensive than it needs to be:

• llm_judge (invoke_anthropic_judge in src/coder_eval/evaluation/judge_anthropic.py) uses the synchronous Anthropic client — a blocking network call.
• agent_judge (SubAgentRunner.run in src/coder_eval/evaluation/sub_agent.py:185-188) bridges into the async Claude Agent SDK via a fresh asyncio.run() per invocation.

Because check_all runs all of a task's criteria sequentially inside one asyncio.to_thread call:

1. If a task stacks more than one judge-type criterion (e.g. llm_judge + agent_judge, or two llm_judges), their LLM round trips run serially, even though they're independent and could run concurrently.
2. Each judge call pins a thread-pool thread for the full network wait. The default executor is sized min(32, cpu_count()+4), so a batch run with many concurrent tasks that each hold a thread through a judge call risks thread-pool saturation, throttling task-level concurrency in run_batch independent of any real compute limit.

Current state

No task in tasks//experiments/ currently stacks 2+ judge-type criteria on one task, so this isn't biting today. But coder_eval's core workload (dataset fan-out, nightly suites) is judge-latency-heavy by nature, and this is exactly the kind of cost that's cheap to design around now vs. retrofit once a suite pattern relies on it.

Possible direction

Keep the criterion interface sync for the 12 non-judge criteria (no reason to make trivial CPU-bound checkers async). For the judge pair specifically:

• Use async HTTP clients (AsyncAnthropic, async httpx for Bedrock) so a judge call yields the thread back to the loop instead of blocking it.
• Let check_all (or a thin wrapper around it) gather() a task's judge-type criteria concurrently, while the remaining sync criteria still run in one to_thread slot.

Filing to track before it becomes an actual bottleneck, not because it's an observed regression today.