This submission evaluates an agent as response = f(system_prompt, llm, tools | query), then extends the same measurement discipline to a three-agent chain. The important thing is not that a tiny smoke run proves a winner. It is that the code can vary the right components, collect traces, label what caused failures, and say exactly which claims are measured vs. designed.
send_message is explicit workspace state.
The diagram below is the actual Part 1 path through the repository:
synthetic fixture cases become a pydantic_evals dataset, the task function runs
EraAgent, trace attributes feed deterministic/report evaluators,
and AblationHarness aggregates the results into attribution.
Deterministic grading is reproducible and has no dependencies. The optional LLM judge augments — never replaces — deterministic grading. We report both scores rather than collapsing them.
Cost and latency are cross-cutting — they apply to all three tiles. We treat them as production constraints that gate whether any configuration ships, not as sub-metrics of "tool effectiveness."
success is strict
final-task pass/fail; quality is graded keyword overlap, so
it can move even when success stays at 1.0. per-agent deltas
trace which stage changed. propagation coefficients show how
an upstream change travels downstream. coupling compares
isolated reference-input performance with in-chain performance.
Gold injection is only defined when final-task residual error
exists; in the latest smoke it is undefined because final_score=1.0.
The brief says identifying what else matters is part of the work. We identify two tiers beyond prompt, model, and tools.
Tool descriptions (detailed / minimal / ambiguous), temperature (0.0 / 0.3 / 0.7 / 1.0), thinking effort (low / medium / high), tool search (on-demand vs always-loaded). Each is ablatable and attributable, just like the three named components.
Measured: safety compliance, trajectory sequence accuracy, trajectory consistency, environment state verification, reliability@k. Identified but not measured: calibration, RAG faithfulness, output contract compliance. Naming what you don't measure is a strength.
siliconflow:deepseek-ai/DeepSeek-V4-Flash.read_file, calculator, or execute_python has delta 0.0. The variance comes from full/minimal/empty toolset configs; g16 is a hard fail (datetime — the model can't infer the current time without the tool), while g14 and x14 stay robust under the probed removals.1.0; the separate keyword and LLMJudge assertions also pass. No semantic tie-break changed this case selection.| Harness | era_eval/ablation.py |
| Agent | era_eval/agent.py |
| Evaluators | era_eval/eval/evaluators.py + report_evaluators.py |
| Artifact | era_results/part1.md, part1.json, part1_evals_report.json |
Each chain step has its own prompt, model, and tool subset. Golden chain mode decomposes chainable synthetic fixture cases into expected A/B/C intermediate outputs so the framework can ask whether failures start upstream, downstream, or only at the final answer.
query_database from Agent A degrades quality on g33 (0.667), g34 (0.333), g35 (0.667), g36 (0.333), g37 (0.667) — confirming query_database is load-bearing for database cases.query_database with calculator degrades g33 to 0.667, g34 to 0.333, g36 to 0.333 — the agent tries to use the wrong tool and produces lower quality than having no tool at all.final_score=1.0 (all chains succeed) → attribution is undefined — no task-failure residual to attribute. Quality gaps (g25 0.889, g34 0.667) are captured in per-case analysis, not gold injection.| Chain | era_eval/eval/chain_eval.py |
| Golden mode | get_chainable_cases() + build_chain_from_golden_case() |
| Optimization | era_eval/optimization/design.py (method only) |
| Artifact | era_results/part2_golden.md, part2_golden.json |
Swap one agent's model, keep the other two unchanged. Measure Δ accuracy, Δ cost, Δ latency for the whole chain. The asymmetry across the three Δfinal values is each position's leverage.
Add, remove, or replace a tool in one agent. Measure the effect on that agent and on everything downstream. Identifies load-bearing vs ornamental tools in each chain position.
Coupling coefficient = 1 − (score_in_chain / score_isolated). 0 = robust, 1 = fully dependent. Returns null when insufficient data — not 0.0. Compounding factor = chain_acc / (acc_A × acc_B × acc_C): >1.0 means the chain recovers, ~1.0 means errors are independent, <1.0 means errors amplify.
Two methods, both labeled: first-failure heuristic (observational) with intervention-aware override, and gold injection (causal) — a lightweight version of structural causal model intervention.
| Network | web_search, fetch_url — public HTTP(S), SSRF/redirect checks. |
| Retrieval | retrieve_documents — deterministic built-in corpus. |
| Compute | calculator, execute_python — AST-restricted Python subprocess. |
| Data | query_database — SELECT-only SQLite authorizer. |
| State | read_file, write_file — workspace sandbox only. |
| Notify | send_message, get_current_datetime — webhook or local eval-echo log. |
| Proprietary | GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash. |
| Large open | DeepSeek V4 Flash, GPT-OSS 120B, Qwen3 Coder. |
| ≤120B open | Gemma 4 31B, GPT-OSS 20B, Nemotron 3 Nano. |
| MoE tradeoff | Nemotron 3 Super: 120B total / 12B active, included for active-parameter economics. |
| Source | MODEL_POOL in era_eval/agent.py carries provider, params, context, price, latency, and why. |
The search space is (10 × 2^10)^3 ≈ 1 trillion configurations. Brute force is not an option. The deliverable is the method, not a run — the brief is explicit. The design is on paper with full citations.
Structured seeds (all-frontier, all-small, role-optimised, tool extremes) + Latin Hypercube Sampling. Run each 3× for stochasticity.
Matérn 5/2 kernel. Hamming distance for categorical (model) dims, Euclidean for binary (tool) dims. Multi-task GP predicts all objectives jointly.
ParEGO traces the Pareto frontier. Production executor: BoTorch + Ax with qNEHVI (noisy multi-objective) or qLogNEHVI (numerical stability).
Non-dominated configs after ~100 evals. Re-evaluate top-5 with N=10. Phase 6 (stretch): GEPA prompt optimisation — Genetic-Pareto reflective mutation.
Full design, expected outcome, and trade-off characterisation in PART2_DESIGN.md. Citations to Mockus, Knowles, Daulton, Ament, Gardner, Sacks, McKay, and the GEPA paper in REFERENCES.md.
| Measured smoke | Part 1: 3 cases, 6 configs, 1 primary model. Part 2: 8 chainable fixture cases, 1 repeat, variant families. Generated under era_results/. |
| Dataset boundary | 55 hand-written synthetic fixture cases. No prompts name tools explicitly. 8 sequence-graded, 21 arg-checked, 15 state-checked. Good for smoke, regression, tool coverage, and evaluator instrumentation; not hidden, not human-validated, and not a frontier-grade benchmark. |
| Implemented | CLI flags, evaluator/report pipeline, tool sandboxing, model timeout handling, chain propagation, coupling, gold injection, BH-FDR helper. |
| Designed | 10-model/10-tool Bayesian optimization frontier, role-aware model assignment, GEPA prompt optimization stretch goal. |
| Known limits | Low sample size, keyword-overlap strictness, one-model Part 1 smoke, model specs require provider refresh before production use. |
TaskSuccess uses expected-keyword threshold 1.0 so success is reproducible. The limitation is acknowledged; LLMJudge and future embedding grading address paraphrases.
ANOVA-style attribution explains observed variance in the experimental design. Counterfactual tool probes and gold injection are the causal complements.
15 of 55 synthetic fixture cases are safety cases (PII leakage, prompt injection, harmful content, privilege escalation, overrefusal). SafetyCompliance measures Attack Success Rate across 5 attack types and 10 harm categories. A model with 95% task_success but 40% ASR is unfit for production, regardless of how well it does on other tasks.
Latency, tokens, and cost are captured because a configuration that is slightly more accurate but 10× slower or costlier may not be deployable.
Full assumption register: ASSUMPTIONS.md. Methodology: APPROACH.md and PART2_DESIGN.md.
--cases | Limit Part 1 cases for bounded smoke runs. |
--concurrency | Control parallel API-backed evaluation. |
--tier1 | Enable tool descriptions, temperature, thinking effort, and tool search axes. (run_eval.py) |
--golden-chain | Run Part 2 through chainable fixture cases instead of a single request. (run_eval.py) |
--output | Choose result output path. (run_eval.py) |
--swap-model | Override the Part 2 model used for swap experiments. (run_eval_fast.py) |
--tool-sensitive | Select tool-sensitive cases (file_ops, multi_tool, datetime, notification, db_state) for non-degenerate ANOVA. (run_eval_fast.py) |
--chain-cases | Number of chainable cases to run in Part 2 (default 2, max 8). Use 8 to break the success ceiling. (run_eval_fast.py) |
Eight items that would make the attribution percentages defensible, not directional. A frontier lab would require all eight before trusting the smoke results.
Report Cohen's kappa. The fixture suite is hand-written; without agreement, "correct" is a single opinion.
Cosine similarity catches correct paraphrases that keyword overlap scores 0.0. The optional LLM judge exists to catch this; embeddings would make it the default.
N=3 is the minimum for meaningful CIs. N=30 on the Pareto-optimal configs would make attribution claims defensible, not directional.
ToolArgumentAccuracy checks calculator expressions (evaluate-and-compare), file paths, channel names, and SQL key terms (table/filter/column via contains). Full SQL semantic equivalence — where SELECT * FROM products WHERE category='hardware' ≡ SELECT * FROM products WHERE category = 'hardware' — is a harder normalization problem left as future work.
All 55 cases have single correct answers. Real agents face queries with multiple acceptable answers, graded by rubric.
The 6-phase design is on paper. A production executor with BoTorch + Ax would trace the actual Pareto frontier — the brief says this is optional, but it's the obvious next step.
benjamini_hochberg_fdr() is implemented in stats_utils.py but never called. With 7 per-case evaluators × 20+ configs, 350+ p-values run uncorrected. Auto-applying BH-FDR would control the false discovery rate instead of reporting raw p-values that overstate significance.
Part 2 tests one topology: research→analyse→synthesize. Real agent teams use fan-out (parallel sub-agents), fan-in (merge multiple sources), and iterative refinement loops. Different topologies may show different error propagation patterns — the current propagation coefficient is measured for one shape only.
Selected highlights below. The full citation list — 40+ papers spanning experimental design, Bayesian optimisation, prompt optimisation, multi-agent attribution, and evaluation metrics — is in REFERENCES.md.
Every document opens on GitHub, where markdown, tables, and code blocks render in full. The full source for this submission lives at github.com/aryaminus/era-eval.