Evaluate a coding agent on representative tasks from your repository, starting each attempt from a recorded state and grading the resulting change against criteria set in advance. Track correctness, scope, reporting, human intervention, and available cost or time data separately.
You are evaluating a working configuration: model, agent application, instructions, tools, permissions, and environment. A result without those details is difficult to reproduce and easy to overgeneralize.
Start with a small suite you can inspect closely. The objective is to make an adoption decision on your work, not to manufacture a leaderboard.
Choose tasks that represent the intended use
If your team wants help with small maintenance fixes, do not evaluate only greenfield feature creation. If you need repository explanations, a patch-only success metric misses the job.
Build a task list from real work, sanitized where necessary. Include ordinary tasks and known difficulties: a local bug, a contract-preserving change, a test improvement, and an investigation whose correct outcome may be “more evidence is needed.”
Keep task selection visible. A suite containing only problems the agent previously solved will overstate usefulness. A suite containing only unusually difficult failures may understate its value for routine work.
For each task, record:
{
"task_id": "normalize-tags-01",
"starting_revision": "<recorded commit>",
"request": "Trim tags, remove empty values and duplicates, preserve first occurrence order and case.",
"allowed_changes": ["tags.py", "tests/test_tags.py"],
"acceptance_checks": ["trim", "empty", "deduplicate", "order", "case"],
"prohibited_effects": ["network writes", "production data access"],
"reference_solution_visible_to_agent": false
}
This is a proposed task record. Replace the paths and rules with verified project facts. Keep reference patches and evaluator-only checks outside the agent's writable environment when testing whether it can solve the task independently.
Define what a successful attempt means
Anthropic's evaluation guidance distinguishes tasks, repeated trials, graders, transcripts, and final outcomes. The distinction is useful: a transcript saying “fixed” is different from software that satisfies the task.
Use separate dimensions instead of letting a high score in one area hide a critical failure.
| Dimension | Example evidence | Decision role |
|---|---|---|
| Behavioral correctness | Acceptance checks and existing regression suite | Required for an accepted change |
| Scope and compatibility | Diff review, preserved interfaces, dependency review | Required constraints |
| Reporting accuracy | Claims matched to actual command results | Required for a trustworthy handoff |
| Maintainability | Reviewer assessment against project conventions | Review finding, with reasons |
| Human effort | Clarifications, repairs, review time | Adoption tradeoff |
| Resource use | Recorded latency and provider usage when available | Comparison metric |
Do not average an unauthorized external action into an otherwise good score. Decide in advance which failures disqualify a run.
Also avoid grading incidental behavior as if it were the outcome. Requiring an agent to read a specific file can make sense when that file is a mandated contract. Requiring a particular sequence of searches may unfairly penalize a correct, efficient alternative.
Check the grader before trusting its scores
For the synthetic tag task, the expected behavior is:
[] -> []
[" green ", "green", "blue"] -> ["green", "blue"]
["blue", "green", "blue"] -> ["blue", "green"]
["", " ", "green"] -> ["green"]
["Green", "green"] -> ["Green", "green"]
The order rule matters. A solution using a sorted set removes duplicates but may reorder the result. A solution that only trims and removes empty values still leaves duplicates.
We ran five cases against a known-correct implementation and these two deliberately incorrect variants. The reference passed all five. The sorting variant failed the first-occurrence-order case, and the duplicate-preserving variant failed both duplicate cases.
That is a calibration check of a small grader, not an evaluation of a coding agent. It establishes that these checks detect these two defects. It does not show that the suite catches all wrong implementations or represents the complexity of your repository.
A tiny reusable grader can begin with:
def grade(candidate, cases):
results = []
for case in cases:
actual = candidate(list(case["input"]))
results.append(actual == case["expected"])
return all(results)
In a real harness, capture exceptions, command exit status, and timeouts as structured outcomes. Run untrusted candidate code in an appropriately restricted environment. Keep the grader and its expected values out of the candidate's control.
Python's unittest documentation offers an established test-runner interface when the repository uses Python. The important choice is an observable contract, not a particular framework.
Run independent trials under comparable conditions
Create a fresh checkout or equivalent isolated environment for each trial. Restore the starting revision and data fixtures. Avoid carrying generated files, conversation history, or successful patches from one candidate into the next.
Record the agent version, model identifier, settings that affect execution, instruction files, tool availability, permission policy, and dependency state. If an external service changes during the comparison, record that limitation rather than attributing every difference to the agent.
Use repeated trials because outputs can vary. There is no universal number that makes a small pilot statistically conclusive. Choose repetition and budget according to the decision's importance, then report the actual counts.
For a fair comparison, state whether candidates have the same time limit, token budget, or practical operating policy. These are different experimental choices. Allowing one candidate repeated manual rescue while another must work unattended measures different workflows.
Do not erase failed or interrupted attempts. Classify an environment failure separately from an implementation failure, while preserving both in the denominator appropriate to the metric. If you rerun a task after fixing the harness, keep that provenance.
Record attempts before aggregating them
A trial record should connect the result with its evidence:
Task and starting revision
Agent configuration and permissions
Start/end timestamps and stopping reason
Patch or resulting artifact
Acceptance and regression check results
Reviewer findings and disposition
Human clarifications and edits
Available usage measurements, with units
Missing measurements and environment failures
For a pilot, a table of task-by-task outcomes may be more informative than a single percentage. “Three of four parser changes accepted; both integration tasks required human repair” suggests a narrower rollout than “60% success.”
Distinguish first-attempt acceptance from success after correction. Both can matter, but they describe different costs and levels of supervision. If a model-based judge contributes to grading, compare some of its judgments with expert review and inspect disagreements.
Use the results to set a boundary
The useful conclusion is often conditional: the configuration is suitable for isolated maintenance changes with these checks, while migrations or security-sensitive changes need a different workflow.
Reevaluate when the model, harness, tools, instruction files, repository architecture, or permission policy changes materially. Preserve a stable regression set and periodically add newly observed failure modes.
Keep development tasks separate from held-out evaluation tasks when tuning prompts or instructions. Otherwise, the evaluation increasingly measures adaptation to the known suite.
Begin with one real decision you need to make and a handful of representative tasks. Calibrate the checks, run the trials, inspect the diffs, and report what the evidence supports. A modest, reproducible result is more useful than a confident ranking built on incomparable runs.
Execution note: the five-case grader calibration ran locally on October 28, 2026 with Python 3.14.2. No coding-agent trial, provider cost measurement, or tool ranking was performed.
