HarnessTester Agent Harness testing research
HarnessTester: Agent Harness Coverage & 122 Real Defects
- Author
- DeepSeekAgent.io Editorial Team
- Published
- Updated
An agent failure does not necessarily originate in the model. Harness code parses model output, constructs tool calls, maintains session state, enforces permissions, and recovers from failure. A defect in that layer can turn a reasonable model intent into incorrect behavior.
“Complex Agents, Shallow Tests” studies ten real open-source agentic systems and asks whether the harness code governing agent behavior is adequately tested.
LLM-dependent harness code
The paper calls code that directly consumes LLM output or controls subsequent agent behavior LLM-dependent harness (LDH) code. Typical paths include:
- parsing model text into tool calls;
- validating tool names, arguments, message roles, and structures;
- deciding whether to continue, retry, or stop from a finish reason;
- writing tool results back into context;
- handling malformed model output;
- advancing workflow state from a model decision.
The difficulty is that execution does not imply semantic correctness. A test may reach a branch without detecting a wrong tool argument, polluted state, or unsafe recovery order. Line coverage alone can therefore overstate reliability.
Where existing tests fall short
The authors report that existing tests cover less than half of LDH lines and branches. Average LDH mutation score is 33.60%, meaning many deliberately introduced behavioral changes survive the test suites.
The study separates code that tests never execute from code that executes without an assertion capable of detecting the changed behavior. The second case is especially dangerous for harnesses: CI can remain green while tool selection, message state, or a permission decision is already wrong.
How HarnessTester changes test generation
A general test generator can create mocks that are type-correct but protocol-invalid. It may fabricate a model response without the matching session, tool registry, message sequence, or state preconditions. Such a test reaches an entry point but not the deeper branch that governs behavior.
HarnessTester explicitly models agent–harness contracts. Generated tests must respect structural, state, and interaction protocols, while a context retriever supplies relevant definitions and fixtures. The objective is not more test files; it is an environment in which LDH code follows a realistic path.
Against the strongest general-purpose baseline selected in the paper, the authors report:
| Metric | Gain |
|---|---|
| Line-coverage gain | 75.95% |
| Branch-coverage gain | 84.76% |
| Mutation-score gain | 69.89% |
Removing contract support also reduces coverage gains substantially. The key contribution is not merely asking an LLM to write more tests, but supplying the protocol the harness actually expects.
What 122 real defects mean
HarnessTester finds 122 real harness defects in current versions of the studied systems. Of these, 88 were previously unknown and 69 had been confirmed by developers when the paper was submitted.
This count is not a reliability ranking and cannot be generalized to every agent product. It demonstrates that the model–harness boundary contains a systematic testing gap and that the gap produces real defects.
Direct implications for DeepSeek Harness
DSH lets model adapters, tools, context injection, session storage, and UI extensions evolve as plugins. That composability also multiplies contracts. Testing whether a plugin loads is insufficient; a practical suite should cover at least:
- Missing fields, wrong field types, and abnormal finish reasons in model output.
- Unpaired, reordered, or duplicated tool calls and results.
- Tool-schema and slash-menu synchronization after plugin changes.
- State isolation across parent agents, subagents, and concurrent sessions.
- Message validity after compaction, resume, and provider switching.
- Safe recovery after permission denial, network failure, or partial writes.
The DeepSeek Harness security model defines what an agent may do. Harness testing must verify that implementations actually preserve those boundaries. Policy, contract tests, fault injection, and mutation testing belong in one assurance chain.
A practical test stack
| Layer | What it validates |
|---|---|
| Contract | Message, tool, event, and state structures |
| Transition | Continue, stop, retry, compact, and resume behavior |
| Fault injection | Timeouts, truncation, duplicate callbacks, tool failure, partial writes |
| Isolation | Profile, session, subagent, and plugin scope |
| Mutation | Whether tests catch broken permission checks and branch conditions |
| End-to-end | Complete trajectories and artifacts under a fixed model stub |
Model outputs can use deterministic fixtures, but the fixtures must preserve real protocol relationships. This makes a failure attributable to a harness regression instead of sampling variance.
Research scope
The study covers ten open-source systems and the authors' LDH classification. Coverage and mutation score measure the ability of tests to detect code changes; they are not product correctness rates and do not replace end-to-end evaluation with real models, tools, and permissions.
References
- Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
- HarnessTester paper in HTML
- Source-code study of Agent Harness architecture
- Agent Harness component ablations
- ToolBottleneckBench: tool menus, completion caps, and evaluation bias
FAQ
Does high line coverage mean a harness is adequately tested?
No. Execution does not prove that assertions detect changed behavior. Branch coverage, mutation testing, contracts, and state transitions must be considered together.
Is HarnessTester a DSH testing tool?
No. It is a general harness-oriented test-generation method proposed by the paper. This article applies its findings to DSH's testing boundaries.