HarnessDev Explained: Can AI Build Its Own Agent Harness?

The paper “HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?”, submitted on September 1, 2026, moves evaluation from one task output to runnable agent infrastructure. A model starts with a weak seed, builds a harness, and then revises it using execution feedback.

HarnessDev is independent research with no affiliation to the official DeepSeek Harness project. DeepSeek V4 Pro is one of six harness-creator models in the study, and the findings help define the current limits of agent-driven harness engineering.

Evaluation design

HarnessDev has two stages:

• Creation: a model builds a complete execution system from a minimal seed and a small set of examples.
• Evolution: the model iterates on its own harness using feedback from designated development tasks.

Creation covers six creator models, four domains, and five downstream benchmarks with 2,207 unique instances: SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp. Evaluation records task performance and executor-token cost.

The study separates Self-Eval from Unified-Eval. Self-Eval runs a harness with its creator model. Unified-Eval uses Gemini 3.1 Pro as the common executor to test portability across models.

Main reported findings

Models can produce runnable harnesses

Several models build complete systems for tool calls, context, state, and lifecycle management from the weak seed. Opus 4.8 records the highest Self-Eval aggregate score at 67.8, compared with 86.2 for the paper's human-engineered reference aggregate.

Results vary sharply by domain

Generated harnesses approach the selected reference on short-form writing and sometimes exceed it in machine-learning experimentation. Code retains a clear gap, while search and research show the largest shortfall.

Evolution can help, with unstable gains

The Evolution stage produces improvements and regressions. After freezing a version and moving to tasks hidden from the creator, gains transfer only partially.

Harnesses remain coupled to executor models

Changing the executor can collapse performance. One reported Code harness built for Opus hard-codes a 120-step limit and nearly fails under Gemini. Strong performance with the original model does not establish broad portability.

More tokens do not guarantee a better result

On MLE-bench, a GPT-5.5-created harness reaches a 19.1 medal rate with 29.3 million execution tokens. A DeepSeek V4-created harness uses 208.4 million tokens for a 19.6 medal rate. Similar outcomes carry roughly a sevenfold token difference in this comparison.

Important limitations

The human-engineered references are verified public system results paired with their original executor models. They were not all rerun under one common executor, so they illustrate mature-system ceilings rather than a fully paired control.

Execution-token totals exclude the creator's cost of writing and revising the harness. A production cost model must also include creation, diagnosis, validation, and rollback.

Evolution currently focuses on code harnesses. Long-term self-evolution in other domains needs additional evidence, and the reported results await independent replication.

Implications for the DeepSeek Harness ecosystem

1. Agent Presets should identify versions, models, task sets, and reproducible evaluations.
2. Agent-authored plugins need regression tests, permission ceilings, and human review.
3. Plugin catalogs can disclose suitable models, cost, and known failures.
4. Complete trajectories help expose step limits, stopping rules, and context policies that create hidden regressions.
5. Harness value needs joint measurement of success, stability, and cost.

Relationship to JIT-Agent

JIT-Agent studies just-in-time harness generation for a task. HarnessDev supplies a benchmark for creating and evolving harnesses. The first reports performance potential, while the second documents engineering gaps, cost variation, and limited cross-model transfer.

Related reading

• JIT-Agent: just-in-time Agent Harness evolution
• MemoHarness: Agent Harnesses that learn from experience
• Architectural convergence across DSH, deepagents, and pi

FAQ

Is HarnessDev official DeepSeek research?

No. It is independent research that includes DeepSeek V4 Pro alongside Opus, GPT, Gemini, Qwen, and Seed model families.

Does the paper show that agents can fully take over harness development?

No. Models can create runnable systems, while code, search, and research still trail mature human-engineered harnesses. Evolution also exhibits regressions and transfer limits.