JevAgentHarness decision-model research

JevAgentHarness: Decision Models, Tool Choice & Agent Cost

Author
DeepSeekAgent.io Editorial Team
Published
Updated

Why should an agent harness call a large language model at every step? Many intermediate calls ultimately choose from a finite set: which tool to use, which argument set to submit, or whether to continue. JevAgentHarness studies whether a smaller decision model can handle routine choices and fall back to the large model when uncertain.

This is not a simple model replacement. The paper first shows that isolated multiple-choice accuracy does not predict complete agent success. It then fits confidence thresholds for each decision type on the agent's own trajectories and trains a small model on states the agent actually reaches after mistakes.

Decisions that a harness can delegate

The paper represents recurring choices as typed multiple-choice decisions. The state contains what the agent currently knows, and the harness enumerates the available options. The study covers eight decision points, including:

  • choosing the next tool;
  • choosing or constructing tool arguments;
  • selecting the next action when no tool is used;
  • choosing among execution, recovery, and stopping paths.

Good delegation candidates have explicit options, verifiable outcomes, and controlled error costs. Open-ended writing, complex reasoning, irreversible permission changes, and long-horizon planning still belong with stronger models or deterministic policy.

Why isolated accuracy is insufficient

A decision model can answer a clean tool-selection question correctly without controlling a sequential agent well. One early mistake changes later states, pushing subsequent decisions outside the benchmark distribution.

The paper reports a useful contrast:

  • StartLux-9B leads BFCL tool-choice accuracy at 92.2%;
  • it solves only 69.5% of complete agent tasks;
  • AgentDecision-4B scores a lower 89.3% on BFCL but reaches 91.5% task success;
  • step-level decision accuracy ranks models more like task success, with a Kendall correlation of 0.84.

The evaluation unit must therefore expand from isolated choices to complete trajectories. A model that recovers in states created by earlier mistakes may be a better harness controller than one with a higher static-question score.

Confidence thresholds must be fitted per decision

JevAgentHarness does not let the small model control every choice. Each decision type has a confidence threshold fitted on real agent trajectories:

  1. the harness converts the current choice into structured options;
  2. the decision model returns an option and confidence;
  3. the small-model answer is used above the fitted threshold;
  4. lower-confidence cases fall back to the large model.

Without a fitted threshold, delegating tool choices lowers success by 19.0 points. The lesson is not that small models always fail. Different decisions have different error costs, option structures, and calibration behavior.

Why train AgentDecision-4B

Existing decision models usually see clean, isolated examples rather than the states an agent reaches after its own mistakes. The paper trains AgentDecision-4B on visited states, using Jev answers as labels.

On tasks unlike its training data, the paper reports:

  • 91.5% task success when AgentDecision-4B controls tool choices;
  • 82.0% when Jev controls them.

Training data must include deviation, recovery, and non-ideal intermediate states—not only successful trajectories. For an agent runtime, real trajectories better approximate deployment than a static question set.

Cost and speed results

On tasks similar to the training distribution, the paper reports that 4B models trained for each agent's states make the agents:

  • 81.1%–89.0% cheaper;
  • up to 2.5 times faster;
  • no more than three success points worse.

These numbers come from the paper's customer-service and household environments and should not be projected directly onto every coding or research agent. They establish a harness design space: a large model need not answer every finite choice when delegation is trajectory-calibrated and retains a fallback.

Implications for DeepSeek Harness

DSH already provides Profiles, plugins, Subagents, tool schemas, and trajectories. Layered decisions could be explored for:

  • filtering tool candidates by task and permission;
  • routing among read-only retrieval tools;
  • choosing retry, fallback, or a user question;
  • deciding whether to escalate reasoning effort;
  • selecting budgets and concurrency for Subagents.

Permission escalation, production writes, file deletion, and high-value external actions should not be delegated to a cheap model alone. They require deterministic policy, explicit approval, or stronger-model review.

A practical deployment sequence

  1. Identify recurring decisions with finite option sets in trajectories.
  2. Record the correct choice, error cost, and final task result for each type.
  3. Fit per-decision thresholds offline instead of one global threshold.
  4. Run the small model in shadow mode without executing its suggestions.
  5. Compare step accuracy, complete task success, cost, and recovery behavior.
  6. Enable low-risk read-only choices first and keep large-model fallback.
  7. Continue collecting off-track states and monitor calibration drift.

This complements ToolBottleneckBench: one asks who should decide, while the other asks which tools the model sees and how harness budgets change the result.

Research scope

The reported cost and speed gains depend on the selected decision models, trajectories, tasks, and pricing. The paper states that its models, data, and harness will be released, but a production system must recalibrate thresholds and validate provider latency, error costs, permissions, and state distribution.

References

FAQ

Does JevAgentHarness replace the main model with a 4B model?

No. The small decision model handles defined and calibrated finite choices. Low-confidence decisions fall back to the large model.

Why does isolated tool-choice accuracy not predict full-task success?

Each decision changes the next state. Recovery and calibration on off-track states affect the complete trajectory.

Is JevAgentHarness built into DSH?

No. This article covers independent research and maps its design to decision points that DSH could implement.