ToolBottleneckBench tool-menu and budget research

ToolBottleneckBench: Tool Menus, Caps & Harness Bias

Author
DeepSeekAgent.io Editorial Team
Published
Updated

Why can the same model score differently on the same task under another agent harness? ToolBottleneckBench turns that question into a controlled experiment. It holds the task, answer, evidence rule, and tools fixed while changing the tool menu, workflow, record encoding, and budget constraints exposed by the harness.

Its central result is straightforward: an agent success score measures both the model and the harness running it. Many failed episodes end at a harness limit rather than after a complete but incorrect model decision. A benchmark that reports only success, without tokens, call counts, turns, and truncation rules, can mislabel an infrastructure constraint as a model capability gap.

Four controlled pressures

ToolBottleneckBench compiles matched variants of one task and changes one pressure at a time:

  • Registered Tool Catalog (R): how many tools exist in the system;
  • Visible Tool Menu (V): how many tools the model actually sees in the current call;
  • Certified Workflow (W): required dependency and execution constraints;
  • Data Encoding (D): how returned records and their relationships are encoded.

The main experiment freezes the episode budget, call limit, turn limit, and other harness contracts. A deterministic grader checks nine obligations spanning availability, selection, workflow, answer, evidence, safety, governance, budget, and completion integrity. Each failure records both the first unmet obligation and the limit that terminated the episode.

This distinction prevents a common error. If the model's first answer is cut by the completion cap, an obligation-based score may call it a tool-selection failure even though the trajectory actually ended at an output limit.

A large catalog is not the same as a large visible menu

The paper evaluates 21 model–harness configurations. Among the pressures applied directly to the model, the visible tool menu is the only one with a large penalty.

At a visible menu of 128 tools, 472 of 3,024 matched pairs lose success; 343 of those losing variants are truncated. Those truncation-associated pairs account for 11.3 of the 15.6 percentage points lost.

The authors then raise the completion cap sixteenfold to 16,384 tokens for selected truncation-heavy configurations, removing most of the menu penalty. This does not imply that every agent should receive more output tokens. It shows that much of the measured menu effect was a joint effect of menu length and a harness completion limit.

At 2,048 registered tools, retrieval determines availability

A large catalog creates another failure before selection: the required tool may never enter the visible menu.

At a registered catalog of 2,048 tools, the paper reports:

  • 19.4% required-tool recall for a flat lexical retriever;
  • 77.8% for a connector-first router;
  • 24.5% of Discovery episodes failing before the first model call.

Those episodes cannot be blamed on the model choosing the wrong tool. The correct tool was absent from its choice set. Tool discovery, routing, and model selection therefore need separate measurements.

Repairs that actually help

The study compares prompt repairs, routed menus, and configuration switches. The repairs that work across configurations change which tools reach the model or who issues the call; they do not merely ask the model to try harder.

The routed-menu repair produces a 4.9-point stress gain, rising to 7.1 points in repeated runs. The connector-first router works by increasing availability. Neither prompt repair produces a detectable stable gain.

This does not make prompts universally irrelevant. When the root cause is an invisible tool, truncated output, or failed calling layer, additional prose does not repair the infrastructure.

Direct implications for DeepSeek Harness

DSH can register many tools through plugins, but the complete schema inventory should not be injected indiscriminately into every request. A stronger pipeline would:

  1. filter tools by Profile, task, and permission;
  2. route first by connector or capability domain;
  3. expose only the candidates needed for the current step;
  4. record whether a tool was registered, retrieved, visible, selected, and executed;
  5. preserve completion, episode, call, and turn limits in the trajectory.

The resulting evidence separates four failures: the tool did not exist, routing missed it, the model selected incorrectly, or the harness truncated execution at a call or budget boundary.

What an agent benchmark should report

A comparable evaluation needs more than one success score. It should publish:

  • model, provider route, and reasoning settings;
  • prompt template and tool protocol;
  • registered and visible tool counts;
  • completion, episode, call, and turn limits;
  • context trimming, compaction, and retry policy;
  • terminal-reason and truncation distributions;
  • the funnel from registration to retrieval, visibility, selection, and execution.

This is why two products using the “same model” cannot be compared from a headline score alone. The model is one component. The harness controls what it sees, what it can call, and where the run ends.

Relation to DSH context and dynamic tools

DSH long-session compaction addresses history growth, while dynamic tools and KV-cache behavior address tool-definition loading. ToolBottleneckBench adds the evaluation view: tool menus and budgets alter both cost and success attribution.

For an agent runtime with a plugin marketplace, the objective should not be “support the largest number of tools.” It should be “show the right tools at the right step and retain complete evidence about the budgets.”

Research scope

The paper uses synthetic enterprise tasks and a frozen harness contract. Its sixteenfold sweep changes only the completion cap while other budgets remain fixed. The result explains how controlled pressures interact with harness limits; it does not prescribe a 16,384-token output cap for every real task.

References

FAQ

Do more tools always make an agent worse?

No. The large penalty comes from the interaction between the visible menu, retrieval recall, and fixed budgets. A large catalog can be routed without exposing every schema on every call.

Does a higher completion cap solve the problem?

Only when truncation dominates. Retrieval misses, call failures, invalid workflows, and episode limits remain separate problems.

Why did prompt repair not help?

The relevant bottlenecks were in menus, routing, and budgets. A prompt cannot expose an absent tool or recover output that the harness already truncated.