DeepSeek Harness vs Pi Community Benchmark
DeepSeek Harness vs Pi: 30-Task Benchmark Explained
- Author
- DeepSeekAgent.io Editorial Team
- Published
- Updated
On September 8, 2026, Composio published a community comparison of DeepSeek Harness and Pi that later drew methodology criticism on Reddit. DeepSeek V4 Pro attempted 30 tool-use tasks involving real applications and data, with a 900-second cap per task.
This is a community benchmark, with no official status from DeepSeek, Pi, or an independent academic evaluator. This article separates the author's reported data from architectural observations and experimental limits.
Reported results
| Metric | Pi | DeepSeek Harness |
|---|---|---|
| Tasks passed | 21 / 30 | 20 / 30 |
| Median time | 362.9 seconds | 252.1 seconds |
| Cost per shared success | $0.031 | $0.028 |
| Reported average runtime tokens | 924,990 | 88,562 |
The two systems share the same outcome on 27 tasks. Pi passes one additional task, while DSH records a lower median time and slightly lower cost on shared successful tasks in this run.
The largest limitation: different API paths
DSH uses the native DeepSeek endpoint. Pi calls the model through OpenRouter. Provider routing, cache implementation, billing, and token accounting can differ.
The 88,562 versus 924,990 runtime-token figures therefore do not establish a roughly tenfold DSH efficiency advantage. The original author also warns against that interpretation. Reported cost per shared success differs by only $0.003.
Reddit participants argue that a harness-isolating comparison should point Pi and DSH at the same DeepSeek endpoint with the same model, cache, and billing path. The public article also omits a complete task list, exact runtime versions, DSH Profile, randomness controls, and repeated-run counts, limiting reproducibility.
What the run still tells us
Similar pass rates across 30 tasks show that the first-party DSH setup does not produce an overwhelming completion-rate advantage in this sample. Harness choice still changes latency, measured cost, and some individual outcomes.
This supports the broader observation that one model can behave differently across harnesses. Reliable agent evaluation needs success rate, latency, actual spend, token-accounting rules, and runtime configuration together.
Core architectural differences
Pi starts from a small core with a few default tools and extends through TypeScript extensions. Its code and default context are easier to inspect, making it attractive for personal daily coding.
DSH supplies a Cordis dependency graph, replaceable agent loops, sandboxes, subagents, MCP, schedules, replay, and append-only trajectories. It asks more from extension authors and offers broader runtime governance and composition.
Part of DSH's non-DeepSeek model support uses Pi's pi-ai model layer. The projects share infrastructure at one layer while choosing different levels of product complexity.
A stronger replication design
- Fix both systems to the same native DeepSeek endpoint and model snapshot.
- Publish exact Pi and DSH versions, DSH Profile, tool catalogs, and system prompts.
- Run every task multiple times and report means, variance, and failure causes.
- Separate uncached tokens, cache hits, billed tokens, and final spend.
- Release the task set, verifier, raw trajectories, and per-task results.
- Test both Minimal and Standard so one Profile does not stand for the entire DSH runtime.
Selection guide
| Need | Start with |
|---|---|
| Small, inspectable terminal coding agent | Pi |
| Deep control of agent loops, sandboxes, and lifecycle | DSH |
| Full trajectories, replay, and plugin dependency management | DSH |
| Frequent provider or local-model switching | Pi |
| Best measured performance | Re-run both on your own tasks and endpoint |
Related reading
- Complete DeepSeek Harness guide
- Architectural convergence across DSH, deepagents, and pi
- DeepSeek Harness hands-on review
FAQ
Does this benchmark prove that Pi has better quality?
Pi passes one more task in a 30-task run. The sample size, runtime controls, and replication detail do not support a universal conclusion.
Does DSH use ten times fewer tokens?
This test cannot establish that claim. The systems use different API paths, so token and cache accounting are not directly comparable; their reported cost per shared success is very close.