Reproduce DeepSeek V4.1 Flash DeepSWE with DSH Minimal

DeepSeek published a reproducible evaluation path for V4.1 Flash on DeepSWE v1.1. The release is unusually useful for Harness research because it includes fixed Pier and DeepSWE revisions, a dsh-minimal integration patch, container constraints, run commands, and the location of per-task trajectories.

The official model card reports 74.2 resolved with the official mini-SWE harness and 72.6 with DSH Minimal. Those are separate scaffold results. Reaching the same number requires matching the evaluation conditions; a single local run is a diagnostic, not a reproduction of the reported aggregate.

What DeepSeek reports

The scaffold table holds V4.1 Flash constant and changes the agent runtime:

DeepSeek says the DeepSWE table uses eight samples per task. The shared conditions include Linux containers, temperature=1.0, top_p=0.95, one-million-token context, and a maximum of 500 agent steps. The result shows why the Harness must be named alongside the model score.

Reproduction prerequisites

You need Docker, Python 3.12 or newer, uv, and a DeepSeek-API-compatible endpoint. Keep the key outside scripts and logs:

export DEEPSEEK_API_KEY="your-key"
export DEEPSEEK_BASE_URL="https://api.deepseek.com"

The official instructions pin Pier and DeepSWE to exact commits. Keep those revisions unchanged for the baseline run, then record any later experiment as a separate configuration. The supplied dsh-minimal.patch connects the Harness SDK to Pier's ATIF trajectory format and adds evaluation constraints.

Why the patch matters

The patch covers several details that can change a score even when the model is identical:

• It bind-mounts the prepared Harness SDK read-only into each sandbox.
• It tells agents to work in /app, leave /tests untouched, and avoid network or package mirrors.
• It passes test-runner concurrency limits into containers.
• It enables IPv6 loopback for test suites that bind to ::1.
• It preserves Pier's default /logs mounts while adding the DSH mount.

These are experimental controls, not incidental setup. Omitting one can change which tests execute, how much CPU a test runner consumes, or whether the trajectory and patch are collected.

Prepare DSH Minimal

The official path installs a deepseek-harness-sdk 0.1.5-series artifact into a host directory for Python 3.12 on x86_64 manylinux, then mounts that directory at /opt/dsh-minimal in every task container. The SDK is not installed inside each trial image.

Use the source document's exact command for the first reproduction. At a high level, your run should preserve these facts:

The worker count is not universal. Size concurrency to both CPU and memory; otherwise host contention turns a Harness comparison into a scheduler comparison.

Run mini-SWE as a control

The same instructions include the official mini-swe-agent path. Pier installs it into each task image, and its model string uses the LiteLLM-style deepseek/deepseek-flash form. DSH Minimal uses deepseek-flash directly.

Running both gives you a stronger diagnostic:

• If both regress, inspect endpoint behavior, model routing, container resources, and task revisions.
• If only one regresses, inspect its agent prompt, tool loop, patch, step budget, and trajectory conversion.
• If the pass rates match but cost or tokens diverge, verify provider and cache accounting before attributing the difference to the Harness.

Read and audit the outputs

Pier stores a top-level result.json, one result directory per task, verifier output, the collected patch, and an ATIF trajectory for DSH Minimal. Inspect at least:

1. Aggregate pass rate and token totals.
2. Reward and fail-to-pass/pass-to-pass counts per task.
3. Tool calls and reasoning transitions in the trajectory.
4. Test output for failures caused by the environment.
5. Tasks that pass in one scaffold and fail in another.

The browser command uv run pier view jobs/<job-name> is useful for reviewing one job. Preserve raw outputs and a manifest containing commit hashes, SDK version, endpoint, run time, concurrency, model ID, and reasoning effort.

How to report your result responsibly

Publish the number of tasks, samples per task, mean resolution rate, uncertainty across repeated runs, actual spend, and failure categories. A result from a different API route or a moving model alias is a new experiment. Label it accordingly.

The official 72.6 DSH Minimal score and 74.2 mini-SWE score do not establish a permanent ranking between Harnesses. They describe one model snapshot, one benchmark version, and a detailed but bounded evaluation setup.

Related reading

• DeepSeek V4.1 Flash overview
• Configure V4.1 Flash in DeepSeek Harness
• V4.1 Flash vs V4 Flash in Harness workloads
• DeepSeek Harness vs Pi community benchmark

FAQ

Is 72.6 the overall DeepSeek V4.1 Flash DeepSWE score?

It is the score DeepSeek reports for DSH Minimal in the cross-scaffold table. The primary model comparison uses the official mini-SWE harness and reports 74.2.

Can one run reproduce the official score?

No. The model card states eight samples per task for DeepSWE scaffold evaluation. One run can validate the pipeline and reveal failures, but it cannot reproduce that aggregate.

Why pin Pier and DeepSWE commits?

Tasks, verifiers, container behavior, and adapter code can change. Fixed revisions make the baseline inspectable and keep later changes from silently entering the comparison.