November 5, 2026 · 5 min read · Leer en español

One model, four harnesses: a code review in search of vulnerabilities

A comparison of pi, DeepSeek Harness (dsh), OpenCode and Vercel's fx with the same model, the same prompt and the same codebase.

The question

People compare models all the time in security work, but almost never the harness, even though the harness decides what the model can really do: which tools it receives, how many times it iterates, how it packs the context, how the result ends up written. So I ran an experiment to find the best harness for looking for vulnerabilities in code. I changed the harness four times and measured what moved: volume, precision, cost and time.

Methodology

What was compared

harness	version
pi	1.5.0
DeepSeek Harness (dsh)	0.2.1-rc.5
OpenCode	v2.2.5
fx (Vercel Labs)	0.0.16

What was held fixed

Model: deepseek-v4.1-flash:1017 at https://ollama.com/v1.
Prompt: Find critical and high severity defects, report the origin line, the source-to-sink chain, the attacker requirements and the impact. Quality before quantity.
Target: a large Node.js application, with an isolated copy of the same tree per run.
Sandbox: one Docker image, one container per harness, unprivileged user. Outbound network allowed to reach the model endpoint.
Limits: 28-minute timeout per cell (the experiment’s limit, not the harness’s), 3 harnesses in parallel.

How the benchmark was run

It was a very crude setup. Each harness ran in its own Docker container, opened with the same prompt. I ran that prompt 5 times per harness, then averaged the findings, the cost and the time of each harness.

The verification filter

A reported defect is a claim until someone tries to break it. I grouped the reports by root cause and gave each one to a read-only reviewer told to refute it, with a fixed list of questions: what access does the attacker start with, how does each prerequisite arrive, does that prerequisite already grant the same power, is a real privilege boundary crossed, and was the impact demonstrated or only assumed. Anything doubtful counted as refuted.

Results

Four gate verdicts: vulnerability in green, degraded in yellow, inflated severity in orange and false positive in red, each with its definition. — Fig. 1 — The four labels that appear in the rest of the article.

One square per reported defect in each harness, green for real, yellow for degraded, orange for inflated severity and red for refuted, with the precision percentage on the right. — Fig. 2 — Headline results of the benchmark

Two panels showing, per harness, the cost range per run with the median as a dot, and the duration range per run with the median as a dot. — Fig. 3 — The price of a run. The line runs from the cheapest run to the dearest; the dot is the median. The same four rows in both panels, so a slow and expensive harness shows up immediately.

Scatter of total spend against verified vulnerabilities per harness: pi the most efficient point, OpenCode the most expensive per finding. — Fig. 4 — What a vulnerability costs in each harness

Dot matrix of the nine verified root causes against the four harnesses, sorted by how many found each, above a bar per harness that splits its findings into unique and shared. — Fig. 5 — What each harness found alone, and what it shared with the others

Conclusions

With the same model, the same prompt and the same codebase, precision ran from 80% in the best harness to 36% in the worst. The harness is not something we can overlook today: it decides what gets found and how much noise you have to discard afterwards.

dsh wins on quality. It was the only one that produced no false positive at all: 10 reports, 8 held, one came out degraded and one with its severity inflated, none collapsed. It also covered the most distinct causes, 6 of the 9, the same as pi, and it is one of the two harnesses that found a cause the others missed. For deepseek-v4.1-flash:1017, DeepSeek Harness is the one that performs best.

That result needs one caveat: DeepSeek optimised the model I ran for its own harness, as the release announcement says, so dsh ran on the harness it was trained for and the other three did not. Part of that advantage belongs to the model, not to the harness.

pi wins on price. $0.29 per run and $0.18 per verified vulnerability, against $0.44 and $0.26 for dsh. And it does not find less: pi and dsh tie on real findings (8) and on causes covered (6 of 9). The only thing that separates them is noise: 4 refuted reports out of 12, against none out of 10.

	pi	dsh
median cost per run	$0.29	$0.44
cost per vulnerability	$0.18	$0.26
median duration	12 min	19 min
real findings	8	8
distinct causes, of 9	6	6
refuted reports	4 of 12	0 of 10

pi misses three causes. dsh and OpenCode also found one of them, only fx found another, only dsh found another. It does not fail in the core that everybody shares: it falls short at the edges.

The other two drop out of the final. fx lost 3 of its 8 cells and still reached 75% precision with 5 causes: it performs when it finishes, and it does not always finish. OpenCode is the cheapest per run and the dearest per finding, with 36% precision and 4 causes.

My own choice is pi, for cost and for efficiency. With the same coverage and less spend per finding, the gap it keeps is precision, and precision goes up without changing harness: an AGENTS.md with the reporting criterion, and skills with the flows this kind of review repeats. dsh carries part of that criterion inside the model weights; pi receives it written in the repository.

That part is not measured. Here I compared four bare harnesses, with no AGENTS.md and no skills, five runs per harness. That is enough to see the difference that exists, not to claim how much closes when you add instructions. It is the next measurement, not a result.

My final conclusion:

The harness tells you a lot about what you are going to find, so it is no longer something to overlook: research it and use the one that fits your workflow best.