Is it the model or the harness? Benchmarking an AI threat hunter across six LLMs Back to Blog
Engineering

Is it the model or the harness? Benchmarking an AI threat hunter across six LLMs

Rakesh Mukherjee November 17, 2026 ~13 mins

We ran six LLMs through Scout's full harness on ten frozen attack scenarios, three times each. The model sets how good an AI threat hunter can be; the harness decides whether that quality shows. Four lessons for anyone evaluating agents, and a leaderboard.

TL;DR: We put six LLMs, from a 30B open model on two of our own GPUs to the newest Gemini, through Scout's full harness on ten frozen attack scenarios, three times each. Both matter: the model sets how good Scout can be, and the harness decides whether that quality shows at all. One harness fix lifted every model it was tested on by about 0.12 recall, half the gap between our production model and a small self-hosted one. Along the way we learned four lessons about evaluating agents, and we close with a checklist for teams doing the same.

→ See how Scout hunts across your own data

At Huntbase we build Scout, an AI threat hunter that investigates across a security team's cloud, identity, endpoint and SIEM data. It plans a hunt, writes queries in each source's own query language, runs them through Huntbase, and turns the results into findings and a verdict. "The harness matters more than the model" has become a common claim about AI agents, and most of Scout is harness, so we tested the claim on it. We ran six models, from a 30B open model on two of our own GPUs to the newest Gemini, through the same harness, the same frozen environment and the same ten attack scenarios, three times each.

The short answer: both matter. The model sets how good Scout can be, and the harness decides whether that quality shows at all. One harness fix lifted each of the three models we tested it on by about the same amount, roughly 0.12 recall. That is half the gap between our production model and a small self-hosted one, and across all six models the best-to-worst gap is more than three times larger.

Along the way we learned four things about evaluating agents that we wish we had known at the start: single-call benchmarks overstate agent readiness, lenient scoring and single runs make models look interchangeable, the serving stack is part of the model, and one run tells you almost nothing about reliability. This post covers how we tested, what we found, what it means for Huntbase customers, and why we build Scout the way we do.

Why we ran it

Scout is the centre of Huntbase. Security teams connect their data sources to Huntbase, and Scout does the hunting across them. An analyst asks a question in chat, or a detection or intel item seeds an autonomous hunt. Scout then plans the investigation, writes and runs the queries, reads the results and reports what it found. Around the model sits about twenty sub-agents' worth of engineering: an investigation planner, query authoring for each platform's query language, validators and query repair, a pipeline that runs queries across the customer's connected platforms, and the state that ties a hunt together.

Scout's hunt flow: model calls and harness steps
Scout's hunt flow: model calls and harness steps

The model makes the judgement calls: what to look for, how to ask each source, and what the results mean. Everything else is harness, and that is where most of our engineering goes. That raised two questions we wanted to answer for ourselves.

The first was practical: could Scout run entirely on open-weight models, on infrastructure we or a customer control, with no calls to a hosted LLM? If it could, Scout could go where hosted models can't.

The second is the one every agent builder argues about, and for us it decides where engineering time goes: does the harness (the prompts, tools, planner, validators and execution pipeline around the model) matter more than the model itself? If it does, model choice is a cost decision and our engineering effort belongs in Scout's harness. If it doesn't, we should chase the frontier.

Both questions have the same shape. You can't answer them with a public leaderboard, because public leaderboards measure models in isolation, and Scout is not a model in isolation. So we built a benchmark that measures the whole agent, and made it repeatable so we can run it again whenever a new model ships or the harness changes.

How we tested an agent, not a model

Three design choices did most of the work.

1. Test at several layers, from one call up to the whole hunt. We scored single sub-agent calls (write this query, plan this investigation), live multi-turn chats where the model chooses every tool call, and fully autonomous hunts where Scout's planner turns a seed alert into a graph of queries that the platform executes. The layers differ in who controls the flow of work: the model, or the harness. That turned out to be the most useful axis in the whole study.

2. Freeze the world. Every run starts from the same saved snapshot of a complete Huntbase stack: 22 simulated data sources (cloud, identity, endpoint, SIEM, SaaS) with planted attack evidence for six incident stories, all anchored to a fixed point in time. The ten hunt scenarios come from those six stories: four of the stories are hunted twice, each time from a different first clue. We restore the snapshot before every repeat. Without this, results drifted with simulator randomness, timestamps that moved on every restart, and leftover hunts from earlier runs that the agent could find and copy.

3. Only count evidence the agent established. A hunt scores recall on planted entities (users, hosts, IPs, domains, files) only when they appear in the agent's own conclusions: verdicts, findings, the hunt summary and its facts. Restating the alert, or naming something in a plan step that never ran, earns nothing. Our first scoring version gave credit for words in node labels, and quietly rewarded agents for repeating the question.

Every agent in a trial runs on the model under test (a per-run override follows the request across HTTP and Kafka into every worker), and we turned off Scout's silent fallback to Gemini. Otherwise a typo in a model name would have produced a perfect Gemini score labelled as something else. Each configuration ran three times; we report the average, how often all three repeats passed, and how often at least one did.

A hunt passes only when every check passes: it settles before the time limit (a final status, or every step finished), it plans at least two steps, no more than a third of its steps fail, it finds at least half the planted entities, and its verdict matches the expected one. One of the ten scenarios has no expected verdict, so it is scored on the first four checks. A hunt's score is the share of its checks that pass. The single-call planner score uses a rubric graded by an LLM judge (Gemini 3 Pro), which never runs as the model under test; every other score is deterministic. Where we call a difference significant, its 95% confidence interval, from a paired bootstrap over scenarios, excludes zero.

Lesson 1: single-call benchmarks overstate agent readiness

On the narrow task of writing one detection query from a natural-language request, a 30B open model running on two of our own GPUs scored 90 (with the decoder fix described in Lesson 3) against Gemini 3.8 Flash's 97, statistically indistinguishable, and answered faster. On that evidence alone you would ship it.

Put the same model in Scout's chat loop and it scored 20 against Gemini's 93. The reason was not query quality. It wrote good queries and then stopped: "I've generated queries that will find this activity." It never called the tool that runs them, so its answers never contained the evidence. gpt-oss-120b, larger in total but with only about 5B active parameters, behaved the same way. DeepSeek V4.1 Flash, with about five times the active parameters of the 30B model, wrote, ran, read and followed up, and landed near Gemini.

Qwen3-30B vs Gemini 3.8 Flash at three benchmark layers
Qwen3-30B vs Gemini 3.8 Flash at three benchmark layers

On one query the gap is small. Once the model has to carry an investigation itself, the gap opens wide. The autonomous hunt sits in between, because Scout's planner and the platform do part of the driving.

The gap between "can do the step" and "will carry the task through" is invisible to any benchmark that scores one call at a time. For agent products, follow-through is a capability in its own right, and in our runs it behaved like a threshold: below it, prompt nudges and tool-level fixes did not help (we tried accepting malformed arguments, recovering tool calls written as text, and "now run it" hints: no measurable change over three repeats), above it models converge quickly.

Lesson 2: lenient scoring can make every model look the same

Our first autonomous-hunt results (one run per model, four scenarios) looked like a clean win for "the harness matters more": the small self-hosted model matched Gemini on hunts, with the same score and slightly better recall. When the planner and the platform drive the work, it seemed, the model barely matters.

That reading was wrong, and the way it was wrong is the most transferable lesson in this post.

Every model, from the smallest to the strongest, seemed to sit at the same 0.35–0.4 recall. We read that as a shared ceiling and spent three rounds removing suspects:

The ceiling was in how conclusions were formed. The hunt summary that downstream steps read contained row counts ("Returned 14 rows") but never the rows themselves. And a background verdict was never recorded, because the agent's own writes marked the hunt as "analyst engaged", which blocked autonomous verdicts by design. The model saw the evidence while it worked, but nothing it saw reached the conclusion.

Hunt conclusions before and after the evidence-grounding fix
Hunt conclusions before and after the evidence-grounding fix

Both problems sat after the queries had run, which is why fixing the environment, the routing and the lake never moved recall.

Once we gave the conclusion-forming steps a sample of actual result rows and let autonomous hunts record a verdict, recall rose by +0.10 to +0.15 for each of the three models we tested (Gemini 3.8 Flash, DeepSeek V4.1 Flash and Qwen3-30B; four scenarios, three repeats each). The ranking did not change. And the "parity" between the 30B model and Gemini turned out to be noise: measured properly, with three repeats and evidence-only scoring, Gemini sat about 0.2 recall above it both before and after the fix.

So the shared ceiling was partly an artifact of how we measured: with single runs and lenient scoring, two models 0.2 recall apart looked the same. Per-model confidence intervals are wide at four scenarios. Pooled over all twelve model–scenario pairs, the lift averaged +0.12, and ten of the twelve were zero or better.

Hunt recall with grounding off and on: 4 scenarios, 3 repeats each
Hunt recall with grounding off and on: 4 scenarios, 3 repeats each

Each model moved by a similar amount, and the order did not change.

Two things are true at once. The harness determined whether model quality could show up at all, and one harness fix was worth about half the gap between Gemini and the 30B model. And model quality still mattered, as soon as the harness stopped hiding it.

Lesson 3: the serving stack is part of the model

The single biggest swing we measured on one model came from a serving flag, not the model.

Scout's agents ask for strict JSON output against a schema. With vLLM's default settings, the constrained decoder allows unlimited whitespace between JSON tokens. On long prompts the open model drifted into writing spaces and newlines until it hit the 16k-token cap, and produced no query at all. Query-authoring score: 51.6.

Forcing compact JSON (no whitespace) fixed it: 94.8. It also broke the planner. Denied its natural ": " token, the model filled string fields with literally ", ", and planning fell from 73 to 51. We ended up patching the decoder to allow whitespace but cap it at 32 characters per run, which recovered both (90.2 and 71.9).

Qwen3-30B on vLLM under three JSON decoding settings
Qwen3-30B on vLLM under three JSON decoding settings

Each setting that helped one agent hurt the other, until we capped the whitespace.

This is not a self-hosting quirk. Google's managed endpoint for an open model showed the same whitespace runaway on our planner schema. And Anthropic's API rejected our planner schema in strict structured-output mode: the compiled grammar was too large. In JSON mode, with the schema in the prompt, Claude Opus 5.5 produced the best plans we measured in single-call tests.

The practical rule: a model's score is a score for model + serving stack + output contract. Every serving change has to be re-run across every agent, because a fix that helps one agent can break another. And "strict structured output everywhere" is an assumption that holds for one provider and fails for others; output mode should be configurable per model and per agent.

Lesson 4: report reliability, not just capability

Single runs lied to us twice.

DeepSeek V4.1 Flash scored 97.6 on our live chat suite in one run, apparently matching Gemini. Over three repeats it averaged 80. It solved 86% of the cases at least once (pass@3, the same as Gemini) but only 43% every time (pass^3; Gemini managed 86% in the same run). Its bad runs were ones where its own queries failed to execute against our data and it answered "can't tell yet". For a security analyst, a tool that is right on Monday and silent on Tuesday is a different product from one that is right every day. Gemini's own pass^3 on the same seven conversations ranged from 43% to 86% across three separate runs, so even the baseline needs repeats.

Live chat reliability: Gemini 3.8 Flash vs DeepSeek V4.1 Flash
Live chat reliability: Gemini 3.8 Flash vs DeepSeek V4.1 Flash

In this run, the same pass@3 and half the pass^3: DeepSeek's misses came and went between repeats.

The second lie was about our own work. A set of tool-call robustness fixes appeared to lift the small model by 21 points in one run. At three repeats it had no measurable effect.

On hunts, even the best model is far from reliable on our harder scenarios: Gemini 3.8 Flash passed 5 of 10 scenarios at least once, but only 2 of 10 on every repeat. We now report the mean, pass@k and pass^k side by side, and treat any single-run delta under about 10 points as noise.

A related trap: an averaged score can hide how a model fails. Claude Haiku 5.5 found as much planted evidence as Gemini on hunts, but passed only 1 of 30 trials. Its plans were 60% larger than Gemini's (a median of 13.5 steps vs 8.5) and about half of each plan's steps failed to execute, which our "executable plan" check catches. Breadth without precision looks like recall until you check whether the plan actually ran.

The leaderboard

With the harness fixed and the world frozen, we ran six models through ten autonomous hunts, three times each. Every model ran the full Scout harness; only the model changed.

Huntbase Scout benchmark, November 2026: ten hunt scenarios, three repeats per model. Score = share of checks passed per hunt; recall = share of planted entities in the hunt's own conclusions.
Huntbase Scout benchmark, November 2026: ten hunt scenarios, three repeats per model. Score = share of checks passed per hunt; recall = share of planted entities in the hunt's own conclusions.

Gemini 4 Argon led on score, recall and pass rate: it passed every scenario at least once, found the most planted evidence on 7 of the 10, and took about as long per hunt as the current production model. Its lead over Gemini 3.8 Flash (+6.8 score, 95% CI −3.5 to +16.7) is not yet statistically significant at ten scenarios. Claude Sonnet 5.5 sat within noise of Gemini 3.8 Flash and finished hunts about 30% faster. In a separate head-to-head on live chat, Argon also beat Gemini 3.8 Flash: 98 vs 91 over seven conversations, three repeats each, a statistically significant gap, at about 1.7× the time per conversation.

The open-weight models fell behind on hunts for two different reasons. Each failed to produce a plan at all in 7 of 30 hunts. The small self-hosted model found little (mean recall 0.18). DeepSeek, strong in chat, wrote plans with more steps that could not run. Both fail in ways a single-call benchmark would never show. Both also ran on the kind of structured-output serving Lesson 3 describes, and we have not yet separated model errors from serving errors in their failed plans.

And nobody is reliable yet on the harder scenarios: no model passed more than two of the ten on every repeat. That is the next ceiling to find.

Limitations

What this means for Huntbase customers

Scout runs on Gemini 3.8 Flash today, and the benchmark backs a hosted Gemini model: Gemini 3.8 Flash led the four-model chat tests (against DeepSeek V4.1 Flash, gpt-oss-120b and Qwen3-30B; the Claude models were not run in chat) on quality and consistency, and Gemini 4 Argon led hunts on score, recall and pass rate. In a head-to-head chat run Argon also beat 3.8 Flash (98 vs 91), though each conversation took about 1.7× as long. At about 2.2× the model cost of 3.8 Flash per hunt and per chat, we are now evaluating whether Argon's lead is worth it for Scout. More importantly, we can offer customers options backed by evidence rather than guesses.

And the harness work keeps paying: the evidence-grounding change found in this study lifted every model's hunt results, including the self-hosted one.

A checklist for teams evaluating agents

  1. Benchmark the agent, not the model. Include at least one layer where the model drives the loop and one where your harness does; the difference between them is the most informative number you will get.
  2. Freeze and restore the world before every repeat: data, timestamps, search indices and leftover state.
  3. Score only what the agent established, never words it could have copied from the prompt.
  4. Kill silent fallbacks and assert which model actually answered.
  5. Run k ≥ 3 and report pass^k next to the mean. Treat single-run deltas as rumours.
  6. Treat serving config as a variable: decoding settings, structured-output mode and tool-call parsing, re-tested across every agent after each change.
  7. When every model scores the same, look for a ceiling in the harness or the environment (or in your scoring) before concluding the model doesn't matter.

So, is it the model or the harness? In our data, both: one well-placed harness change was worth about half the gap between our production model and a small self-hosted one. Across all six models, the best-to-worst spread was more than three times that lift. The harness decides how high any model can go, and the model decides how close to that ceiling it gets. The work that paid off most was finding where the harness was hiding the model.

Why Huntbase

This study was not a one-off project. It shows how we build Scout.

If your team hunts across cloud, identity, endpoint and SIEM data and wants an AI hunter it can hold to the same standard, we would like to talk.

#AI #Detection #Automation
More Articles