How we built a threat-hunting agent by engineering the harness around the model instead of praying to the model itself.

The model was never the hard part

At some point the industry decided the frontier model was the whole product. Swap in the newest one, watch the benchmarks tick up, ship. Put an agent in front of a security analyst at 2 a.m. during an incident, though, and you learn quickly that the raw LLM is the least interesting component in the room.

Scout is not a model. Scout is many LLM’s plus a harness. The model reasons. The harness does everything that makes the reasoning useful: it supplies context, exposes the right tools, spawns sub-agents, runs retrieval, and drives the execution loop. Upgrade the model and Scout gets sharper. But the harness is where we spent our engineering time, and it's the part we’ve been most excited to watch in action.

The harness is what decides whether a hunt survives past the chat window, whether a wrong answer ever reaches a human, and whose data the model is even allowed to look at. The rest of this piece is about those three decisions.

We said no to the swarm

The tempting design is a multi-agent swarm: a router here, a planner there, a committee of peer specialists passing messages around like a very expensive game of telephone. It demos beautifully. Then the latency compounds with every hop, the token bill starts to read like a ransom note, and the audit trail turns into something no human or machine can follow after the fact. In our benchmarks, the swarm design came with a substantial token tax per hunt and added latency at every hop, enough that we eventually killed it. When a hunt goes sideways, nobody wants to spend the morning working out which of nine agents hallucinated the pivot.

So Scout runs as a single central reasoning agent. One brain makes every decision, and it stays coherent because it never has to reconcile the opinions of a dozen peers.

The sub-agents still exist, but we demoted them. A sub-agent like a data analyst is a tool, not a colleague. It also serves as a context firewall: it runs in its own isolated context window, does its work, and hands back a structured conclusion. It returns a verdict, not a diary. The messy intermediate reasoning never leaks into Scout's working memory. One agent thinks; the rest report.

Figure 1 — The anatomy of a Scout turn. Sub-agents think in isolation and return conclusions, not transcripts.

Context is RAM, and we manage it like it matters

Karpathy's Software 3.0 framing works well here: the model is the CPU and the context window is RAM. You build an agent by deciding what to load into that RAM on each turn, and just as importantly, what to keep out.

We rebuild Scout every turn from a live Run Context snapshot. Two components do the heavy lifting:

• The Dynamic Prompt Builder renders instructions from the current session state, the active hunt, and whatever triggered the turn. Scout never reads from one giant static system prompt; it gets the prompt that fits the moment.
• The Context-Gated Tool Selector filters Scout's capabilities down to what the current turn actually needs. Hand a model 200 tools when 4 are relevant and it slows down and picks the wrong one. Fewer, sharper options cut latency and tool-use errors — our evals showed a clear reduction in tool-selection errors with gating.

We also refuse to micromanage. Detailed step-by-step prompts feel safe and actively hurt reasoning. Narrating every footstep to a capable model works about as well as it does with a capable engineer. Scout's context supplies the goal and the methodology, then gets out of the way, because the evidence never looks the same twice.

Underneath the analyst-facing answer, Scout keeps an internal monologue. It questions itself, picks which sub-agent or tool to invoke, and tests its own assumptions before it commits to a response. You see the conclusion; it did the arguing in private.

Hold on to the statelessness point. Right now it looks like a constraint. Two sections from now it becomes the payoff.

The many-languages problem

Threat hunting has an inconvenient property: the data lives everywhere and every source speaks its own dialect. A hypothesis in an analyst's head has to become SQL, KQL, SPL, ES|QL, STIX, AQL, or osquery, across 200+ cloud integrations and thousands of tables that no single person has ever fully seen. Along with this every action has a potentially different method (Rest API, GraphQL, MCP, etc) and a subsequent scale impact to a consider.

The naive fix dumps every schema into context. You could do that. You could also set your token budget on fire and get a comparable result. We do three things instead:

• Semantic schema discovery. Scout runs semantic search over capability and specific artefacts of the capability including descriptions rather than ingesting the whole catalog. For the non-standard customer tables that arrive without tidy docs, it performs dynamic schema discovery, augmented by user annotations.
• An extensive library of canned queries. We maintain a pre-indexed repository of tens of thousands of validated queries. These short-circuit generation for common patterns, and they do something subtler: they teach Scout how complex tables get used in the wild, which no schema alone will tell you.
• Organizational memory. Scout ranks and remembers the queries your team leans on, weighting trust by frequency of use, so your own proven patterns rise to the top over time.

The library and organizational memory accelerate patterns that have been seen before. A genuinely novel schema still costs a full discovery pass on first contact, and promoting a finding into shared knowledge remains a human step — the write path belongs to people, by design. Those tradeoffs are deliberate, not solved.

Figure 2 — Assembling just enough context to write the right query, in the right dialect, without ingesting the whole catalog.

RAG that walks instead of reads

Most RAG retrieves static text chunks and hopes the relevant sentence was in there somewhere. That works for a support bot. It fails for threat intelligence, where the relationships between entities are the whole point.

Scout uses agentic graph RAG: it traverses a knowledge graph rather than fetching pages. The graph links CVEs, MITRE ATT&CK and D3FEND, CWE, CAPEC, ThreatFox, KEV, and live malware telemetry. "Is this exploitable here" becomes a walk: from a known-exploited vulnerability, to the technique that abuses it, to the countermeasure that detects it, to whether your own hosts sit in the blast radius.

Two properties keep this safe and current:

• OCSF entity graph integration. Scout automatically extracts OCSF-standard observables — IPs, hashes, domains, users — into a live streaming entity graph, so the local picture never goes stale.
• Tenant isolation with shared intelligence. Canonical public knowledge such as CVEs and techniques merges onto shared global graph nodes, while everything internal to a tenant stays structurally scoped and isolated. You benefit from the world's threat knowledge. Nobody benefits from yours.

Figure 3 — "Is this exploitable here?" answered as a graph walk from world knowledge into your environment, never the other direction.

Hunts that outlive the session

Every Scout turn is stateless by design; the harness rebuilds the context fresh from the Run Context snapshot every time. That sounds like a limitation. It's the unlock. If a snapshot can reconstruct the agent's whole state, the agent doesn't need to be running to exist. Persistence stops being a model problem and becomes a harness problem.

So we gave the harness a memory that outlives the chat window: watchers. A watcher is anything that delivers an item back onto Scout's radar with enough context attached to continue a hunt — a pattern completing in the live entity graph, fresh intelligence landing on an indicator Scout flagged, a data source coming online that an open hypothesis was waiting for. When a hunt runs out of present-tense evidence, Scout doesn't close the case. The harness compiles the open hypothesis into a watcher and plants it. The watcher does the waiting, evaluating incrementally as events stream in: no polling, no scheduled re-runs, no LLM burning tokens in a loop. A dormant hunt costs almost nothing, because while it sleeps there is no agent, just a watcher listening.

When a watcher fires, in week seven or month nine or whenever the adversary finally moves, the harness does what it does on every turn: it rebuilds Scout from the Run Context with the same hypothesis and plan intact. The hunt doesn't restart. It resumes mid-plan, as if no time had passed. And because watchers are just another thing the harness exposes, Scout creates, retriggers, and retires them itself, escalating to an analyst or planting three sharper watchers and going back to sleep.

Here's what that looks like in practice. An analyst opens a hunt on suspected credential-stuffing staging in week one. The present-tense evidence runs dry by week two — a few anomalous logins, nothing conclusive — so Scout compiles the hypothesis into a watcher and goes quiet. In week nine, a service account authenticates from an ASN that completes the pattern. The watcher fires, the harness rehydrates Scout with the original plan intact, Scout runs the two follow-up queries it had already scoped, confirms the chain, and pages the analyst with the evidence assembled — not a raw alert asking them to remember what they were thinking two months ago.

This breaks the most limiting assumption in today's agents: that the agent only exists while a chat window is open. A conversation-shaped agent can only reason about evidence that already exists. A watcher-armed one reasons about evidence that doesn't exist yet, which is exactly the shape of low-and-slow campaigns built to out-wait human attention spans. Scout measures its attention span in graph events, not coffee.

The economics follow from the mechanism: a dormant hunt is a watcher listening for events, not a process on a machine. It consumes no compute, no tokens, and nobody's attention. Coverage stops scaling with headcount because waiting became free.

Figure 4 — A hunt compiled into a watcher. Dormancy costs nothing because nothing runs; a firing watcher rehydrates Scout mid-plan, and the agent plants its own next wake-ups.

Your documents are evidence too

Everything above concerns structured data: logs, tables, telemetry. A real hunt leans just as hard on knowledge that lives in prose. The incident response plan. The network architecture doc. The contract that defines what "sensitive data" even means for this customer. Last quarter's pen-test report. Today that context lives in the analyst's head or a wiki nobody has opened since onboarding. We're building it into Scout's harness as a first-class knowledge layer.

The design principle matches the one that governs the entity graph: ownership before retrieval. Every piece of knowledge has an owner — you, your organization, or the platform — and Scout can only retrieve from the owners its current principal is allowed to see. The server resolves that scoping from the session; the model never chooses it. An LLM should never be the component deciding what data it's entitled to read.

Three layers share one retrieval surface:

• Personal knowledge. Upload a document and it becomes usable in answers within minutes, visible to you and no one else, ever.
• Organizational knowledge. Admins maintain a shared, governed body of knowledge that every member's agent draws on, with an audit trail of who added and removed what.
• Curated global knowledge. We publish versioned releases of security knowledge to every tenant, with rollback. Everyone gets the same vetted version, and nobody's private data is in it.

Because these are your documents, answers carry source citations. When Scout claims your retention policy says 90 days, it shows you the document and passage it read that in. Grounded means checkable.

Two more properties matter for the customers who care most:

• Isolation as a tier, not a fork. Some customers contractually require hard isolation of their data. They get a structurally dedicated scope, enforced at the platform layer rather than by application-level filters, behind the same product surface everyone else uses. One product, two guarantees, verified by continuous automated cross-tenant leak tests.
• Read-only agents. In v1, agents retrieve knowledge but never write it. The write path belongs to humans. We've designed for agent write-back, but it will arrive behind review-and-promotion controls, not as a default.

The payoff looks like this: "does this finding violate our data-handling agreement with the client?" requires walking the entity graph and reading the contract. Scout answers it in one turn, with citations for both halves.

Figure 5 — The knowledge layer: three ownership tiers, one scoped retrieval surface, and answers that cite their sources.

Loop engineering: we refuse to let it grade its own homework

An agent that plans and executes but never checks itself is an expensive way to generate confident nonsense. Scout runs a Plan → Execute → Judge → Replan cycle, driven by executable Hunt Workbench DAGs. The plan is a real graph of steps, not a vibe.

Around the gate sits ordinary, unglamorous robustness. Scout retries automatically when KQL or SPL comes back with a syntax error, and it replans dynamically when fresh evidence invalidates an assumption the plan was built on. When a branch of the DAG turns out to be wrong, Scout updates the branch instead of stubbornly finishing a plan that reality already contradicted.

A side effect of planning in executable DAGs is that the audit trail writes itself. Every step, retry, and verification verdict is a durable record — when a hunt's conclusion gets questioned six months later, the answer is a replay, not a reconstruction.

Autonomy isn't a single on/off switch either. We give analysts two dials:

• Session Mode: Guided, Supervised, or Autonomous.
• Initiative Level: Low (Ask First), Medium (Balanced), or High (Proactive).

Together they span the range from "explain every step and wait for me" to "go hunt and tell me what you found." The right setting depends on the analyst, the environment, and how much sleep anyone has had.

Figure 6 — The maker/checker loop. The agent that runs the query never decides whether the results are good enough.

The takeaway

Scout's advantage isn't a smarter model. It's a harness that keeps a hunt alive for months at near-zero cost, never lets the agent grade its own homework, and never lets the model decide what data it's entitled to see. The model is the CPU; we built the rest of the machine.

The fastest way to evaluate any of it — the hunt loop, the harness internals, or the isolation guarantees — is to watch Scout run against your environment.