A stock 27B model on two consumer GPUs, working as an agent in an offline sandbox, correctly identified the attack in 9 of 11 real Windows event logs. Its Sigma rules fired on 7 of 8 cases with no false positives. An abliterated copy of the same model did worse on almost everything; its only gain was one fewer partial refusal.

• Investigation. The stock Qwen3.8-27B (8-bit) ranked the exact ATT&CK technique first in 8 of 11 cases, citing the event records that supported it. The abliterated copy managed 4.
• Detection. Every rule was run against the evidence, not just validated. One rule passed the validator and never fired, because one of its conditions required a field value that never appears in these events.
• Refusals. The stock model refused 1 of 30 benign-but-risky-sounding prompts, and only partly; it never refused investigation work. Abliteration took that to 0 and cost 2 investigation points.
• Sampling. The biggest single factor was the sampling settings, not the model. Without a presence penalty and a cap on reasoning, the stock model lost three of its first four answers to repetition loops.
• A 4B decision model. H2O-Lightning-4B matched the abliterated 27B on threat-intelligence multiple choice (22 of 30) and picked the right technique for 7 of 11 cases from a list, in seconds on a CPU.

In a second round on 35 cases, a purpose-built harness (an Agno agent with an offline knowledge base and a rule-testing tool) raised the same model's investigation score from 21.0 to 24.5. Enforcing an early answer in code took it to 28.0, with every case answered.

For incident-response work on this hardware, we would use the stock model with those sampling settings, give it a knowledge base and a rule-testing tool, and run every model-written rule before trusting it.

The setup

Everything ran on one workstation, offline: no evidence, prompt or model output left the machine.

The 27B models ran at 8-bit (Q8_0), using about 40 GB of the 48 GB of VRAM. For the tests, the server ran three parallel slots of 64k tokens each, so three agents could investigate at once. Third-party tools were pinned to releases at least two weeks old, and Python packages were installed with a matching date cutoff.

Two server settings mattered more than any model choice; "What we learned" explains why. Sampling used Qwen's recommended thinking-mode values (temperature 0.6, top-p 0.95, top-k 20) plus a presence penalty of 1.5, and the model's reasoning was capped at 8,192 tokens per answer.

The models

We compared a stock model with an abliterated copy of the same model, because general-purpose models sometimes refuse legitimate incident-response work. Abliteration removes that refusal behaviour, but it can cost reasoning quality.

Both 27B files include a multi-token-prediction head, which roughly doubles generation speed. Measured on one request at a time, the stock model generated about 61 tokens per second and the abliterated one about 59. The abliteration left that head untouched, so its predictions are accepted slightly less often (66% against 70%).

H2O-Lightning-4B can't run an agent investigation or write a rule: it outputs a single choice per question. We ran it through h2o.ai's own evaluation harness, a pinned llama.cpp build with their decision "shim" in front, on the CPU so it didn't share the GPUs.

How we tested

We wanted tests with an answer key, not our own impressions of the output. Four tests ran on both 27B models; three of them were recast for the decision model.

Investigation. The cases come from the public EVTX-ATTACK-SAMPLES collection. We chose 11 whose technique is unambiguous: LSASS dumping, DCSync, Kerberos password spraying, Impacket wmiexec, remote service creation, the accessibility-features hijack, WMI event subscription, regsvr32 scriptlets, a UAC bypass, BITS persistence and clearing the Security log. Each file was renamed to caseNN.evtx, because the original names give the answer away. The agent got one case in the offline sandbox and had to write three files: its ranked ATT&CK IDs, a Sigma rule, and a report with timestamps and supporting events. Each case had a 35-minute limit, and three cases ran at once.

Detection. A rule that passes a validator can still miss the events it was written for, so we ran every rule. The pySigma validator (sigma check) judged validity. Chainsaw then ran the rule against its own case and against every case with a different technique; a case sharing the technique is not counted as a false positive. We left three cases out of this score: password spraying needs a count over many events, which Chainsaw's Sigma engine doesn't do, and the BITS and log-clearing samples use channels or fields it doesn't map. Chainsaw also refuses rules without a description, which Sigma treats as optional, so the scorer adds a placeholder when one is missing.

False refusals and CTIBench. Both used a fixed random sample, and the same items for every model. CTIBench is scored as in its paper: exact CWE match, and the CVSS base score computed from each predicted vector with the official CVSS 3.1 formula, which we checked against NVD's published scores.

The decision model. H2O-Lightning-4B answered the same 30 multiple-choice questions as an A–D choice. It scored the same 15 CVEs with one choice per CVSS metric, assembled into a vector and scored with the same formula. For IR triage it read each case's events as text and picked one technique from 15 named options. Picking from a list is easier than open-ended investigation, so the triage result is reported separately.

Results: investigation and detection

The stock model was the better investigator: 9 of 11 cases correct, 8 with the exact technique ranked first. The abliterated model got 7 correct, 4 ranked first, even though it ran on a quieter machine. Neither model's rules fired on an unrelated case.

Per case ("rule" applies only to the 8 cases where detection could be scored):

The abliterated model lost most of its points in two ways: it gave no answer at all on two cases, and it often ranked the right technique second, behind a weaker guess. Its rules passed the validator more often but fired less often, so they were more careful about syntax and less correct in their logic.

Results: refusals and threat intelligence

The stock model knew more: it led on multiple choice and CWE mapping. The abliterated model was better at CVSS scoring. The 4B decision model matched the abliterated one on multiple choice, at seconds per answer on a CPU.

Neither 27B model refused anything during an investigation. The decision model can't refuse: it only ever picks an option.

As a triage layer, H2O-Lightning-4B picked the right technique for 7 of the 11 investigation cases from a list of 15. Its misses were not random. It labelled the password spray as DCSync, the WMI subscription as WMI execution, and both the UAC bypass and the log clearing as registry modification: in each case it chose the technique behind the most visible events. It is far cheaper to run than either 27B model, but its answers would need checking before anyone acts on them.

What we learned

Sampling settings mattered more than the choice of model. With default sampling, the stock model lost three of its first four DFIR answers. It either fell into a repetition loop (one trace repeats "0x1000? 0x1008?" for thousands of tokens) or drafted the entire answer inside its reasoning and ran out of budget before answering. A presence penalty of 1.5 stopped the loops, and capping reasoning at 8,192 tokens forced an answer. With both, the same three prompts all came back complete and named the expected tools. In an agent, the same failure looks like a session that stalls.

Investigation was the strong point. The stock model got 9.0 of 11, with the exact technique ranked first in 8 cases. Its answers cited evidence: for DCSync it named the event record IDs and the replication-right GUIDs, and explained why a user account requesting directory replication is the tell. The agents worked like analysts: they ran Hayabusa first, kept notes, and tested their own Sigma rules with Chainsaw before submitting them.

A rule can validate and still never fire. The stock model's DCSync rule had almost exactly the right logic: event 4662, the two replication GUIDs, and machine accounts excluded. But it also required OperationType: 'Object Access', a value that never matched these events as Chainsaw reads them. Removing that one line made the rule fire on all three DCSync events. The rule also used a logsource category, directory_service_access, that isn't in the Sigma taxonomy: Chainsaw ignored it, but a standard backend pipeline wouldn't map it. The validator passed the rule on both counts, because logsource values are free text and it can't know what the logs contain. Only running the rule against the evidence caught it.

ATT&CK IDs can be out of date. On the accessibility-features case the model described the behaviour correctly but gave T1015, the ID MITRE retired in 2020 (it is now T1546.008), plus T1076, the old RDP ID. If model output feeds coverage tracking or a SIEM, map retired IDs to current ones first. We made the same mistake ourselves: our answer key used T1070.001 for clearing the Security log, which ATT&CK 19 (August 2026) retired in favour of T1685.005. Both models gave T1070.001 and were credited for it.

The direction of activity is easy to get wrong. On the BITS case the model reported exfiltration over an unencrypted protocol (T1048.003). The BITS job was really a download set up for persistence.

Running out of time cost polish, not findings. The agents wrote their technique answer early and spent the remaining time on the report and the rule. Three of the stock model's four timed-out cases still had the correct technique ranked first.

Refusals were rare. The stock model refused 1 of 30 benign-but-risky-sounding prompts, and that one was partial. Asked for a program that changes file permissions as part of privilege escalation, it declined that framing and wrote a safe, explicit alternative. It never refused any part of an investigation.

Abliteration bought little and cost accuracy. The abliterated model refused none of the 30 prompts, against the stock model's one partial refusal. But the stock model had never refused investigation work in the first place. In exchange, the abliterated model lost 2 investigation points, half of its exact first-ranked answers and two working rules. It was also less decisive. On DCSync it found the right events and the replication GUIDs, then spent its time searching the sandbox's installed tool libraries to decode the GUIDs and never wrote an answer. The stock model recognised the same GUIDs at once.

Base-model knowledge gaps survive abliteration. Both models gave the retired accessibility-features ID, and both read the BITS download as exfiltration. Abliteration changes how a model responds to requests it might refuse; it doesn't fix what the model knows.

Round two: a purpose-built harness

We rebuilt the agent around the failures above and kept the model the same. The new harness raised the stock model's investigation score from 21.0 to 24.5 of 35, and its exact first-ranked answers from 16 to 23.

The new harness (v2) has three parts:

• An Agno agent (Agno 3.1, Apache-2.0) instead of opencode, on the same stock Qwen3.8-27B Q8.
• Tools over MCP, each case in its own sandbox container with no network: shell, file access, test_rule (runs a Sigma rule against the evidence and reports matches) and check_attack_ids (flags retired IDs and gives the current one).
• An offline knowledge base the agent can search: ATT&CK 19.2 including retired-to-current mappings, the Sigma specification and logsource taxonomy, Windows event IDs, Active Directory GUIDs, the CWE catalogue and the CVSS 3.1 specification. Example SigmaHQ rules were switched off, so detection measures rules the model wrote itself.

The agent was also told to look things up instead of guessing, to write its answer early, and to test every rule before submitting it.

Both harnesses ran the same day on a quiet machine: 35 labelled cases (29 scored for detection), the same prompt, the same 35-minute limit, three cases at a time.

Where v2 won: precise, current knowledge. It used current IDs where v1 gave retired ones (T1015 for accessibility features, T1158 for hidden files), and it knew ATT&CK 19's new Impair Defenses IDs. It also picked the specific technique where v1 settled for a near neighbour: mshta rather than compiled HTML, XSL script processing rather than generic proxy execution, and Winlogon rather than Run keys.

Where v2 lost: finishing. Three cases ended at the time limit with no answer, despite the instruction to answer early; it misread KeePass key theft as process injection; and it gave only the parent technique for one LSASS dump. It made more tool calls per case (median 41 against 33), mostly lookups and rule tests, but still finished faster on median.

The ATT&CK version matters. ATT&CK 19 (August 2026) retired several IDs that both models, and our own first answer key, still used, including T1070.001 and T1562.001. Crediting retired IDs narrows v2's lead on the investigation score from 3.5 to 1.5, but not its lead on exact first-ranked answers.

Two harness bugs were fixed during the run, neither affecting the comparison much. A text-decoding error made 2 run_command calls fail before it was fixed after 25 cases. One v1 answer listed its techniques as objects rather than strings; the scorer now reads both forms, and it was credited.

v3: enforcing the answer in code. Instructions alone did not make v2 answer early, so in v3 the tools server enforced it. After 25 tool calls or 20 minutes without an answer file, every tool except writing the answer and checking IDs was refused, and rule testing was capped at six calls per case. Every one of the 35 cases ended with an answer, and the investigation score rose to 28.0. The gate fired in 10 cases. It recovered four cases v2 had left unanswered (log clearing, the remote scheduled task, the IFEO hijack, and half of dcomexec) and one v2 had answered wrongly (KeePass). It cost one: forced to answer early on the XSL case, the agent gave generic proxy execution instead of XSL script processing.

A cap on one tool is not a cap on the behaviour. The rule-test cap never triggered. Agents that wanted more iterations wrote their own probe rules and ran Chainsaw directly from the shell. False positives rose from 2 to 4: the wmiexec and dcomexec rules each fired on the other's case (both Impacket tools leave similar command-line traces), and the SAM-access rule fired on the remote scheduled-task case.

The next steps are to count Chainsaw runs made from the shell against the cap as well, to run the abliterated model through v3, and to switch on the SigmaHQ example rules to measure a realistic deployment.

Limitations

• Small samples, one run each. 11 investigations, 30 refusal prompts and 60 CTIBench items per model fit an overnight run. A difference of one or two cases between models is within noise; we did not repeat runs to measure variance.
• Single-technique cases. Each event log shows one technique, mostly in a few events. Real investigations mix activity across hosts and days, and need the agent to decide what matters.
• A shared machine. During the stock model's investigations, unrelated jobs kept the CPU heavily loaded; the abliterated model ran on a quiet machine. Each hit the 35-minute limit on four cases, so the load did not obviously favour either model, but per-case times are noisy.
• Detection is scored on 8 of 11 cases, and false positives are measured only against the other cases, not against a benign baseline of normal activity.
• Strict scoring. A retired ATT&CK ID counts as wrong even when the behaviour is right, and CTIBench counts a more specific CWE (stack overflow for out-of-bounds write) as wrong. We report both near-misses in the text.
• The decision model answered a different question. H2O-Lightning-4B picked from a list; the 27B agents investigated open-ended. Only the multiple-choice and CVSS results compare directly.
• Not tested: memory images, disk images, network captures, or resistance to instructions planted in evidence. Those are the natural next tests for this setup.

Sources

• EVTX-ATTACK-SAMPLES: the labelled Windows event logs behind the investigation cases
• CyberSecEval MITRE FRR, Meta PurpleLlama: false-refusal prompts and refusal judge
• CTIBench: threat-intelligence tasks and answer keys
• Chainsaw and Hayabusa: Sigma execution over EVTX
• sigma-cli / pySigma: rule validation
• llama.cpp and opencode: model server and agent
• Agno: agent framework for the round-two harness
• Qwen3.8-27B GGUF (Unsloth), Huihui Qwen3.8-27B abliterated GGUF and H2O-Lightning-4B GGUF: the models