What I Learned Putting Language Models on Offensive Security Work

A few months ago I got curious enough to spend real time on a question I kept hearing at talks and on timelines. Can a language model actually do offensive security work, or does it just talk about attacks well?

I started the way most people start. I opened ChatGPT and described a lab host. The answers sounded competent. Named attacks, named tools, a plausible next step. Then I tried to work a box with it and the wheels came off. Early GPT models refused a lot of the offensive requests outright. When they did play along, they had a general picture of how attacks work and almost no picture of how a human pentester proceeds. That second part is what you actually learn in offensive security trainings and a few hundred Hack The Box machines: the order of work, the dead ends you stop feeding, the one check that tells you a path is real.

I wanted to know if I could give a model that methodology. So I built a harness around it. Then I noticed the harness was still missing the notes I had been writing for years. Then I had to keep the whole thing cheap enough to leave running. This is the story of those three problems, and the piece I am putting in the open.

The model had the attacks and none of the method

The first weeks were mostly frustration. ChatGPT could list Kerberoasting, LLMNR poisoning, and a dozen web bugs. It could not tell me which of those belonged on the host in front of me. It would jump to an exploit because the service name and the exploit lived near each other in training data. A human who has sat through offensive security trainings does something slower and ruder: they ask what they already have, what they can see, and what would actually change the state of the target.

That gap is the whole job. Offensive security is a small set of facts and a large set of judgment calls. Which tool, why now, what the output means, when a scanner is being optimistic, when to stop and talk to a human. None of that is in the man page. It lives in the practitioner's head and in messy notes.

Once I could run Codex on my own machine, the refusals eased and the methodology problem got louder. A local agent will happily run commands. Without a phase structure it will run the wrong ones with great confidence. So I stopped asking the model to be a pentester. I started asking it to follow one.

A harness is how you give a model a spine

A harness, in this context, is the layer around the model that decides what happens next. Inputs, tools, prompts, which model to call, what state to keep, when to halt. Andy Gill put this cleanly in Harnessing Harnesses: people argue about prompts and model names, and the gains sit in the orchestration.

I already wrote about other people's harnesses here. RAPTOR, Audit, Visa's VVAH, Anthropic's reference pipeline. Those are mostly source-audit machines. I needed something that could sit on a live authorized target and still behave like an engagement.

What the work has to look like

• Reconnaissance first, and deliberately shallow until something earns a deeper look.
• Observations written down as typed facts, not a chat log the next stage has to re-read.
• A matching technique stays a hypothesis until target-specific evidence supports it.
• Anything that mutates the host waits for me.
• A finding requires evidence that survives a second look. A scanner hit is an observation.

The private harness I run for Red Hound does that. Authorization is a package, not a disclaimer. Scope and window get checked before a command fires. Cheap models classify and summarize. Strong models take the hard calls. I get the checkpoint when the work would leave the current phase or change the target.

Read it left to right. I sit on a browser console. An orchestrator owns the run: which worker gets the next capsule, what is already true on the situation board, when a report should be written. Workers are the models. They do not get a shell on the target. They call tools through a gateway that re-checks scope, then the command lands on a Kali VM running HexStrike. That is the only attack path. When a sample needs reversing or a disk image needs a timeline, the orchestrator sends that work to local REMnux, SIFT, or FLARE VMs instead of mixing it into the live attack session.

The surprising part, at least to me, was how much stronger the same model became once the work had a shape. GPT-5.6 Sol on a nasty lab path is genuinely good. The same model with no phase, no evidence rule, and no stop condition produces a confident tour of the internet's average pentest blog. The model did not get smarter. The job got specified.

The notes were the other half of the brain

Even with a methodology, the harness was still starting every session cold. I have years of notes from HTB, offensive security trainings, client-shaped labs, and the little tricks you only write down after they waste an afternoon. Which check tells you Kerberoasting is even on the table. Why this banner is a waste of time. What "interesting" looks like on a quiet Windows box.

A model cannot acquire that on its own. It was never written in a form it could load.

I added a corpus the harness can search: public methodology on one side, private entries on the other. An entry is small. Situation, what to do, and one decisive test that separates a real path from a dead end. After a lab, I file what I learned. After the next lab, the model can find it.

That loop is the part I care about most. The methodology tells the model how to work. The notes tell it what I have already paid to learn. Together they compound. One lab at a time.

I will not publish those notes. They include engagement-shaped lessons and I have no interest in handing a client's estate to GitHub. The public repo ships the schema and an empty entries directory on purpose. Shared methodology. Private tradecraft.

Frontier models do not know this week's exploit

Methodology and old notes still leave a hole. A frontier model is trained on a cutoff. It can talk fluently about last year's Kerberos paper and miss the zero-day that landed on Monday. I watched capable models invent a patch status, skip a KEV entry, or reach for an exploit family that died two versions ago. That is a context problem, and I did not want the operator to paste advisories into the chat every time a banner looked familiar.

Every week I pull the latest CISA KEV catalog and public exploit research into a knowledge base. That corpus is exposed to the harness as an MCP the workers can query the way they query any other tool. Once the harness fingerprints a piece of code - a package, a banner, a version - it asks for current CVEs and known exploits against that exact thing. The answer is this week's intel, not whatever the model happened to memorize at training time.

The rule I keep is the same one I use for scanner output. A hit from the knowledge base is a hypothesis. It becomes a finding only after the harness sees evidence on the authorized target. The MCP exists to close the recency gap. It does not get to declare the box owned.

Six hundred fifty dollars, spent as subscriptions

The next problem was money. I ran the same kind of work through API keys first, including paid OpenRouter routes, so I would have a real number instead of a feeling. It gets expensive very fast. A long authorized session with a frontier model, plus a few parallel workers, plus the retries you always pay for when a tool call comes back noisy, will burn more in a week than a month of subscriptions. OpenRouter made the metering honest. It also made the conclusion obvious.

I switched the harness to spend subscription seats and treat API tokens as overflow. OpenAI, Anthropic, and xAI cover the daily load. OpenRouter stays for cheap throughput when a worker does not need a frontier brain. The stack that actually stays on looks like this:

• GPT-5.6 Sol for the hard challenge. Long reasoning, ugly paths, the box that will not yield.
• Claude Opus to orchestrate. Break the work up, keep state honest, decide what deserves the expensive brain.
• Grok 4.5 to take tasks off the critical path.
• GPT-5.6 Luna for anything that does not need deep reasoning. Parsing, classifying, summarizing, drafting the next check.
• DeepSeek V4 Flash through OpenRouter when I need still cheaper throughput.

The practical advice is about where those workers run. Configure the harness so workers sit on Codex, the Claude CLI, the Grok / OpenCode path, or Hermes, and let those CLIs consume the subscription you already pay for. Driving the same models through raw API keys is how the bill leaves the building. That stack, left on, costs me about $650 a month. No surprise invoice. The models can keep doing authorized lab work or vulnerability research while I read the output and tighten the harness. I learn as much from watching them fail as from watching them land.

Keep the context window on what is true now

The last piece is the one that decides whether a long engagement stays coherent or turns into amnesia with a high token bill. A naive agent resends the whole conversation on every turn. After a few hours that transcript is mostly noise: raw scan dumps, failed rabbit holes, the same banner parsed four times. The model starts forgetting the credential you stole on Tuesday and re-testing the hypothesis you already killed on Wednesday.

The whole run is stored in local logs. Every tool dump, every failed attempt, every raw banner. That archive is for me, and for a later report. It does not get stuffed back into the model. What the workers see is the situation board: the current attack picture, the open hypotheses, what was tested and worked, and what was tested and did not. Each worker is fed that slice as needed. The next one does not receive the novel. It receives the state.

What stays in the capsule

• Where we are: phase, target, foothold, and the last action that changed anything.
• What is true: hosts, services, credentials, and evidence ids, written as typed facts.
• What already worked, so a later worker can reuse it on a newly found machine.
• What already failed, classified tightly enough that a delivery failure does not retire a whole attack family.

That last point matters more than it sounds. Early on I let a single "failed" mark kill a technique for the rest of the run. Plenty of those failures were a mistyped path or a listener that was not up. The ledger now distinguishes "the primitive is dead" from "that attempt did not land." Workers stop repeating the dead ones. They are still allowed to retry a family when the reason was mundane.

This is how you run a long session against a large authorized environment without losing the plot. A hash dumped on day one is still sitting on the board when a new workstation shows up on day three. Two medium findings stay visible long enough for someone - me, or a later worker - to notice they chain. The model never needed the full chat history to make that connection. It needed the relevant facts kept warm, and the irrelevant ones kept out of the window.

What I put on GitHub

The private harness stays private. It is how Red Hound runs AI-assisted pentest and research, and it carries too much of the practice to ship.

What I did publish is arsenal. It is the methodology layer, written as Agent Skills so Claude Code, Codex, Cursor, and anything else that reads a SKILL.md can load it. The point is not another tool list. The point is that a capable model stops improvising from an average of the internet and starts working the way a practitioner would: phase first, tool second, evidence before a finding, stop when a human should decide.

Methodology: how to reason about the phase

Sixteen specialist skills sit under skills/. Each one teaches how to think about a slice of the work, not which flags to type. security-engagement is the front door. It states the objective, the phase, the target, and the next action, then routes to the smallest useful set. From there the specialists cover reconnaissance, network discovery, web applications, Active Directory, credentials, Linux and Windows privilege escalation, post-exploitation, binary exploitation, exploit development, source review, research, evidence, environment and route selection, and cleanup.

One workflow skill, single-target, runs a single authorized host from connectivity through reporting and cleanup, pulling those specialists in as each phase needs them. Underneath are 11 methodology chapters, 8 OSINT guides, and 50 recipe cards. The cards name known technique families and frame them as hypotheses to weigh against evidence. They are never an answer to apply and never permission to act.

Tool depth: how to operate the instrument

Seventy-six tool skills live under knowledge/tools/. Each one is a full operator reference - typically around 450 lines - covering what the tool is for, how it fails, how it chains to the tools around it, and which ATT&CK techniques it serves. They span recon, web, Active Directory, Kerberos, credentials, lateral movement, privilege escalation, C2, evasion, cloud, wireless, reverse engineering, malware analysis, DFIR, and OSINT. nmap, bloodhound, impacket, netexec, hashcat, ghidra, sliver, volatility3, and sixty-eight more.

Those tool files load only when that tool is the instrument. The engagement does not dump 76 manuals into the window on turn one. Recon loads security-reconnaissance. When a scan is actually the next move, the nmap skill arrives with the tradeoffs. That split is the value: methodology decides whether to scan; the tool skill decides how.

Governance stays in the corpus

Four policies cover authorization, operator checkpoints, sensitive-conversation handling, and recovery. They tell the agent where to stop, what evidence a finding requires, and how credential material is handled. I wanted that in the same repo as the skills. A model that knows Kerberoasting and does not know when to ask is not useful on a real engagement.

Install it, then talk to the agent the way you already talk to a junior:

claude plugin marketplace add jperezduerto/arsenal
claude plugin install arsenal@arsenal

# Then, in the session:
# This is an authorized lab engagement. Start from security-engagement.
# Target: <the host you authorized>. Do not exploit anything until I approve.

Without arsenal, a capable model runs a wide nmap, sees 445, and reaches for EternalBlue because SMB and exploits are neighbors in its training data. With it, the engagement skill speaks first. It states the objective, the phase, the target, and the next action. Recon stays shallow until something earns a deeper look. A matching recipe stays a hypothesis. Exploitation waits for you.

That is the transfer I actually wanted. Not a smarter model. A model that can load how practitioners work.

Clone it, then write the first entry yourself

If you already point agents at authorized labs, install arsenal this week and start from security-engagement. Give it a real scope, a real target, and a hard stop on anything that mutates the host. Watch where the model still improvises. That gap is your first private entry.

If you want this run against your environment by people who already operate it this way, that is the offensive work Red Hound sells. Price a test on the penetration testing estimator, or book a session and we will talk scope.