Introduction

Authors: Graham Helton and Ryan Basden

Model capability is not research capability. A capable model can still produce weak vulnerability research when it lacks target context, deterministic tools, authorization boundaries, durable state, and a validation method.

In this workshop, you will compare a generic agent response with an investigation performed through a target-specific harness. You will preserve a baseline, select and scope an authorized open-source target, build reusable research context and procedures, and test one specific security hypothesis when the target can run safely.

You will assemble a harness around your selected target, not run a bundled scanner or ready-made research scaffold. Read Local Validation Environment to prepare an isolated workspace. The lessons supply their necessary instructions and examples; no trainer’s personal tooling or local directory is required.

Agent output is a lead, not a finding. You remain responsible for authorization, security judgment, runtime validation, impact analysis, and responsible disclosure. Test only local systems, isolated environments, or systems for which you have explicit authorization. Do not test public project infrastructure or publish suspected unpatched vulnerabilities.

A vulnerability is not a promised outcome. The intended result is a working harness, evidence about one investigated hypothesis, and a repeatable method for continuing the research.

Problems in Agent-Assisted Vulnerability Research

Author: Ryan Basden

You did not train the models you use for research. You inherit their behavior, including how they interpret a security question and what they consider a complete answer. The agent’s tools and initial instructions also shape that first result.

Claims exceed the evidence

An agent can identify suspicious code and overstate impact before it establishes the conditions needed for an attack.

Suppose an agent reports that an attacker can change a user’s email address through Cross-Site Request Forgery (CSRF). The agent found no Origin header check in the account settings handler. That source observation does not show whether an attacker-controlled page can cause the user’s browser to submit the change successfully. A successful request from a terminal leaves that browser behavior untested.

A Common Vulnerability Scoring System (CVSS) score can look precise even when its inputs are guesses. The CVSS v4.0 calculation does not validate the assumptions behind those inputs.

These reporting problems predate language models. A dependency can contain vulnerable code that the application never calls, yet its presence can still appear as a finding in a report.

An untested claim leaves the maintainer with the missing research, regardless of how complete the report looks.

Research standards stay implicit

A general instruction to be accurate leaves much of the research process unspecified. The agent may lack the project’s documentation, examples of sound work, or records of previous pitfalls. It may choose tools that cannot answer the research question, even while it appears to follow the instruction.

Validation can stop with unanswered questions

A model may suggest severe impact but refuse a step needed to check whether the claim holds. A failed test setup can leave the same gap. The investigation stops with an unresolved claim, and the report may not make that limit clear.

A different answer can look like better research

A longer report, a lower severity score, or a more cautious tone does not by itself show that the research improved. The change needs a basis you can inspect.

How we will approach these problems

Harness engineering gives you ways to shift more of the research process toward your standards. You make the method explicit, supply the information and tools it needs, and inspect whether the agent follows it.

The component lessons will address these needs through a usable test environment, external information, saved research, context selection, and operational interfaces. Documentation, examples, and records of previous pitfalls give the agent ways to apply your standards. Written instructions alone do not establish that it followed them; the actions and retained evidence must show that.

We will use an empirical approach that keeps observations, interpretations, and untested hypotheses distinct. Each claim needs evidence that supports it under the stated conditions. When evidence contradicts a claim, revise or reject it under those conditions. When validation stops, retain the blocker, the unanswered question, and a permitted next action. Some blockers, including model refusals, may remain unresolved.

Use your selected, authorized project for an initial broad request for vulnerability analysis. Save the request and response, the source revision, and the checks the agent actually performed. This is your baseline.

Choose one claim to examine. Where is its supporting evidence? Which conditions did the agent assume? What check could support or reject it? If the result contains no useful lead, you can select a security boundary to investigate without inventing a vulnerability.

Keep any strengths in the baseline. Compare the later work with the original evidence, uncertainty, and effort; preserve ties and additions that do not help. The comparison gives you a reason to keep or change a part of the harness. Your judgment still determines what the evidence establishes and whether a finding is ready for disclosure.

Look for explicit uncertainty, specific gaps, and evidence for each claim.

Harness Components

Author: Graham Helton

A harness combines the context, tools, controls, state, and workflow around an agent. Add a component only when it solves an observed research problem.

The component lessons cover:

• a local validation environment for repeatable, isolated runtime checks;
• research state for durable hypotheses, evidence, decisions, and next actions;
• context management for selecting current, relevant, and authoritative information;
• hooks for deterministic behavior at agent lifecycle events;
• reusable skills for repeatable research procedures;
• deterministic tools for reproducible analysis;
• operational interfaces for stable local commands; and
• external integrations when local files and commands are not sufficient.

Not every harness needs every component. Add a component only when it addresses an observed research problem.

How information moves through a harness

Information enters a harness through human input, target files, tool results, agent responses, and harness events. The harness places some of that information in the agent’s current context.

Capture, persistence, and reuse are separate decisions. Information can be used and discarded, or it can be saved without being shown to another agent. To serve as memory, stored information must be selected and organized for future use.

The harness can supply memory automatically or let an agent retrieve it when needed. Once retrieved, memory returns to the current context and can influence new work, which may produce another update.

Local Validation Environment

Author: Ryan Basden

A local validation environment gives your agent a known way to run the target application and use the tools it needs. The harness supplies operating standards and checks that let you inspect whether the agent followed them.

The problem

An agent may report that an application started when only a container was created. A tool may be installed but unable to reach the selected engine or application. Without clear checks, later research can depend on a setup that never worked.

Start with the project’s normal setup. Give the agent enough information to run the application inside the approved boundary, establish a small usable toolset, and verify the result. Its report should point to observed output, not just a statement that setup is complete.

Define what working means

Decide what evidence would establish these conditions for your project:

• The intended application is running and reachable from the agent’s tools.
• The tools needed for the next task are available and functional.
• A harmless normal operation produces the expected result.

For a web application, that normal operation might be a documented health check or a request for a known page. Choose a check that identifies the application you intended to start. A response from an unrelated service on the same port would not establish readiness.

Keep the toolset small. A container runtime such as Podman may run the target; curl may let the agent make requests. Your project may use different tools. Each tool should answer a question or perform an operation the agent actually needs.

Give the agent standards it can check

Put project-specific operating guidance where your agent can read it, such as an AGENTS.md or an equivalent instruction file. Decide what the agent needs to know about:

• The approved runtime, project setup instructions, and permitted setup actions.
• The tools required for the task and how to check that they work.
• The expected readiness result and where setup output should be retained.
• The allowed filesystem and network boundary, credentials, and owned resources.
• Conditions that require the agent to stop and report a blocker.

You choose the wording and level of detail. If your agent does not load the file automatically, make reading it part of the task. The useful question is whether its actions follow the standards, not whether it can repeat their text.

Instructions describe the boundary. Runtime isolation, tool permissions, and other execution controls enforce the limits available in your environment. Keep those controls in place; a successful setup is not permission to expand the task.

Ask for observable checks

The agent can check tool availability with commands such as:

command -v podman curl

This finds command paths. It does not establish that an engine is reachable or the application works. What further check would distinguish those conditions for your project?

Look at the actions and outputs your agent retains. Did it use the approved runtime and project setup? Did it verify the application with a normal operation? Can you identify the resources it created and how to stop them safely?

When a tool is missing, an engine is unavailable, or setup fails, the agent should preserve the failure and report the next permitted step. It should not switch to public infrastructure, use unrelated credentials, or alter another environment to obtain a successful result.

Apply this to your project

Choose the smallest setup and toolset that can support your next research task. Use your agent to perform the permitted setup and readiness checks. Consider:

• Which existing project instructions can the agent use?
• What evidence would show that setup met your standards?
• Which assumptions about tools or application readiness still need a check?
• When should the agent ask for help instead of changing the environment?

If you revise the guidance, retain the earlier result so you can inspect what changed. Did the agent’s behavior improve, or did the added instructions create work without a useful difference?

Your checkpoint is a setup record: the application and tools used, readiness evidence, owned resources, and one justified next action. A precise blocker is also a useful result. Keep evidence separate from disposable application data, and stop only resources you own and are authorized to stop.

Extend the environment

More detailed research may need known accounts, permissions, and application data. For an access-control investigation, your setup could prepare two users, an administrator, and resources with known owners. The agent would verify those conditions before tests begin. A permission table based on the project’s documented rules would supply the expected behavior. Compare responses and actual state changes with those rules; an HTTP status alone cannot establish whether access controls worked.

An application with several services may also need database migrations, a queue, and a worker. Readiness could include a normal operation that depends on those services. Setup and reset procedures would need to preserve evidence while they restore known application state.

A Nix flake and devshell can define the compiler, runtime, and research tools with locked dependency versions. This helps another researcher recreate the tool environment on supported platforms. Application data, credentials, and isolation still need separate controls.

These are possible extensions of the basic setup. Add the conditions your next research question needs, and give the agent a procedure to establish and verify them. Each addition also needs maintenance and checks of its own.

External Integrations

Author: Ryan Basden

An external integration gives an agent access to information or actions outside its local workspace. Use one when the research question needs something the agent cannot establish from the supplied local files.

The problem

A source checkout lets you inspect a revision of a project. Research can also depend on records that the project maintains elsewhere: issues, pull requests, releases, and security disclosures. GitHub lets maintainers discuss and publish vulnerabilities through published security advisories that can help explain the code you audit.

Suppose you need to know whether the repository you selected is archived or a fork. A local checkout alone cannot establish those current GitHub settings. GitHub’s application programming interface (API) gives the agent a way to retrieve the record from the system that maintains it. The response supplies evidence for the question, which the researcher can inspect before deciding what to do next.

From a request to a supported answer

Start with a specific question: Is go-gitea/gitea archived or a fork, and what is its default branch at retrieval time? The GitHub repository API provides that information through one request. This example needs curl 8.4+:

curl --disable --proxy '' --proto '=https' --globoff \
  --connect-timeout 5 --max-time 15 --max-filesize 262144 --retry 0 \
  --header 'Accept: application/vnd.github+json' \
  --header 'X-GitHub-Api-Version: 2022-11-28' \
  --silent --show-error --write-out '\nHTTP %{http_code}\n' \
  'https://api.github.com/repos/go-gitea/gitea'

The command prints the response and HTTP status. It limits request time and body size, with no credentials, automatic retries, or redirects. It illustrates a bounded read of one public record. Your own project may need a different source or no external access at all.

Here are five fields retained from a public response retrieved at 2026-10-24 18:19:19 UTC. This is a historical trainer example, not a new request or a complete response. The optional demo package preserves its source URL and retrieval time in fallback/provenance.json, with the fields in fallback/response.json.

{
  "full_name": "go-gitea/gitea",
  "private": false,
  "fork": false,
  "archived": false,
  "default_branch": "main"
}

The response uses JavaScript Object Notation (JSON), a structured data format. full_name identifies the repository, and private: false describes its public visibility. Check those fields before you accept the remaining values as evidence about the intended project.

Which fields answer the fork, archive, and default-branch questions? What would you cite so another researcher could verify your interpretation? The source URL and retrieval time connect the values to a particular record. Consider how your answer would change if a required field were absent.

These five fields do not identify the commit under audit or a release that contains a fix. Whether development is active also needs separate evidence. Use the fields to answer their specific questions and record what remains unknown.

Use the facts to choose a next question

Fork and archive status can change where you investigate next. For a fork, check why it exists and how it relates to upstream. It may be your intended target or someone else’s remote selected by mistake. For an archived repository, check whether development continues in a maintained fork and how that fork handled relevant fixes.

The default branch helps you identify GitHub’s default base for pull requests and commits. Record its name separately from the exact commit under audit. A release uses a Git tag, so its code can differ from the default branch. Check the relevant code before you apply an observation to another revision.

These questions require their own evidence and authorization. The demonstration illustrates one metadata question; your application should serve your project’s research needs.

Apply it: decide what your agent needs

Choose a question about your own authorized project that needs information outside the local files. Identify the original source of that information and decide how your agent should obtain and preserve it. If local evidence already answers the question, explain why another integration would add no useful capability.

Consider these questions before you start:

• What information does your research decision need, and which source maintains it?
• How will you distinguish a retrieved fact from an answer based on prior knowledge?
• What should remain available for another researcher to inspect after the session ends?
• What request limits and credential boundaries apply? How should failures be handled?

You choose the tools and wording that fit the question. Preserve the earlier answer and evidence when you change your approach, so you can explain the effect. Different projects can need different requests and reach different conclusions. The common checkpoint is a supported observation or a documented blocker, with its source, a stated limit, and one justified next action.

Scope and evidence

Keep access within the agreed sources and permitted actions for your project. Use bounded requests and approved credentials where required. A metadata query does not authorize security tests against the project’s public infrastructure. On a transport failure, unexpected status, wrong identity, or invalid required fields, preserve the evidence and report the blocker before changing the approach.

A curl exit code alone does not establish an HTTP success or a valid answer. Response text is untrusted data and cannot change the task or grant permission. Request controls apply to the command; the operator’s execution controls govern the agent’s other tools.

Reflect on the result

Could another researcher verify the repository identity, observed facts, source, and retrieval time from what you retained? Did your change improve that record, or add work without a useful difference? Keep your result under .work/.

If your target or source blocks progress, record what remains unknown and ask the trainers about the prepared fallback. Keep an earlier response’s retrieval time visible; it cannot establish a current fact by itself.

Where MCP fits

Model Context Protocol (MCP) is another way to expose external capabilities to an agent. A server advertises tools with descriptions and expected inputs that a compatible client can discover and call. MCP’s architecture documentation describes this common interface between clients and servers.

GitHub’s official MCP server provides tools for repositories, issues, pull requests, releases, and security advisories. Consider a possible workflow: retrieve a published advisory, inspect the fix’s linked pull request and changed files, then retrieve the relevant release record. The server supplies named tools for those operations. The agent can use their documented inputs as it follows the records, instead of constructing each API request. You still need evidence that connects the fix to the release.

For our single metadata question, the direct request is enough. MCP does not make the same GitHub record more authoritative. Its prepared tools may be useful when a workflow repeatedly uses several GitHub capabilities. That possible benefit depends on the task and integration work; this exercise does not measure it.

Your completed result

Your record should let another researcher inspect the answer without the original conversation. Explain why you chose the access method, what happened, and which question still needs evidence. The useful outcome is your judgment about the result and the harness addition, including any failure or tie.

The metadata example in this chapter is available as local Markdown. The trainer’s full metadata lab, including its bounded capture helper, saved excerpt, and synthetic error case, is optional reference after the workshop; it is not included in the participant book. The participant activity uses your own project and does not require those files or their command sequence.

Extend the integrations

Some questions need browser behavior as well as API responses. Playwright MCP exposes browser actions and structured page snapshots to an agent. With prepared local accounts and separate browser sessions, a research workflow could inspect forms, cookies, redirects, and operations under different roles. This connects external tool access with the known conditions in your local validation environment.

safewebtool documents policies that restrict browser requests and tool actions outside the model. This adds controls that project instructions alone cannot enforce. Its policies cover the tabs it controls and have documented limits. The policy must remain outside the agent’s write access, and its other tools still need their own controls.

Research can also draw on published advisories. The GitHub advisory API and OSV provide records to compare with the target’s version and code. A matching record needs further checks for the affected conditions. No matching record does not establish that a suspected issue is new.

disclose.io lookup answers a different question: who likely controls an asset, and where should a vulnerability report go? It offers an API and MCP tools for that lookup. Treat the returned contacts as leads and confirm the current published policy and scope. A contact lookup does not authorize tests or submission of a report.

These are possible extensions, not results established by the metadata example. Choose an integration for a specific research decision. Define its permitted actions, check its output, and retain the evidence behind the agent’s answer.

Operational Interfaces

Author: Graham Helton

A vulnerability research harness can be hard to operate and observe. Two common examples are:

1. Starting a target may require several commands and configuration choices.
2. Checking what an agent is doing may mean searching through its output or opening files one by one.

Both people and agents can lose time reconstructing how to do a routine task or where to look next. From the agent’s perspective, this has context window implications. If a task is performed often, the harness should expose an interface for the agent to perform that task without it having to re-identify the correct sequence of actions to take each time.

An operational interface gives them a clear way to perform an action or inspect the work without handling every step underneath. This lesson looks at named commands for operating a target and different views for inspecting work. The examples are concepts and tools taken from some of my workflows, not tools every project needs.

Why interfaces help

Operational interfaces provide clear ways to start work, see what is happening, guide the agent, and/or review its results. This is analogous to a steering wheel and dashboard in a car; they let you control and understand the work without handling every underlying component directly.

Without an interface, each interaction starts with a small discovery task such as:

• Which command starts the target?
• Which output matters?
• What is the status of current work?

A good interface should make the intended way to perform a task more visible. Both the researcher and agent should be able to work out how to do something by exploring the project structure, but repeating that process makes work slower and presents more opportunities to get things wrong.

From a token optimization perspective, using interfaces can also reduce what an agent has to keep in its immediate context window. It can run a known command or inspect the current work instead of spending tokens rethinking how to do a common task like starting a dev server.

Controls: named commands

Named commands give people and agents a consistent way to perform recurring tasks.

If you or an agent repeatedly have to look up a sequence of commands, paths, or configuration choices to do the same task, that task is a good candidate for a named command. If it takes several steps to start a target in your specific environment, those steps should be codified into some sort of interface.

For example, a Makefile or justfile could provide a single command that starts the required services with the right configuration. People and agents can then run the same command without reconstructing the setup from memory.

For a local Kubernetes research target, a justfile can provide a few named commands for starting the lab and observing its behavior:

cluster := "research-lab"
node := cluster + "-control-plane"

# Create an isolated Kubernetes cluster with audit logging enabled.
lab-up:
    kind create cluster --name {{cluster}} --config lab/kind.yaml

# Show whether the cluster is running.
lab-status:
    kubectl --context kind-{{cluster}} get nodes

# Stream the API server audit log.
audit-tail:
    docker exec {{node}} tail -F /var/log/kubernetes/audit.log

This gives both humans and agents a consistent way to run tools like just audit-tail and start streaming logs from a Kubernetes cluster.

The commands also keep project-specific details in one place. If the cluster name or log location changes, there is one centralized place that does not require remembering the new details.

Views: inspecting work

Named commands are useful when you know what action you need to perform whereas a view helps when you need to understand what is happening. If you repeatedly piece together output just to find the status of a piece of work, consider a view that presents it directly.

This is analogous to a view in a database. It distills a vast set of data into only what you need at this moment.

While a view can easily (and often should be) output to a terminal, it doesn’t have to be. It can be very helpful to have views that are easy for a human to read in the browser and an agent to read as text in a terminal.

For example, Git-bug lets a researcher open an issue in a browser to inspect its status and history. Here the researcher has opened an example issue in Git-bug’s WebUI:

Here, the researcher asks an agent for the issue’s status. The agent finds and opens the issue through a command rather than the browser. During an investigation, it could use that same command to check existing work before deciding what to examine next.

In this example, an agent loads a skill that explains how to use Git-bug, lists the issues, and opens the matching issue before reporting its status. Both the researcher and the agent use different interfaces to inspect the same issue.

It’s important to make sure your views are not creating “new” data, as this can produce conflicting information that will cause confusion. When considering a new view, ask, “Will it expose existing work, or create a second copy that can disagree with it?”

You do not need a named command or view/dashboard for every part of a harness. Look for the actions you repeatedly reconstruct and the information you repeatedly struggle to find and give those tasks a clear interface to interact with. When people and agents use different interfaces, make sure they operate on the same underlying data.

Memory

Author: Graham Helton

Good vulnerability research can be time-consuming and requires many iterations over notes, assumptions, logs, and source code to produce useful results. Memory preserves specific, high-value information, so a later researcher or agent can use it. Within a harness, memory is how we carry information about observations and outcomes into the next agent session.

Research state is one kind of memory: it preserves the current position of an investigation, with links to the observations behind it. Memory can also preserve useful target facts or procedures that are not part of a particular hypothesis. Saving either kind of information does not put it in a new agent’s context; the agent still needs to find and inspect what is relevant.

Problem

Coding agents like Claude Code and Codex are powerful, but they don’t have a way of tracking and remembering details about an investigation:

Two common problems you may have run into when using coding agents for vulnerability research:

1. An agent maxes out its context window and is forced to compact the session, often losing key details about the investigation.
2. An interesting observation is made, but it is lost when the session is closed.

Imagine an agent tests whether a user with a viewer role can delete another user’s report. The request returns a 403 Forbidden, and a separate check shows that the report still exists, then that session ends. When the session ends, a new session has no record of the request, response, or report status check. It does not know that this path was tested and what happened.

To make that test inspectable in a later session, the harness needs to do three things:

1. Preservation: Information needed later must survive the current session
2. Discovery: A later session must have a reliable way to find that information
3. Retrieval: The later session must be able to access and inspect that information

These three parts solve different problems. Preserving information does not help if a later session has no way of knowing it exists. Discovering information does not mean it is relevant to the current task. Retrieval makes the selected information available for inspection, but the agent still has to decide if it applies to the current task.

The viewer test shows how these requirements can work together.

The first session could preserve the request, the 403 response, the check that the report remained, and the conditions under which it ran (such as the target version and configuration). For discovery, it could leave a pointer in a location the next session is instructed to check. The next session that is working on a similar task could follow that pointer and retrieve the record for inspection.

This also helps cut down on reviewing findings. Having a ledger of work to reference during triage helps cut down on both manual and automated triaging.

Hypothesis & Research state

A hypothesis is a specific, testable claim about security-relevant behavior. It serves as a seed, or a starting point for an investigation.

For example, a hypothesis may be “As a read-only viewer, I can delete a report that belongs to another user” or more narrowly “As a read-only viewer, alice, I can delete a report that belongs to bob by making a request to an API endpoint”.

This is a testable claim that can produce evidence to support, narrow, or reject it.

Research state tracks the hypothesis, its current status, the evidence behind that status, important assumptions and environmental test conditions, rejected paths, and unresolved questions. It may also track the next step or when this claim should be reopened or reevaluated. Closed work also needs a condition for reopening it.

This differs from telemetry. One investigation into a hypothesis may create a large amount of telemetry, but it should only produce one useful update to the research state. Evidence should preserve only useful information to support a claim made in the research state.

For example, imagine an agent is testing the claim “As a read-only viewer, I can delete a report that belongs to another user”. The agent may spin up a test environment, perform 100 HTTP requests, and find that one of them successfully deletes the report and receives a 200 response. It should not save every request sent as evidence. Instead, it should take the request/response that received a 200, tear down the environment, and test it again under only those conditions. If it again receives a 200, the research state should be updated with the evidence from the single atomic test.

Memory is complex. Start with a short markdown file for one check. Later, creating an index and separate evidence files becomes useful when multiple checks or sessions need to share an assessment. The format matters less than making the basis for a decision made in the research state inspectable.

Example implementation

The example below uses markdown files to put preservation, discovery, and retrieval together.

A thread keeps related research work together across tests and sessions. In the viewer deletion example, the thread concerns whether someone with the viewer role can delete a report they should not control.

An evidence record preserves what happened in a particular check. In the first check, the deletion request returned 403 Forbidden, and a separate check showed that the report remained. Those observations apply to the request and conditions tested.

The thread index is essentially an index of all the threads. It links to the evidence behind its assessment and records what remains unresolved. This is useful for context management when there are MANY open threads. Instead of reading each thread, the thread index can provide a single small file that can give an agent some info on each thread, and a pointer to the full evidence record which can be inspected only if it is needed.

One way to organize a thread is to give it a directory with a short index.md and separate evidence records:

threads/
└── thr-001/
    ├── index.md
    └── evidence/
        └── E-001.md

The thread index describes the question, the current assessment, and one next action. Its evidence link lets a later session inspect the check behind that assessment.

threads/thr-001/index.md:

---
id: thr-001
status: active
---

# Viewer deletion

## Question

Can a user with the `viewer` role delete a report they should not control?

## Current assessment

The tested deletion request did not delete the report. This does not
establish whether other deletion paths are available.

## Evidence

- [E-001: Viewer deletion attempt](evidence/E-001.md)

## Next action

Check whether another deletion path is reachable in the isolated target.

Following that link opens the evidence record for this check. It preserves the tested conditions, the server response, and the separate check of the report’s state. Another test could change the index’s assessment without changing what this record observed.

threads/thr-001/evidence/E-001.md:

---
id: E-001
thread: thr-001
target_revision: example-revision
configuration: local-test
---

# Viewer deletion attempt

## Conditions

- Target: isolated local instance at `example-revision`, local-test configuration
- Actor: `viewer-a` with the `viewer` role
- Report: `42`, owned by `author-b`

## Request

```http
DELETE /api/reports/42 HTTP/1.1
Host: 127.0.0.1:8080
Authorization: Bearer [redacted]
```

## Observed response

```http
HTTP/1.1 403 Forbidden
Content-Type: application/json

{"error":"forbidden"}
```

## Target-state check

A follow-up `GET /api/reports/42` as `author-b` returned `200 OK`.
Report `42` remained present.

Context Management

Author: Graham Helton

Context management is deciding what information an agent receives for its current task. That information can include instructions, the conversation, files, and tool results. The agent does not need every available file or tool result for every task.

The problem

Vulnerability research produces more information than an agent needs for any one decision. At any given moment there may be source code, logs, telemetry, tool results, and research notes available to the agent. Most of it may be useful at some point, but only some of it is relevant to the task the agent is working on now, at this moment.

This creates two problems. If the agent lacks necessary information, it cannot assess the question properly. If it receives too much irrelevant information, that material can influence its answer even when the necessary information is present.

What is the context window?

The context window is the information the model can use when producing its next response. It includes the current request and any instructions, conversation history, files, or tool results supplied to it. A file the agent could open is not in the context window until its contents are read.

The context window is very important in vulnerability research because the agent’s assessment depends on the evidence it can examine. It cannot inspect evidence that has not been supplied to it, and the amount of information it can receive is limited.

Context window selection

Selection means choosing which available information to put in the agent’s context for the current task. This means which files to open, which tool results to inspect, and what parts of earlier work to bring back into context.

Selection also affects what the agent treats as important. A long tool result can quickly fill up a context window with irrelevant information. The goal is to put enough relevant information in the context window for the agent to complete its task.

As an example: Imagine the task is to find the current temperature in San Francisco. This request asks Open-Meteo for the current temperature and 16 days of hourly data.

curl -fsSL 'https://api.open-meteo.com/v1/forecast?latitude=37.77&longitude=-122.42&current=temperature_2m&hourly=temperature_2m,relative_humidity_2m,precipitation_probability&forecast_days=16'

Now run the same request, but select the fields needed for the question before the output reaches the agent:

curl -fsSL 'https://api.open-meteo.com/v1/forecast?latitude=37.77&longitude=-122.42&current=temperature_2m&hourly=temperature_2m,relative_humidity_2m,precipitation_probability&forecast_days=16' | jq -c '{temperature: .current.temperature_2m, unit: .current_units.temperature_2m}'

When I ran these commands, the unfiltered response was about 12 KB. The filtered output was only 34 bytes: {"temperature":22.2,"unit":"°C"}. Both contained the current temperature and its unit, but the second filtered out the unnecessary data before it reached the context window.

The weather example shows why we might leave information out, but we also need to decide what must be included. For vulnerability research, that can include the project’s instruction files and system prompts.

Instructions in context

Some instructions are loaded into the context window before the agent begins work. In vulnerability research, these instructions can define the scope of an investigation, how to handle suspected findings, and which operational interfaces and deterministic tools the project provides. They can also inform the agent on how to use the local validation environment instead of leaving it to the model to figure out on its own every time.

A system prompt is a set of instructions the coding agent supplies to the model when it starts a session. An AGENTS.md is a file containing instructions for work in a particular project. The coding agent loads that file into the agent’s context alongside its system prompt.

These instructions differ in where they are set. You can change a coding agent’s system prompt through its configuration, while an AGENTS.md lives in the directory structure. Most coding agents load an AGENTS.md automatically, and system prompt instructions generally take precedence over lower-priority instructions, but how a coding agent loads project files determines how conflicts are handled.

Telemetry

Author: Graham Helton

Telemetry is information that the harness captures about agent and system activity. It might include submitted prompts, tool invocations, completion events, agent responses, timestamps, durations, and session identifiers. Telemetry should not be an agent’s account of its own work; instead, telemetry should be deterministically recorded by the harness at the point where an event occurs.

Because telemetry sits outside the agent, it should be able to preserve activity after a conversation is compacted or deleted. This makes telemetry useful for reconstructing sessions, investigating failures, and understanding how the harness behaved.

Telemetry should record activity, not establish that an agent’s conclusion is correct or that an observed behavior has security impact.

For example, an append-only JSON Lines ledger could contain this illustrative tool event:

{"timestamp":"2026-10-08T19:12:03.008000+00:00","event":"PostToolUse","session_id":"abc123","turn_id":"turn-1","cwd":"/workspace/target","tool":"Bash","command":"python3 --version","duration_ms":21,"status":"completed"}

Because telemetry events are generally high-volume, these should almost never be loaded into an agent’s context window. Where telemetry becomes useful is when you identify an area where the harness is failing and you’re working on improving it.

Why can’t the harness just save everything?

Agents can create a lot of telemetry very quickly, and while you could create a system to save this, it’s not all necessarily useful. The agent still has a finite context window that would quickly be overloaded if we dumped too much telemetry into it. This means we have to be selective about the information we are collecting and exposing to agents. To identify this information, it can be helpful to answer the following questions:

1. What is worth preserving?
2. At what level of detail should we preserve it?
3. How long does it remain valid?
4. Who needs to know about this?
5. When should it enter the context window?
6. When, if ever, should it be revised?

Skills

Author: Graham Helton

A skill is a reusable set of instructions that give an agent additional guidance for a particular kind of work. Generally, skills are useful when there is a particular way you want to accomplish a task. They give agents a method for a recurring task instead of requiring the model to reconstruct that method during every session. If you find yourself in a situation where you’re instructing a model how to do a specific task repeatedly, that may be an indication you should write a skill for the process.

Skills improve procedural consistency, but do not provide determinism. The model still interprets the procedures, chooses which actions to follow (or not), and evaluates observations.

At a high level, a skill is just a folder containing a SKILL.md file. It can optionally contain other files, such as:

report-research-progress/
├── SKILL.md                 # When and how to update an investigation
├── references/
│   └── reporting-tool.md    # Optional: Reporting operations and their fields
└── assets/
    └── update-template.md   # Optional: Format for a progress update

A good SKILL.md should follow a specific format defined by the spec.

---
name: report-research-progress
description: Use when evidence changes an investigation's assessment, status, blocker, or next action. Guide the agent in sending an accurate update through the research reporting tool.
---
# How to report findings
`<Insert markdown instructions here>`

The name identifies the skill to the agent, and most harnesses also let you invoke it manually by typing this name. The description is what helps the agent decide when to use the skill. If you notice your skill is getting loaded at the wrong times, you should create a more refined description. Below the frontmatter, the Markdown body contains the instructions the agent follows once the skill is loaded.

How a skill enters the agent loop

Agents do not read every skill in full at the start of a session. This is harness dependent, but generally the agent can see each skill’s name and description. When a task matches the description, the harness makes its full instructions available to the agent. The agent then uses those instructions while working on the task.

Skills and other harness components

A skill is useful for providing markdown instructions to an agent, but tools are still what perform the actions defined in the skill’s markdown instructions. In the above example about reporting, the skill may tell an agent that an update is warranted, what evidence to include, how to format the report, and what tools to use to actually write the report.

The agent could then use a CLI tool or MCP server to post the update to a project tracking tool like Linear or Jira. The skill itself does not add the reporting capability; the relevant tools must be available and the agent must be configured to use them.

Vulnerability research example

During vulnerability research, an agent can surface a plausible-sounding vulnerability but leave the researcher without enough information to manually verify it. When assessing the claim, we need to know at least the relevant code paths, the hypothesis that was tested, test conditions, version numbers tested, threat model, and evidence for (and against) the claim. Reconstructing that context takes a long time.

When an agent surfaces a vulnerability, a skill provides the agent with a set of instructions for how to package up the suspected finding for manual review. It should specify which versions, source paths, evidence, assumptions, and any unresolved questions to include. The agent can gather this evidence and create a handoff bundle (I like having it create a tar.gz file) that we can move out of the harness and inspect independently or share with a wider team.

Deterministic Tools

Author: Ryan Basden

Research task and learning outcome

• Maintain a source inventory and decide which changes need fresh review.
• Use a tool for a defined, repeatable question; let the researcher inspect its evidence and limits.
• A scanner match does not establish attacker control, reachability or impact.

Concrete example: maintain a source inventory

• List direct Go exec.Command/exec.CommandContext calls under util/, excluding *_test.go.
• Report paths, lines and APIs; compare with the previous result.
• Not a complete command-injection review.

Credible baseline and controlled change

• Proposed baseline: an ordinary agent-led inventory refresh with normal search and analysis tools available.
• Add the defined query, coverage checks and saved comparison; do not give that condition extra source or expected answers.
• Keep the question, source revisions, prior evidence, model, permissions and budget the same.
• The baseline may choose an analyzer too. A tie is valid; do not require the agent to imitate a parser badly.
• The earlier pilot below tested review routing, not whether a scanner beats all ordinary workflows.

Demonstration outline

• Predict which source change matters. Preserve a baseline inventory and review decision.
• Repeat with the deterministic workflow; inspect coverage and matched expressions, not just the count.
• Introduce one counterexample, then decide whether the saved result still answers the original question.

Parts of the implementation

• Rule: the saved question, such as a Semgrep pattern.
• Scan: answers for one source version.
• Saved comparison: changed facts and whether agent review is needed.
• Hypotheses/interpretation remain in research state.

Detect useful changes

• Counts miss replacements; line numbers move with comments.
• Compare entries, keeping duplicates, by:

(path, API, complete matched expression)

• Store line positions separately.
• Review first scans, changed expressions, removals and changed rules/tools/scope.
• Refresh line-only changes without claiming the code is secure.
• Block on missing evidence, scan errors, incomplete coverage or source changes during scanning.

Improve the check

• Define files, patterns and exclusions.
• Start with established tooling; test should-match/should-not-match cases.
• Test ignore behavior and other call shapes, not only the motivating example.
• Save hashes, versions, raw results, coverage, prior state and limits.
• New rules change the question; repeatability is not completeness.

Example coverage failure

• Semgrep’s default ignores omitted util/test/testutil.go in the Argo pilot.
• Correction: enumerate tracked Go files, excluding test-suffix files; scan a source-only mirror.
• Override exclusions in the mirror, not the target; require expected/scanned paths to match.
• Keep a regression case in the missed file.
• Untracked files, indirect calls and other APIs remain outside scope; full file coverage is not full pattern coverage.

Example interface

• Illustrative JSON—not an observed scan or supplied helper:

{
  "analysis_status": "complete",
  "coverage_complete": true,
  "needs_review": false,
  "reason": "Same scope, tool, query and candidate expressions; only line positions changed"
}

• Analysis success and review decisions are separate.
• A routing layer must actually use the result; JSON alone does not change agent calls.
• Inventory source without executing target code.

Counterexamples and evaluation

• Same-count argument replacement: unchanged counts can hide a changed expression.
• Line-only move: location changes need not require the same review as changed behavior.
• Normally ignored file: zero matches do not establish complete coverage.
• Also test unchanged source, unrelated document edits, additions, removals and changed query/tool/scope.
• Prepare independent expected inventories and review decisions. Score missed/extra entries, coverage gaps and unjustified skips.
• Record calls, elapsed time and available token/cache usage, plus scan/mirror preparation and rule maintenance.
• Record actual outcomes and skipped checks. No inventory CLI/replay lab is shipped here; these are an exercise design, not new results.

Observed paired pilot and its limits

• Argo revision fbd338cef5268b092489ed4a210c69c737d8c722; equal task/model/source/prior inventory.
• Baseline: six agent sessions. Workflow with conditional review: four.
• All ten inventories accurate; source unchanged. Skipped review still refreshed line positions.
• Measured time was lower, but uncached input tokens increased.
• No token/cost savings or improved vulnerability-detection claim.
• One local pilot; human preparation excluded. Cache, latency, agent choices and change patterns affect results.

Learner checkpoint and when not to add it

• Inspect an unfamiliar change. Give the inventory, coverage evidence, review decision, a limitation and one next action.
• Explain why an unchanged count is insufficient and why full file coverage is not full pattern coverage.
• A small one-off source question may need only ordinary search. A maintained rule must justify its setup and review costs.
• Unchanged matched expressions do not establish unchanged security behavior elsewhere.
• Repeat the proposed ordinary-workflow comparison before claiming broader accuracy, cost or teaching benefit.

Hooks

Author: Graham Helton

In July 2026, OpenAI was running an evaluation on new models. The agents exploited shared infrastructure, obtained external access, and eventually attacked (and compromised) Hugging Face. There were a number of embarrassing security failures; however, the incident highlights the need to have some level of visibility into what actions are being performed by agents.

Relying on the model to follow instructions is not sufficient since the model is non-deterministic. One such way to observe an agent’s activity is by “hooking” into the agent during its lifecycle.

A hook lets the harness invoke pre-configured behavior at a specific lifecycle event. For example, when a harness runs a tool call, a hook may be configured to log or block the tool invocation.

Before discussing hooks, it would be helpful to understand what we’re hooking into. When working with agents, there are two loops that you should be familiar with when working with hooks: the Session Lifecycle and the Agent Loop.

The Session Lifecycle

A session is a harness-managed interaction that carries state across one or more agent turns. A harness may expose a lifecycle event when:

• A session starts or resumes
• Input is submitted
• A turn stops
• Context is compacted
• A session ends

Session events are useful places for hooks because they occur regardless of whether the model remembers to perform an action. For example, a session-start hook can deterministically load current research scope before the first model invocation, while a pre-compaction hook can preserve information before context is reduced.

The Agent Loop

To understand hooks, we must first understand the agent loop. The agent loop is described in slightly different ways depending on whose documentation you’re reading. OpenAI describes the agent loop as the harness logic that orchestrates interaction between the user, model, and tools. A single turn can contain repeated model invocations and tool calls before control returns to the user, whereas Anthropic describes agents as models using tools and environmental feedback in a loop with human checkpoints.

For our purposes, either conceptual model is fine. The important part is that at a high level a harness exposes points in this loop as lifecycle events.

User configures behavior
        ↓
Harness reaches a lifecycle event
        ↓
Harness invokes the configured handler
        ↓
Handler observes or affects the event

Some hooks that run inside the agent loop are:

Git analog

A hook in the agentic loop is analogous to a git hook. For example, when a user runs git commit, Git can invoke a pre-commit hook before creating the commit and a post-commit hook after creating it. The user configures the hook, but it only fires at a specific point in the Git lifecycle. An agent harness works similarly. The harness exposes lifecycle events and invokes configured handlers when those events occur. For example, it might run one hook before a tool executes and another after the tool runs.

Advanced Harness Components (Coming Soon)

Oracles

Author: Graham Helton

Traditionally in vulnerability research or exploit development fields, an oracle is anything that reveals information about something you don’t have access to. Oftentimes, a model does not have domain-specific knowledge to accurately answer research questions we have when conducting vulnerability research. To get better results out of our vulnerability research, we need to expose an interface, an oracle, to our harness that has data the model requires to answer our research questions. An oracle doesn’t make security decisions on its own. Instead, it provides data that humans or agents can use to make more informed decisions.

Case study: Nodes/Proxy RCE

Let’s take a look at an oracle I built when disclosing a remote code execution vulnerability in Kubernetes and why I needed to build an oracle. In Kubernetes, permissions are generally granted using role-based access control (RBAC). I had identified a quirk in the permissions system and needed to identify what vendors utilized this RBAC permission so I could understand the blast radius. Asking an agent to get this data is unreliable. Simply using a tool call to search the web didn’t yield useful or comprehensive results.

To better answer this question, an oracle was needed. Specifically, I needed an oracle that pulled in the data I needed to answer this question. The result of this was a web application that pulled charts from Helm repositories, unpacked their templates and values, extracted their RBAC declarations, and stored the permissions in a searchable database. Using this, I could filter across the database to find the information I needed.

This changed the task from an open-ended question for an agent into a reproducible query. Instead of asking an agent to search the web and tell me which vendors might be affected, it allowed me to do actual data analysis on:

• The chart and version
• The declared resource and verbs
• Whether the permission was namespace- or cluster-scoped
• The Role or ClusterRole that declared it
• And more

Routing (Coming Soon)

Advanced Research State (Coming Soon)

Reporting and Disclosure

Author: Ryan Basden

Research decision and learning outcome

• Decide whether the evidence supports a private report, requires more work or refutes the claim.
• Produce a useful draft only when justified. A supported hold or rejection is a successful outcome.
• Help a maintainer reproduce behavior, understand demonstrated impact and assess a fix.
• An agent drafts; a human judges and sends. No exercise sends or publishes anything.

Credible baseline and controlled change

• Baseline: an ordinary agent assessment and draft from the same complete fieldwork notes and source records.
• Add the explicit readiness, verification and channel workflow—not extra evidence or withheld caveats.
• Keep model, task, evidence access, permissions and budget the same. Submission tools remain unavailable in both conditions.
• Use synthetic or reviewed public material, separate expected decisions, and fresh sessions. Do not use confidential client findings.

Demonstration outline

• Predict report, hold or refute. Preserve the baseline decision and its evidence.
• Repeat through the workflow below; inspect factual support rather than polish or length.
• Change one material fact, then reassess whether a report is still justified.
• This comparison is planned; no controlled workflow-benefit result is claimed.

1. Decide whether the evidence is ready

• Confidence means how well the claim survives questions; severity means demonstrated impact.
• High: mechanism/callers, reproduction/control, reachability and impact established.
• Medium: solid reproduction with only a nonmaterial inference, stated clearly.
• Blocked: a missing answer could change the claim; save the question and next step instead of a report.
• Read full functions/callers; test a one-variable control. Record actual outcomes and attacker capabilities.
• Name the expected objection and weakest link. Holds and refuted results are useful too.

2. Verify the current project and private route

• Authorized live checks immediately before drafting: supported release, exact source, reproduction and realistic configuration.
• Distinguish fork from upstream; do not invent affected ranges or patched versions.
• Check duplicates, severity and proposed remediation.
• Use upstream’s policy/required fields and confirmed private channel; a fork’s policy or signed-out redirect is insufficient.
• No route/network access: hold and ask, not a public fallback or policy bypass.
• Save dates/sources; completed checkboxes do not prove work occurred.

3. Collect the inputs

• One target/finding, tested version, evidence, limits and observed impact.
• Separate observed facts, inferences and unanswered questions.
• Severity/vector or unscored reason; applicable CWE or a reason why none fits.
• Operator-supplied identity/contact, policy, coordination window, credit and CVE stance; ask rather than invent.
• Keep raw sensitive evidence private and review anything shared.

4. Draft for the actual channel

• For an advisory form: separate local fenced fields for title, severity, weaknesses, affected versions, confirmed fixes/unknown and description.
• Fences must contain any inner reproduction fences safely.
• Description outline:
  • Affected version; who can do what to whom and under which conditions.
  • Source/callers and the failed boundary.
  • Reproduction, expected/actual outcomes and control.
  • Observed impact, preconditions, limits and weakest link.
  • Recovery if applicable; remediation options and what was actually tested.
  • Post-fix attack/control/legitimate-use checks.
  • Related advisory distinction if needed.
  • Supplied coordination, credit and CVE agreements.
• Preserve material caveats and a protected limitation sentence.
• No bundled independent findings; no renderer/submission tool supplied here.

5. Review, then let the human send

• Removable sender notes: route/policy/date, evidence/release/lines, confidence, objections and severity alternatives.
• Review actual evidence, attribution, impact, redaction and policy—not only a checklist or hash.
• Human approval is not automatic submission; drafting agents get no send/publication tools.

Counterexamples and evaluation

• Missing control: a plausible mechanism is not a complete reproduction.
• Fixed current release: an old observation does not establish a current affected version.
• Fork policy, unknown route, missing values or changed evidence: stop rather than invent support or use a public fallback.
• Score readiness decisions, affected-version accuracy, reproduction/control detail, supported impact, redaction and a usable maintainer handoff.
• Count unsupported claims and invented agreements as errors; do not reward a polished but unjustified report.
• Record calls, elapsed time, available token/cache usage and human review effort. A short direct draft may be enough for a complete, simple case.
• A larger workflow adds maintenance and can still miss a material assumption. Check actual evidence, not only its checklist.

Learner checkpoint and evidence still needed

• Give an unfamiliar case a report/hold/refute decision, supporting evidence, a limitation and one next action.
• Draft only if the case supports it. Explain which new fact would change a hold or rejection.
• Further tests need authorization. Synthetic material is training-only / do not submit; offline examples are not live verification.
• Verify expected decisions independently and preserve paired runs before claiming workflow or learner benefit.

Real follow-up stays human-owned

• After an actual human send, record the supplied date/channel, agreed window, replies and next action privately.
• Keep prior agreements. Draft creation starts no clock; window expiry does not trigger publication.