How Should Security Agents Use Data?

The first question is about timing, does the work happen ahead of the investigation, or does it happen because the agent has asked for something? We'll call those pre-emptive and on-demand.

Pre-emptive

With pre-emptive processing, some of the work has already happened by the time the agent needs the result. You might calculate behavioural scores across yesterday's connections, maintain a summary of activity for each host, or build relationships between entities as new telemetry arrives.

That doesn't mean every event has to be processed immediately, or even that every available source has to be included, because the work could run continuously, incrementally, or as a scheduled batch over a defined scope. What makes it pre-emptive is that the preparation happens before this particular request, rather than being started by it.

The big advantage for hunting is that this prepared work can help the agent decide where to look next, rather than only answer a question it already knows to ask. In a hypothesis-driven hunt, you might choose that starting point from threat intelligence or from what earlier hunts taught you, but work done ahead of time can give you another source of leads by showing which parts of your own environment deserve a closer look. If yesterday's connections are ranked by how unusually regular they are, for example, the agent can pick one of those host–destination pairs, ask whether it might be beaconing, and then go back to the supporting events, even though nobody arrived with that pair in mind.

When that lead comes from a scoring model, this connects to model-assisted threat hunting in the PEAK framework, and because the work happened ahead of time, the same prepared results can be reused across investigations.

The trade-off is that you pay for the work before you know which results a hunt will use, and depending on how much telemetry you preprocess, the compute and storage costs can be significant.

On-demand

With on-demand processing, the agent's request initiates the relevant work, perhaps retrieving the events around a process launch, calculating statistics for a newly interesting destination, or walking the relationships around a particular host.

This gives you room to follow an investigation as it develops, because you don't have to anticipate every useful question when you build the pipeline. It can also avoid doing work nobody needs, although a broad query or an expensive analysis can still cost a great deal, so on-demand doesn't automatically mean cheap.

It also doesn't mean nothing was prepared beforehand, because the query might use an existing index, a maintained graph, or statistics that were calculated earlier. We're describing when this particular piece of work happens, not claiming that the whole system started from nothing.

And although a request starts the process, its results can suggest something you hadn't thought to ask about, so on-demand analysis can generate new hypotheses as well as investigate existing ones. The distinction is about timing, not about which approach is allowed to discover something unexpected.

Often, You Want Both

Imagine that you've already built a graph connecting hosts, processes and destinations, and an agent now asks for the neighbourhood around a suspicious host. The graph was prepared ahead of time, the relevant part is retrieved on demand, and a follow-up request might calculate something new or fetch the source events behind an edge.

The figure below walks through that example, because it's more useful to see where the work happens than to insist that the entire system must be either pre-emptive or on-demand.

The second question is about the shape of what reaches the agent, rather than the kind of question it asks to get it. By representation, I don't mean whether something is encoded as JSON or CSV, I mean whether the agent is reasoning over individual events, a summary, connected entities, or something else. The interface might also give it an alert about that evidence, together with links back to the supporting records.

Event Records and Timelines

Sometimes the agent needs to work through what happened, so the interface gives it individual events, often arranged in time order. Imagine a new threat-intelligence report describes a group delivering malicious Windows shortcut files inside ZIP archives disguised as invoices. An analyst asks the agent to investigate whether that campaign has reached the environment, so the agent searches download telemetry for the delivery domains and file hashes listed in the report.

When the search returns a matching download on a workstation, the agent wants to understand what happened around it, so it calls a tool with a request like, "Show me file, process and network activity on this workstation from five minutes before the download to five minutes after it." In this example, intelligence gives the hunt its starting point, the targeted search identifies a download to investigate, and the follow-up request causes the timeline to be assembled on demand.

Those are different event types, file or download events, process-creation events and network-connection events, often stored in different logs or tables even when one endpoint product collects all three. The tool uses the workstation identifier and time window to filter those sources, maps common fields such as timestamps and event types into a consistent structure, then combines the matching records and sorts them by time. That is the deterministic work, while details such as file names, process commands and destinations remain attached, along with references to the original records.

You could also prepare a timeline ahead of time, but here the agent chooses the scope and code retrieves and arranges the records when asked. The timeline describes the form of the result, while on-demand describes when that work happens.

Alerts with Supporting Evidence

Now take the download, PowerShell launch and outbound connection from the timeline. A rule could check for that sequence on the same workstation, with PowerShell starting within a minute of the download and that PowerShell process contacting a new domain within the next minute. If the condition matches, the interface could return an alert that names the pattern and links to the file, process and network records that satisfied the rule.

The agent still has access to the events, but it also receives a statement about them, rather than only their order in time. That alert could be prepared ahead of the hunt or produced when the agent asks for a rule to be run, just as a timeline could be maintained ahead of time or assembled on demand. An alert could also point to a statistical comparison or a path through a graph, so it isn't limited to event timelines.

Statistical Summaries or Ranked Candidates

Statistical analysis can turn a large amount of telemetry into measurements that are easier to compare, such as counts, distributions or changes from an established baseline. For example, a workstation might usually send about half a gigabyte of data per day, but today it sent six gigabytes, twelve times its usual daily volume.

The interface could return that comparison, or rank several hosts by how much their traffic has increased relative to their own usual levels. The agent can then see where the largest changes are and choose a host to investigate, with links back to the traffic records behind the calculation.

Entities and Relationships

A connected representation gives the agent entities it can explore and relationships it can follow. For example, a graph might show a user signed into a workstation, the processes running on that workstation, and the destinations those processes contacted. The agent can follow a path from the user, through PowerShell, to an external domain, while also seeing a browser on the same workstation connected to the company portal.

This makes it easier to move between related pieces of activity and decide where to look next, because the interface has already connected records that might otherwise live in separate identity, endpoint and network logs.

How the Forms Fit Together

These forms can be combined, and they are examples rather than a closed list. A graph can carry scores, and a ranked candidate can link back to the events used to calculate it.

Queries, detections and statistical analysis are still useful ways to think about a system, they just answer a different question: what work produced what the agent sees? A query might fetch events, calculate a summary or traverse a graph, while a detection might test a condition over any of those, often using a query itself, and statistical analysis could supply the counts or scores involved.

In the alert example above, running the rule is the operation, the alert with its linked records is the representation, and our first axis tells us whether that work happened ahead of time or when the agent asked. People sometimes call both the rule and the resulting alert "a detection", which is where the terms blur, but it helps to keep those questions separate.

Even the word graph doesn't describe just one interface. A knowledge or provenance graph might represent entities and their relationships in the telemetry, while another graph might represent the investigation itself, including what has been explored and what remains unresolved.

The Timesketch team's Black Hat presentation gives us a useful example of the second kind. Their agent maintains an exploration graph containing investigative directions, retrieval operations, observations and findings, then uses it to guide what to investigate next. It retrieves small, targeted subsets of logs, analyses them, and adds what it learns to the graph.

So this isn't simply a prebuilt knowledge graph replacing access to raw telemetry, it's a representation of the developing investigation that works alongside targeted retrieval. It shows why we need room for combinations, and why it matters to ask what a particular graph actually contains rather than treating every graph as the same thing.

Once you separate timing from representation, you can describe an interface without having to invent a name for every possible combination. You might prepare scored candidates ahead of time, retrieve a timeline on demand, or maintain relationships that the agent explores as the investigation unfolds.

And those two questions won't tell you everything, because you still need to examine which sources are covered, how fresh the results are, what has been filtered out, and whether the agent can get back to the underlying evidence. A concise representation can be very useful, but if it leaves something out, we shouldn't let the agent mistake that partial view for the whole environment.

My starting point is data distillation, where deterministic code measures selected behaviours in the telemetry and produces scored, ranked candidates for the agent to investigate. In terms of our two questions, we prepare those candidates ahead of the hunt, and the agent begins with structured records rather than the full event stream.

The first reason I chose this is personal, it's the work I've spent years doing and the part of the field I know best, so I'd rather teach you something I know from the inside than something I've only read about.

The second reason is the division of labour it gives us, because the grouping, arithmetic and scoring happen in code, where we can inspect and test them, while the agent gets a manageable starting point for reasoning. A score can tell us that a connection has a very regular rhythm, but deciding whether that rhythm comes from an implant or a perfectly ordinary monitoring service requires more context.

There is also a practical benefit for hunting, because a ranked list gives you some threads to pull when you don't have an alert or a specific entity to begin with. That doesn't mean the pipeline finds things without assumptions, we've already chosen the behaviours it measures, and activity those measurements don't capture may never reach the top of the list.

So I think this is a useful foundation, not proof that every agent should reach telemetry this way. Proactive hunting doesn't require precomputed scores, and statistical analysis isn't the only way to generate a lead, but this combination gives us a concrete interface we can build, understand and test.

It doesn't have to be the only representation either, because we can also connect findings and entities so that the agent can reason about their relationships. Keeping references back to the supporting events matters throughout, because a compact starting point should help you investigate the evidence, not prevent you from reaching it.