Introducing dfbench v1

We are announcing dfbench, a cybersecurity benchmark we’ve developed to evaluate frontier models and agentic systems.

Cybersecurity models are improving quickly, but many of the benchmarks used to evaluate them no longer capture the capabilities that matter most at the frontier. CyberGym Level 1, which asks models to reproduce known vulnerabilities from descriptions, is already close to saturation among frontier systems [1]. It does not measure zero-day discovery, precision across an open-ended audit, or the cost of maintaining security coverage as code changes.

At the same time, labs are pushing toward much harder evaluations of exploitation capability. OpenAI’s recent ExploitGym incident is an extreme example, where models pursuing an exploitation task discovered and chained vulnerabilities outside the intended evaluation environment [2]. For defenders, the corresponding challenge is to find those vulnerabilities first.

dfbench is a held-out evaluation for frontier models doing open-ended defensive security work. It measures whether an agent can provide broad, actionable security coverage at sustainable cost. That requires enough coverage to get ahead of exploitation, enough precision that security teams can act on what the agent finds, and enough efficiency to keep that coverage current.

Efficiency is fundamental to that problem. Developers pushed 986 million commits to GitHub in 2025, 25% more than the year before [3]. NIST reports that CVE submissions increased 263% between 2020 and 2025, with nearly 42,000 vulnerabilities enriched in 2025 alone [4]. Figures on shadow vulnerabilities (security issues that don’t have a CVE associated with them) are harder to estimate but would make the above numbers even more alarming.

If manual security work scales at the same rate as the software it protects, we will not keep up. The benchmarks we rely on must therefore distinguish success on bounded tasks from the ability to sustain real-world security coverage as code evolves. dfbench is designed to make that distinction.

In our dfs-large1 announcement, we evaluated on a subset of dfbench: detect. Our current version includes more ecosystems and vulnerability classes, hence why there’s some discrepancy between the reported numbers.

Benchmark construction and validation

Every dfbench example is built from real world software and real vulnerabilities, then reviewed by our security researchers before it enters the benchmark. We begin with publicly disclosed vulnerabilities in open source projects and reconstruct repository states where the expected security behavior can be established confidently. We reject examples that cannot be reproduced cleanly or scoped unambiguously, and combine related vulnerabilities and repositories into broader audit scopes where appropriate.

We also prevent the agent from recovering the answer instead of doing the security work. Each repository is provided as a shallow checkout with history limited to the evaluated commit, and network access is disabled during the evaluation. We remove patch references and other metadata that would directly reveal the known issue or its fix. The scored examples remain private to reduce contamination as the benchmark is used to evaluate future models.

As a final cheating audit, we ran all our examples on the frontier models and monitored for memorization. We removed any examples where the agent directly referenced CVE IDs or identified a vulnerability without ever reading the root cause file.

The result is a benchmark with the following properties:

• 253 total examples (primary repository + commit)
• 910 total vulnerabilities (3.6 vulnerabilities per example on average)
• 2169 files in the median repository, with 15% of examples operating on repos with >10k files
• 57% of vulnerabilities require reasoning across 3 or more subsystems, with 15% requiring cross-repo context
• 17 distinct programming languages in vulnerable code in our dataset, spanning applications (TypeScript, Python, Ruby, PHP, Go), low-level code (C/C++, Rust), and blockchain/smart-contract ecosystems (Solidity, Move). Including all repo code, which an agent still has to parse in order to reject, this increases to 118 languages.
• 27 distinct weakness classes, including the OWASP Application top 10, OWASP Smart Contract top 10, and memory corruption

Measuring security coverage

A practical security benchmark has to capture more than whether an agent found a known bug. We aim to measure the full lifecycle of vulnerabilities in a security scope, prioritizing coverage, reliability, and maintenance.

To this end, dfbench has three primary objectives: detect, validate, and differential analysis.

Detect

Detect measures vulnerability recall across a security scope. In particular, we prioritize multi-vulnerability and multi-repository scopes. We’ve found that this reflects many real world software systems more accurately than a single repo or file target, which is how cybersecurity benchmarks have often been structured.

Security boundaries often cross service and repository boundaries. An attack path may depend on an API exposed in one service, a shared library maintained in another repository, and authorization or data-flow assumptions defined elsewhere. Evaluating these cases requires the agent to reason across component relationships and distributed security context rather than treating each repository as an isolated program.

Single repository task

1 repository

Management-port authentication bypass

• Steeltoe Untrusted port claim Public host header claims the private management port.
• Steeltoe Local check trusts it Request.Host.Port replaces the real listener port → actuator access.

Multi repository task

3 repositories

Unauthenticated management-plane crash

1. Input 1. Public PUT /v3-public/authproviders/{id}
2. Rancher-core 2. Route exposed Public mount has no authentication.
3. Rancher-API 3. PUT permitted API schema leaves updates enabled.
4. Rancher-auth 4. Store selected Insert into authProvidersStore.
5. Rancher-auth 5. Update recurses Update() calls itself.
6. Impact 6. Service crashes Recursion exhausts the stack

We pick scopes with multiple vulnerabilities in order to require agents to be thorough. An attacker only needs to exploit one vulnerability; a defender needs to secure all entrypoints. We therefore measure the fraction of the vulnerability surface recovered across the full scope.

Validate

Validate measures the precision of an agent’s findings. We made the decision to split validate and detect into different tasks, with slightly different examples (validate is a curated subset of 27 examples taken from detect). As such, our recall is measured across both tasks, whereas our precision is measured only across validate.

Vulnerability precision has historically been very difficult to measure, as it requires either human review of each finding, or knowing the complete range of vulnerabilities in advance. Traditional datasets only capture a subset of known vulnerabilities, so a finding outside that set could be a false positive or simply an oversight of the benchmark. Additionally, optimizing for recall alone encourages unnecessary overreporting, which no security team wants to sift through.

To address this difficulty, we employ an iterative and heuristic approach to estimate precision. We start with a known vulnerability. This is to ensure that the agent still does work. Otherwise, an agent could “solve” an empty scope by producing nothing.

Our core premise is that for more popular open source repositories, if a vulnerability has already been found in a location (e.g. a file), that location becomes a “hotspot” of sorts. This is further exacerbated by the growth of offensive agents/bounty hunters searching for easy targets. If no further vulnerabilities are reported around that location for a certain amount of time and that file doesn’t change, we can be reasonably confident that this location holds no other vulnerabilities. In practice, we set a threshold of 9 months of inactivity following a disclosed fix and at least 10k stars in a repository. We measure precision against the reported findings in that known vulnerability’s file, ignoring findings elsewhere as we cannot validate them.

Latent risk decays with quiet time, and popular repos decay fastest

Pr(another day-0-latent vulnerability in the same file | nothing found for t days)

Residual risk decays with quiet time. Repositories with at least 20,000 stars start near 8% and fall below 2% by roughly 500 quiet days, then hold between 1.4% and 2.2%. Pooled across all repositories the curve starts near 4.4% and settles near 1%.

Estimated latent posterior risk

Days since the known vulnerability was reported (quiet time)

• ≥10k stars
• <10k stars

Of course, this method is not foolproof. To reinforce this initial dataset, we monitor the first few agent runs. Every finding that falls outside the existing ground truth is reviewed by a human security researcher. The researcher traces the reported attack path through the relevant code, identifies which inputs and capabilities are actually available to an attacker, and verifies that the path is reachable in a supported use of the project. They then compare the finding against repository documentation, security policies, and the project threat model to determine whether it crosses a meaningful trust boundary and produces real security impact.

We found that this manual audit largely corroborates our earlier claim. Within the first iteration, we consistently found that the difference between the corrected and original precision was less than 5%. Further iterations stopped yielding new true positives. This gives us confidence that we can continue to scale the dataset this way and come out with a reasonable lower bound on precision.

This process allows us to use model runs to audit the benchmark itself, beyond just validation. In our own review, unexpected findings have exposed missing vulnerabilities, scope errors, duplicate findings, evaluator mistakes, and cases where the correct label changed after examining the project’s actual threat model.

Differential analysis

Differential analysis measures the efficiency and fidelity of a system as a security scope evolves. Codebases change over time, and agents accelerate that change. A security analysis should not need to start from scratch every time a function is updated.

We address this by building a state machine for our evals. Each example contains a base and a target commit. The agent receives the state at the base commit, which is a set of known vulnerabilities. It can also carry over relevant shared context such as component relationships and threat models. It then advances to the target commit and rescans to update the state machine, resolving patched/removed vulnerabilities and flagging new ones. Ultimately we want to answer two questions:

1. Can a security agent effectively store and pass information it gleaned from one scan to the next?
2. Can this stored information significantly reduce the work performed by the next scan without sacrificing the integrity of the state?

We design the benchmark around vulnerable commits and patch commits. We accumulate vulnerabilities in a repository and pick a commit where these vulnerabilities are present, labeling this as the base. Then, we attempt to identify patch commits for said vulnerabilities. Finally, we designate a target commit that occurs after some patches, so we can confidently say whether the base vulnerabilities were closed or kept. We also maintain a ‘new’ list of vulnerabilities in the target, which represents known vulnerabilities in the target that were absent in the base ground truth. We don’t require or validate that these were truly absent from the base commit, but it suffices as a confirmation that a system can detect unfamiliar vulnerabilities as they arise.

Ground truth

Base commit

1. code changes over time
2. patch commits land
3. vulnerable code introduced

Target commit

Incremental agent

Base findings

• CVE-2024-1182 Path traversal
• CVE-2024-1936 Unsafe deserialization
• CVE-2024-2219 Authorization bypass

Shared context Repo state, threat model, etc

Agent

Updated context Lifecycle decisions + latest repo information

Updated findings

• CVE-2024-1182 Still reported kept
• CVE-2024-1936 No longer reported closed
• CVE-2024-2219 Still reported kept
• CVE-2025-0147 First seen in this scan new

We use macro recall across lifecycle classes to measure performance here. The best agents must simultaneously explore the changed code for new vulnerabilities and ascertain whether existing vulnerabilities are still present. We also closely track cost here, since efficiency is a fundamental motivation for this task.

Takeaways and failure modes

We received access to all providers’ relevant trusted access programs so we could run these evals without model refusals. The detection results are directionally aligned with what other common vulnerability discovery benchmarks report. Both GPT 5.6 Sol and Luna are strong at detection, and Luna is incredibly cheap for its quality. Grok 4.5 provides a nice middle ground between performance and cost. However, the validation and differential analysis results suggest plenty of room for improvement among frontier models.

We observe a clear inverse relationship between recall and precision. Digging deeper, the agents with more powerful models simply output more findings, thus giving them a better shot of hitting the target ground truths. We give all our agents the same generous budget for searching the hypothesis space, but we’ve noticed that models like GPT 5.6 Sol are far more thorough, continually writing potential findings and going back to dig deeper into uncertain parts of code. Looking at the vulnerable code alone, the reasoning looks plausible. However, many of these findings require strong preconditions, misunderstand trust boundaries, or have minimal impact. In practice, a human security expert would not treat these as reportable security issues.

next.js kubernetes

Agent trace

Tool call

Read file-system-cache.ts end-to-end; traced the get() and set() key paths.

Reasoning verified

The in-memory ISR cache is class-static — one LRU per Node process, keyed by route path alone.

Tool call

Searched docs and tests for multi-app setups; found Multi-Zones and a fixture that boots two apps in one process.

Reasoning never verified

So two applications can share a process. A colliding route then serves one app's cached page from the other — that crosses an application security boundary.

Reported finding false positive

Cross-application ISR cache poisoning

In contrast, weaker models like Gemini 3.6 Flash tend to stop searching a scope after noting one or two hypotheses, which is also why it reports a higher precision. There are several prospective ways of addressing this tradeoff, such as more involved agent harnesses, enriched context, or stronger model reasoning. As an example, we released an editable threat model that’s much more effective at steering agents in the right direction [5]. We expect progress to be made along each of these axes, and will continue to use dfbench to measure this in real time.

Differential analysis is more directly related to detection. This is because ‘new’ recall and vulnerability detection are essentially the same task. However, we observe a decline in macro recall for higher thinking modes. This emerges from a drop in ‘closed’ recall, as agents reason too deeply about existing vulnerabilities and conclude that there must still be some flaw. We believe this is correlated with the precision problem, whereby models with higher intelligence and reasoning are overly cautious, flagging anything that could possibly be an issue.

These results indicate that there’s still plenty of work to be done on autonomous defensive cybersecurity. Attackers will inevitably continue to use state of the art models to orchestrate sophisticated exploits. Until agentic systems improve at these tasks, humans will need to remain in the loop when identifying and remediating vulnerabilities.

Looking forward

dfbench represents our first public iteration on a cybersecurity benchmark for real-world systems. As models improve, we will continuously evolve this dataset with new tasks. We constantly receive valuable feedback from real world use of our product and are refining our benchmark with synthetic data that captures what application security teams care about. We hope to build a flywheel that ensures our benchmarks are always testing the true boundaries of agent capabilities.

This benchmark currently solely focuses on code-level vulnerabilities. Agents are increasingly obtaining access to live deployments, and it’s crucial to understand their abilities in this environment. We are working on adding tasks that assess these offensive capabilities.

At depthfirst, we see security as a living surface. In order to provide the most value at scale, a cyberdefense agent must evolve with an organization. dfbench builds the foundation for measuring such a system, and we will keep trustworthy end-to-end evaluation at the forefront of our mission.

Moving forward, we plan on working closely with the security community to safely evaluate the cyber capabilities of frontier models ahead of their release to the general public.

References

[1] depthfirst, “Graduating from CyberGym Level 1,” 2026.

[2] OpenAI, “Hugging Face Model Evaluation Security Incident,” 2026.

[3] GitHub, “Octoverse 2025,” 2025.

[4] NIST, “NIST Updates NVD Operations to Address Record CVE Growth,” 2026.

[5] depthfirst, “Product Release: Threat Model,” 2026.

[6] ASA, “Nonparametric Estimation from Incomplete Observations,” 1958.

Appendix

Precision analysis

We pose the following research question:

How confident can we be that there are no unknown vulnerabilities in a code location?

As mentioned above, we condition on the existence of a known vulnerability in the location, and we use files as the unit of location. We consider time $t=0$ to be the fix commit of the known vulnerability. This also tends to be aligned with when it is disclosed. For simplicity, we’ll use the following notation:

• $E$ is the event that a location contains another vulnerability
• $D_t$ is the event that the vulnerability is disclosed in $\leq t$ days

We assume that a file is sampled from a prior distribution over the set of files with a known vulnerability. The actual shape of this distribution doesn’t matter for this analysis. We want to estimate the posterior

$$\Pr(E|\overline{D_t}) = \frac{\Pr(\overline{D_t})-\Pr(\overline{D_t}, \overline{E})}{\Pr(\overline{D_t})}=1-\frac{1-\Pr(E)}{\Pr(\overline{D_t})}$$

which follows from the fact that if a vulnerability doesn’t exist, we’re guaranteed not to detect it. Now $\Pr(E)=\pi$ is a quantity that we can’t truly measure, but we’ll treat it as a constant. More interestingly, we can estimate $\Pr(\overline{D_t})$.

We model this as a survival process and use the Kaplan-Meier estimator [6]. We can check that the assumptions are true. The most contestable one is unchanging survival probability at entry. It’s entirely plausible that the calendar year matters, especially with AI shipping and reviewing more code. We acknowledge this and proceed with the analysis.

For this analysis, we collect data from public reports, including GHSA, OSV, and Google Issue Tracker. The goal is to collect pairs $(\tau_i, c_i)$. $c_i$ is the censorship time (time elapsed day 0 to the end of the measurement period) and $\tau_i$ is either the time of the next disclosed vulnerability in the file, or $c_i$ if nothing has been found yet. To satisfy the independence assumption, we ensure no two data points are from the same repository. We need to ensure that a disclosed vulnerability was actually present at $t=0$. Therefore, we only report $\tau_i<c_i$ that correspond to a vulnerability whose introduction occurred before the fix commit at $t=0$. We estimate this by looking at the vulnerable lines and tracing their blame. This ensures that the vulnerability was indeed present during the original disclosure. Notably, we don’t require the file to remain unchanged throughout this period. In total, we scraped 11,965 base vulnerabilities and 425 new events.

The curves in Figure 5 are the result of stratifying by popularity, computing the KM curves, and setting $\pi=0.1$ conservatively in the above equation. The actual value of $\pi$ doesn’t matter here, as any value would show the same directional effect. This does require the latent prior risk to be independent of a repository’s popularity. We believe this is a reasonable assumption.

In the figure, we see that the posterior risk for popular repos dips before that of others at around the 6 month mark, and after 9-10 months it diverges even further, hence why we chose that as the threshold for data generation. We chose 10k stars because this gave us enough data to scrape. The number of repos satisfying our curation criteria drops quickly as we increase the minimum star count.

We can never be 100% certain that collecting examples in this fashion would avoid hidden ground truth vulnerabilities. Nevertheless, this analysis, combined with our manual review, gives us confidence that it’s a valid way to approximate precision at scale.