General AI vs. Security Focused Code Reviews

AI code review has improved rapidly. General-purpose PR bots are increasingly effective at catching bugs, suggesting cleaner implementations, identifying missing tests, and helping developers move faster.

At depthfirst, we run Codex alongside the depthfirst PR Security Reviewer. This gives us a direct comparison between general-purpose code review and a system built specifically for security.

In one of our internal test repositories, the depthfirst Security Reviewer found 869 security issues compared with 3519 total issues raised by Codex, out of which 123 were security-related. This is 7x as many security issues found, with depthfirst’s true-positive rate greater than 70%.

General PR bots are starting to include security checks, but security review requires more than asking a general code reviewer to just look for vulnerabilities. It requires security-specific application context, understanding the threat model, intel on its dependency vulnerabilities, tools for investigating attack paths, and models optimized for vulnerability discovery.

We think about this across five areas:

1. Functional code can still be vulnerable. Security review has to consider attacker-controlled inputs, authorization boundaries, permissions, and ways legitimate functionality can be abused.
2. Application context matters. Vulnerabilities often involve code outside the PR, another service, or infrastructure permissions. Understanding the broader application can materially change the review.
3. Dependency review requires vulnerability intelligence. A package change can introduce known vulnerabilities, as well as vulnerabilities discovered before an advisory even exists (i.e. shadow vulnerabilities).
4. Reachability and exploitability determine severity. Finding suspicious code is only the beginning. The reviewer needs to determine whether an attacker can reach it and what happens if they do.
5. Security-specific model training improves the underlying reasoning. Models trained for vulnerability discovery become better at following trust boundaries, recognizing attack patterns, and reasoning through exploitability.

Functional code can still be vulnerable

Consider an endpoint that takes a conversation ID and returns a participant’s username. The code can be simple and work exactly as intended. 

A general code reviewer might check the types, error handling, tests, and whether the implementation follows existing patterns. Those are useful checks. A security review has to ask a different set of questions.

Whether this is a vulnerability depends on the application. In a group chat, participants may be expected to see each other’s usernames. In an anonymous conversation, returning the same information could reveal someone’s identity. Checking that the caller belongs to the conversation would not be enough.

The reviewer needs to understand what information participants are allowed to access and which identities should remain private. It then needs to follow the code to check whether those assumptions hold across functions and services.

This is why the threat model matters. The same code can be safe in one application and vulnerable in another.

Application context changes the quality of the review

Security issues often depend on information outside the pull request itself.

A small change may interact with code elsewhere in the repository, call into another repository, inherit an authorization assumption from another component, or become dangerous only when several otherwise safe behaviors are chained together.

The reviewer therefore needs more than the diff. It needs to understand the application’s architecture, trust boundaries, attacker-controlled inputs, sensitive assets, and security assumptions.

depthfirst builds and iterates a threat model of the application that captures this context. The PR Security Reviewer uses it to reason about security implications that may not be visible in the changed files alone.

That threat model evolves as the system learns more about the application through additional context, code reviews and even feedback from developers. This can include a new trust boundary, sensitive data flow, or other security assumptions, giving future reviews a better understanding of where meaningful attack paths may exist.

Dependency review requires vulnerability intelligence

Third-party dependencies introduce another layer of context. When a pull request adds or upgrades a package, depthfirst checks it against public CVEs and advisories, along with vulnerabilities we have discovered that may not yet be public.

We refer to these as shadow vulnerabilities. There can be a meaningful delay between when a vulnerability is discovered and when a CVE or advisory is published, leaving a period where public databases do not yet reflect the risk. depthfirst analyzes third-party packages directly for these issues, and that vulnerability intelligence is available to the Security Reviewer when it evaluates dependency changes.

Across usage of the depthfirst Security Reviewer, roughly 5% of the pull requests we reviewed introduced verified vulnerable third-party code that was actually expoitable.

A dependency change may look routine in a diff. Assessing its security impact requires knowing what changed, whether vulnerable functionality is present, and whether the application can actually reach it.

Reachability and exploitability matter

Finding suspicious code is only the beginning. The security reviewer needs to determine whether an attacker can reach it and what happens if they do. That means tracing attacker-controlled input through application logic, service boundaries, validation, authorization checks, and permissions to the vulnerable behavior.

Reachability can change the assessment entirely. Suspicious code may not be accessible to an attacker, while seemingly harmless code can become exploitable because of how attacker-controlled data reaches it, which trust boundaries it crosses, or what permissions are available along the path.

The analysis often extends beyond application code. IAM policies, Terraform, Kubernetes configuration, authentication middleware, API routes, CI/CD configuration, and dependency changes can all affect whether a vulnerable path can be reached and what an attacker can do once they get there. 

Reviewing these pieces independently can miss relationships that determine whether a vulnerability is exploitable. The Security reviewer therefore needs to look at the whole picture to identify exploitable vulnerabilities that need to be looked at.

Training models specifically for security

Frontier models continue to improve at understanding code, and those improvements benefit every type of code review.

At depthfirst, we also train models specifically for vulnerability discovery. The goal is to improve security-specific reasoning: tracking attacker-controlled data, understanding trust boundaries, recognizing subtle authorization failures, following multi-step attack paths, and assessing exploitability.

The depthfirst harness selects the model best suited to each security task and combines it with the application’s evolving threat model, dependency vulnerability intelligence, and tools for tracing attack paths and validating exploitability. Together, these capabilities help surface vulnerabilities that matter and provide inline remediations.

General PR bots are becoming an important part of the development workflow, and we use them ourselves. Security review, though, requires a deeper understanding of how a change affects the application’s attack surface.

That is what the depthfirst Security Reviewer is built to do on every pull request. The goal is simple: find meaningful security issues earlier and help engineering teams and their coding agents ship secure software at the pace they build.