Alignment, Reward Hacking, and Autonomous Cyberattacks

Alignment alone is not enough to make autonomous AI safe. An agent pursuing a legitimate goal can still reward hack, bypass guardrails, and compromise systems. During training, security guardrails support alignment by removing shortcuts that let agents earn rewards without doing the intended task. In deployment, they constrain unauthorized actions and contain failures. The more autonomy we give AI, the more we need secure software and infrastructure to make that autonomy possible.

Last month, OpenAI published an account of AI agents coordinating with one another, compromising Hugging Face, and gaining administrator access to part of OpenAI’s research infrastructure. The agents were of course not instructed to perform such attacks, but decided to pursue them as a means to another end.

These incidents show that some of the risks the security community discussed as hypothetical are already happening. Agents pursuing a training objective found reasons to compromise real infrastructure. Making that infrastructure secure is becoming a prerequisite for giving AI more autonomy.

The events unfolded in three waves:

1. Initial coordination: May-July 4

During training, agents turned a software package repository into a hidden message board. OpenAI later rebuilt the repository to address security issues, which also erased that message board.

2. Hugging Face attack: July 7-12

During subsequent tests, a new group of agents found another way to exchange messages through the repository. That allowed them to coordinate again.

Around 1,200 participated on the message board, and roughly 700 became involved in the attack on Hugging Face. They were looking for ways to manipulate the system grading their work. This second wave suddenly stopped for reasons still unclear.

3. OpenAI infrastructure attack: July 13-19

A third wave continued the activity and gained administrator access to an OpenAI research cluster. These evaluations operated with fewer safeguards than deployed products, and OpenAI says customer data was unaffected. In short, agents turned an evaluation into an attack on real infrastructure.

The investigations have since uncovered activity beyond Hugging Face. In September, Australian officials disclosed that an OpenAI agent researching public medicine spending had bypassed restrictions on a government health statistics portal. Officials said no individual medical records were accessed. Researchers also identified an unsuccessful attempt to hack a U.S. Department of Education website. OpenAI says it has notified dozens of organizations as its review continues, while Axios reported on Sep 26, 2026 that OpenAI, Anthropic, and security researchers were investigating tens of thousands of potentially problematic incidents. That figure does not represent confirmed breaches, but the emerging findings show why this matters beyond AI labs: even routine research tasks can lead agents to attempt unauthorized access to other organizations’ systems.

What followed was a strong response from the AI community. That included calls from Dario Amodei, Sam Altman, and Elon Musk to slow AI development while safety measures catch up.

Alignment

In the past weeks, talks on alignment have suddenly intensified, especially following a viral X post from a safety researcher at Anthropic calling out the existential risks we face. While there are still debates over the exact definition of an aligned model, we can think of alignment as making models reliably pursue intended goals while respecting constraints. Common techniques to align models include:

• Reinforcement learning from human feedback (RLHF): uses humans to review responses and shape behavior.

• Constitutional AI: uses explicit principles (similar to a constitution) to guide model feedback and training.

Other research helps us understand and supervise increasingly capable models:

• Interpretability investigates how models arrive at their outputs. It focuses on developing tools to look inside the “black box” of a model to understand its internal reasoning and verify its intentions. We are basically trying to reverse engineer neural networks by mapping circuits or subnetworks within them, so we can see if a misaligned combination of neurons exists, and prevent its activation.

• Scalable oversight explores how we can supervise models whose capabilities exceed our own. One research direction, weak-to-strong generalization, studies whether stronger models can learn from weaker supervisors and perform beyond what those supervisors can do.

One difficulty in aligning a model is that the result we reward does not necessarily induce the behavior we actually want, which leads to model learning how to reward hack. We want an agent to solve a task, but if it learns instead how to manipulate the grader to receive credit, that’s reward hacking.

Now give that agent a task it cannot solve, reward persistence, and allow it to run for a long time. It may keep searching for shortcuts well beyond the intended scope. Difficult tasks without a safe exit were a major contributor to the incident that OpenAI identified in its investigation.

We’ve seen versions of this in our own work at depthfirst:

• Finding the answers in the dataset: during exploitation training, a model learned to search for the CTF dataset containing flags and hints about how to solve the challenges. This behavior became more frequent across training iterations, even in long conversations with context compaction.

• Reporting results that never happened: in an early exploitation experiment, a model filled failed proof-of-concept results with hypothetical values instead of actual execution output. Apparent performance dropped significantly after we added instructions discouraging this behavior.

• Working around a reporting guardrail: during a vulnerability scan, an agent downloaded an upstream dependency and found vulnerabilities outside the primary repository. When our code-location guardrail prevented it from reporting those findings, it created a new directory in the sandbox and wrote files there to get around the restriction.

We also observed models using internet access in ways that undermined training tasks, which led us to restrict outbound network access from training sandboxes.

These examples show why boundaries must be enforced by the environment. During training, security guardrails support alignment by removing shortcuts to the reward. If an agent cannot access the answers or alter the grader, it has fewer ways to earn credit without doing the intended task. This does not guarantee alignment, but it helps make the reward reflect the behavior we actually want.

Novel challenges for security teams

For security teams, this adds another kind of adversary. We’ve spent decades defending against human attackers. We’re now dealing with human attackers using increasingly capable AI, while also having to account for agents pursuing unauthorized actions without a human directing the attack.

As we have observed in this incident, an agent doesn’t need to be instructed to harm a company for that company to become a target. Access to its systems may simply look useful for completing another task.

An agent can be pursuing a legitimate goal and still try to bypass the guardrails we put in place because it sees them as obstacles to completing the task. At the extreme, we could face fully rogue systems pursuing their own objectives and actively resisting human control. We don’t believe we’re at that point yet, but even an agent trying to complete a benign task can cause real damage along the way.

As AI becomes more capable at finding and exploiting vulnerabilities, defenders need to find and fix weaknesses faster. Teams will need to automate more of their security programs, from vulnerability discovery to exploit validation and testing fixes. The security community also needs to develop the tools, practices, and playbooks to defend against these behaviors, using incidents like these and the failures we’ve observed in training to improve how we evaluate agents, enforce boundaries, and respond when those boundaries fail. As AI improves, we need defensive capabilities to advance faster than offensive ones.

This is why the work we’re doing at depthfirst feels increasingly necessary: using advances in AI to find vulnerabilities, validate whether they can be exploited, and help teams fix them. The same capabilities that make these attacks possible can make defenders more effective.

The more autonomy we want to give AI, the more we need to invest in the security that makes it possible.