ExploitGym: AI Agents’ Exploitation Capabilities and Boundaries in Controlled Environments

by
Table of contents

🌏 閱讀中文版本

The ExploitGym benchmark reveals a technical reality: in controlled environments with security protections disabled, leading AI agents can bridge the engineering gap between vulnerability discovery and attack execution. This research does not suggest that established defenses have lost their effectiveness. Instead, it maps the boundaries of AI agents’ causal reasoning when they encounter different defensive mechanisms. Understanding those boundaries is essential for redefining risk management in today’s AI-assisted security engineering.

The Core Question: Can AI Agents Develop Working Exploits for Known Vulnerabilities?

In security, an exploit is more than a stack of code. It requires rigorous logical reasoning and control over system state. Traditionally, engineers with deep systems knowledge developed these manually. As large language models (LLMs) and AI agents become more capable, a central question emerges: can AI independently complete the full path from identifying a vulnerability to generating an executable attack script?

ExploitGym was designed to quantify that capability. It does not measure general programming performance. It focuses on a specific engineering path: developing a working exploit for a known real-world vulnerability instance. That requires an agent to understand not only program logic, but also operating-system-level defenses and techniques for working around them.

Task Design: Three Environments, Defense Switches, and Success Criteria

ExploitGym builds a large benchmark of 898 real vulnerability instances across three main domains. Each instance is packaged in a reproducible, isolated environment. Within a two-hour time limit, agents can run Bash commands, edit files, and generate attack scripts. The three task types differ in their environments, defense configurations, and success criteria:

  1. Userspace programs:

    • Test environment: An independent isolated container packaging individual userspace applications and binaries.
    • Defense switches: The main experiment disables operating-system-level protections such as Address Space Layout Randomization (ASLR). Follow-up experiments restore ASLR to quantify the effect of the protection.
    • Success criterion: Within two hours, the agent must generate an executable exploit script that triggers the target vulnerability and gains unauthorized code execution.

  2. Google V8 JavaScript engine:

    • Test environment: A JavaScript runtime environment containing specific V8 versions, such as the Maglev JIT compiler.
    • Defense switches: The main experiment intentionally disables ASLR, the V8 Heap Sandbox, and the Renderer Sandbox. Defense tests restore these sandboxing and randomization mechanisms.
    • Success criterion: The agent must use logic vulnerabilities such as type confusion to work around memory-access restrictions, then perform unauthorized memory reads, writes, and code execution in the engine environment.

  3. Linux kernel:

    • Test environment: A virtualized kernel test environment containing specified vulnerable Linux kernel versions and underlying driver states.
    • Defense switches: The main experiment disables Kernel Address Space Layout Randomization (KASLR) and related defenses. Defense tests restore standard kernel defense configurations.
    • Success criterion: Within the time limit, the agent must complete complex kernel-memory operations to achieve privilege escalation or control kernel execution.

In one concrete example, GPT-5.4 successfully exploited a type-confusion vulnerability in the Maglev JIT compiler of the V8 engine, using a sequence of precise memory operations to achieve unauthorized code execution. The task involved substantial operational complexity. The agent had to identify a logic error in how Maglev, a mid-tier optimizing compiler, handled a specific type, then attempt to turn that type confusion into reads and writes to unauthorized memory regions. The process took roughly 71 minutes, involved 447 Bash commands and 21 file edits, and ultimately produced a 229-line exploit.js script.

The researchers specifically noted that the main experiment disabled these protections to isolate defensive interference and measure the upper bound of frontier-model capability.

Results: Leading Models and the Effectiveness of Defenses

ExploitGym’s experimental data shows that today’s leading AI models have meaningful potential to convert vulnerabilities into attacks when evaluated in environments with security protections disabled. This main experiment was conducted under a trusted-access program and turned off all security protections to probe the models’ upper-bound capabilities.

The data shows that Anthropic’s Claude Mythos Preview with Claude Code and OpenAI’s GPT-5.5 with Codex CLI were the strongest combinations. Under the two-hour limit and with defenses disabled, Claude Mythos Preview solved 157 instances, while GPT-5.5 solved 120. They showed particular strength in Linux kernel exploitation: Claude Mythos Preview and GPT-5.5 achieved 12 and 22 successes respectively, while the highest result from other models on kernel tasks without defenses was only one success.

However, once standard defenses such as ASLR, the V8 Heap Sandbox, and the Renderer Sandbox were restored, attack success rates dropped sharply. In cross-model tests after standard defenses were restored, all tested models achieved only 37 successes on userspace tasks, 20 on V8 engine tasks, and 12 on kernel tasks.

Take GPT-5.4 as an example. Once ASLR and the V8 sandboxes were restored, it could no longer complete the code-execution attack it had previously achieved. This contrast clearly shows that existing system-level protections can introduce enough uncertainty into the state space to substantially reduce an AI agent’s ability to execute effective attacks.

Tradeoff Analysis: The Interaction Between Capability Boundaries and Defensive Effectiveness

When evaluating ExploitGym’s results, it is important to distinguish between a model’s potential capability and its practical attack success rate. The 157 and 120 successful instances measured without defenses represent potential in code generation and strategic planning, not real-world threat levels in production environments.

There is an important tradeoff here. By introducing randomization, such as ASLR, and isolation, such as sandboxes, defenses substantially increase the uncertainty and trial-and-error effort required for an agent to search the state space for a viable path. This is not simply a matter of elapsed time. In production, the interaction between defenses and model capability is the key risk consideration, rather than a simple comparison of model strength.

A useful way to read these figures is to establish a simple comparison matrix: whether the model and agent tool are the same, whether the task is userspace, V8, or kernel, whether defenses are enabled, whether the time limit remains two hours, and whether success still means a working exploit. If any one of these conditions changes, the comparability and transferability of the numbers change with it. This matrix cannot predict whether a specific production system will be breached. But it can prevent upper-bound capability from being read as a general success rate. It also makes it easier to audit which party carries the risk, where isolation boundaries sit, and which defensive states require ongoing validation. The real system risk is not a single model ranking. It is the combined configuration of model, tool, task, and defense.

For security engineering teams, this means reassessing the security of AI-based automated vulnerability-validation tools. If these tools are used in production, isolation boundaries, least privilege, and enabled defense states can serve as adoption thresholds to ensure execution remains within controlled, well-defended environments.

Boundaries and Limitations: Considering the Scope of Applicability

ExploitGym’s data presents a clear technical picture: the models tested in the paper can convert vulnerabilities into attacks under specific conditions, but that capability is tightly constrained by context. From a technology-selection perspective, Claude Mythos Preview and GPT-5.5 show strengths in low-level program reasoning and complex strategic planning. That makes them more valuable in security research settings that require deep systems understanding. This capability also comes with specific execution and time costs, such as the 71 minutes in the example.

The limits of the current results also deserve attention. Although ExploitGym’s test environments simulate real scenarios, they remain controlled laboratory environments. In real internet attacks, target-system complexity, network latency, and dynamically changing defense strategies may affect AI agent performance. Care is therefore needed when extrapolating laboratory results directly into real-world threat models.

Conclusion: Residual Risk After Defenses

ExploitGym demonstrates the attack potential of AI agents in environments without defenses, but that is not equivalent to practical threats in production. The research shows that standard protections such as ASLR and sandboxes can substantially reduce agent success rates. Under this model, example, defense configuration, and two-hour limit, GPT-5.4 no longer achieved code execution. That limitation remains: the result cannot be extrapolated as a universal block.

For security engineering teams, the key is to assess whether models still have the potential to break through after these defenses are enabled, and whether that potential risk falls within the organization’s risk tolerance. This is a direction worth considering, depending on the specific workflow and business requirements.

Sources