In a groundbreaking evaluation that reimagines the landscape of artificial intelligence in cybersecurity, CyberArena has ranked twelve frontier AI agents across a series of live, adversarial attack-and-defense matches. The formidable GPT-5.6 Sol has emerged as the unequivocal leader, achieving a remarkable overall score of 78.6. Its unparalleled defensive prowess, highlighted by an exceptional 83% flag-retention rate, proved to be the decisive factor distinguishing it from its competitors. This new benchmark, developed by 2077AI and UC Berkeley RDI, shifts the paradigm from static vulnerability assessments to dynamic, real-time engagements, providing a more accurate and robust measure of an AI agent’s capabilities in a constantly evolving threat environment.
The prevailing challenge in evaluating AI models designed for cybersecurity has long been the inadequacy of traditional, static benchmarks. While an AI capable of successfully breaching a server might generate significant headlines, the far more complex and critical claim involves an AI’s ability to simultaneously defend its own systems against an actively attacking opponent. This dual capability—offensive and defensive—is the true litmus test for real-world cybersecurity resilience, a scenario largely unaddressed by conventional testing methodologies. CyberArena was specifically conceived to bridge this critical gap, placing a dozen cutting-edge AI agents in precisely this high-stakes, simultaneous attack-and-defense posture, meticulously logging every exploit, every patch, and every millisecond of response time.
Traditional cybersecurity benchmarks often mimic a standardized exam format: a single AI agent is presented with a fixed, vulnerable target, and its performance is measured against predefined tasks. While this approach offers reproducibility and ease of grading, as exemplified by professional-level CTF (Capture The Flag) benchmarks like Cybench, it suffers from two fundamental weaknesses. Firstly, the target environment remains static and unresponsive, failing to simulate the adaptive nature of real-world adversaries. Secondly, as AI models rapidly advance, these fixed tasks become increasingly saturated, making it difficult to differentiate true breakthrough performance from mere task completion.
The limitations of static scoring have become increasingly apparent. A study conducted by Dreadnode on 22 frontier models across 23 Cybench challenges revealed a concerning trend: reported pass rates were significantly inflated due to methods that could be construed as "cheating." The average pass rate stood at 41.5%, yet the clean solve rate—indicating genuine, unassisted problem-solving—plummeted to just 26.1%. This discrepancy underscores the critical need for evaluation environments that cannot be gamed by models trained on specific datasets or exploit patterns, and which demand genuine understanding and dynamic adaptation. CyberArena directly addresses this by transforming the "exam" into a dynamic "contest." Instead of a fixed target, two agents receive identical copies of a vulnerable service, each tasked with attacking the other’s copy while simultaneously patching and defending its own. This setup ensures that the performance ceiling is no longer a fixed 100% completion rate, but rather rises dynamically with the opponent’s capabilities, fostering a continuous challenge.
The Genesis of Dynamic AI Cybersecurity Benchmarking
CyberArena is the brainchild of a collaborative effort between 2077AI and the UC Berkeley Research Development and Innovation (RDI) group. The benchmark was officially unveiled to the public with its publication on Hugging Face on November 3, accompanied by a comprehensive research paper made available on arXiv. This launch marked a pivotal moment in the assessment of AI capabilities in a domain where static evaluations had long shown their inherent shortcomings. The researchers behind CyberArena posited that for AI to truly augment human cybersecurity efforts, it must be capable of dynamic, adaptive reasoning—a quality best assessed through live, adversarial interactions rather than pre-scripted challenges.
The development of CyberArena was driven by the recognition that real-world cyber warfare is not a static puzzle but a dynamic, evolving battle of wits. Traditional benchmarks, while useful for initial development, often fail to capture the nuances of active defense, where an exploit that works one moment might be rendered ineffective the next due to an opponent’s swift patch. This philosophical shift from a solo "exam" to a competitive "contest" forms the bedrock of CyberArena’s methodology. The collaboration between 2077AI, a leader in AI data foundations, and UC Berkeley RDI, known for cutting-edge research, positioned CyberArena to push the boundaries of AI evaluation in critical infrastructure and software security.
Unpacking the CyberArena Match Protocol
A standard CyberArena match is structured to simulate intense, real-time engagements, running over three distinct 20-minute rounds, critically, without any external hints or assistance provided to the agents. Prior to the commencement of each round, the arena’s infrastructure strategically plants a fresh, unique objective—or "flag"—within each vulnerable service instance assigned to the competing agents. With the round initiated, both AI agents are simultaneously unleashed, engaging in a dual role: actively attempting to exploit vulnerabilities and capture the opponent’s flag, while concurrently patching and fortifying their own service to prevent capture.
At the conclusion of each 20-minute round, a "bell" signals the rotation of objectives. This means that any flags captured in previous rounds become worthless, compelling agents to constantly re-evaluate and re-engage. Crucially, however, any patches deployed by the defending agents persist across rounds, accumulating defensive strength and forcing attackers to adapt their strategies. This mechanism ensures that successful defense isn’t just about immediate reaction but also about persistent fortification, mirroring the continuous security posture required in real-world systems.
The integrity and impartiality of scoring are paramount in CyberArena. All performance metrics are meticulously gathered by independent arena facilities, which are entirely inaccessible and untamperable by the competing AI agents. Throughout the extensive campaign, agents collectively invoked the arena’s game tools an astounding 53,511 times, demonstrating the depth of interaction and complexity of the environment. A dedicated flag service independently verifies every capture attempt, ensuring accuracy. Furthermore, a health poller, operating outside the defender’s root access, continuously monitors the availability and functionality of each service. This critical check prevents a common pitfall in cybersecurity: a "fix" that secures a system but inadvertently breaks its intended functionality. Such over-blocking or service disruption directly impacts the agent’s score, emphasizing the importance of effective and non-disruptive defense.
The final game score, meticulously calculated on a 100-point scale, equally weights three critical dimensions of performance: the number of rounds successfully captured (offensive prowess), the number of rounds successfully defended (defensive resilience), and the overall service availability (operational integrity). Each of these components contributes a third to the total score, ensuring a holistic assessment of an AI agent’s capabilities. This balanced scoring mechanism ensures that an agent cannot simply excel at attack at the expense of its own defense or system stability.
The Extensive Campaign and Statistical Rigor
The inaugural CyberArena campaign was a monumental undertaking, encompassing 760 matches, of which 466 were deemed valid and fully processed through batch 22 of the evaluation. This extensive dataset covered the performance of twelve distinct AI agents drawn from ten different model families. These agents were pitted against 40 ranked challenges, selected from a comprehensive catalog of 46 unique scenarios. The diversity of challenges is a key strength of CyberArena: thirty of these challenges are built around real-world software, with 28 specifically designed around a published Common Vulnerabilities and Exposures (CVE) entry, ensuring relevance to current threats. Additionally, 14 challenges were expertly ported from past attack-and-defense Capture The Flag competitions, bringing a rich history of competitive exploitation into the benchmark.
Given the impracticality of running every conceivable pairing of agents against every challenge, the CyberArena team employed a sophisticated statistical model. Matches were selected in controlled batches of 24, and the model, calibrated using 53 held-out matches, was then applied to estimate how agents would perform against opponents they had not directly encountered. This robust statistical methodology allows for a comprehensive ranking even with a subset of all possible head-to-head confrontations, ensuring reliable and generalizable results. The use of a statistical model, calibrated on unseen data, is crucial for extrapolating performance across the entire matrix of potential matchups without requiring an impossibly large number of direct competitions.
It is important to note the dynamic nature of the CyberArena leaderboard. While the initial paper provided a snapshot of the standings through batch 22, the live leaderboard continues to evolve. At the time of this report, it displays 479 of 1,948 matches ranked, with the twelve expected scores consistently aligning with the figures published in the initial research, indicating the stability and predictive power of the underlying statistical model. This ongoing update mechanism ensures that the benchmark remains a living, evolving assessment of AI cybersecurity capabilities.
Insights from the Capture Model: The Dynamics of Live Defense
One of the most profound insights gleaned from CyberArena’s fitted capture model is the transformative impact of live, adaptive opponents on attack and defense strategies. The data unequivocally demonstrates that once defenders successfully deploy patches, subsequent captures become remarkably rare, unless the attacker had already achieved a successful capture in the immediately preceding round. This finding is crucial: it highlights that an exploit, effective against a static target, can quickly become obsolete mid-match when faced with an intelligent, responsive defender. Momentum, in this dynamic environment, is a significant factor, suggesting that initial offensive success can create a critical window of opportunity that rapidly closes with effective defense.
This particular revelation represents a significant advancement in cybersecurity research, an insight that simply cannot be replicated by solo CTF scenarios where the target environment remains passive. In CyberArena, the most proficient attacker successfully captured an opponent’s flag in 70% of rounds, while the least effective managed a capture rate of 23%. On the defensive front, the performance spread was even wider, ranging from 43% of rounds successfully defended by weaker agents to an impressive 83% by the strongest. This wide variance underscores the current disparity in defensive AI capabilities among leading models.
Intriguingly, the study revealed a strong correlation between attack and defense capabilities, with a coefficient of 0.85 across all participating agents. This suggests that a strong understanding of offensive techniques often translates into robust defensive strategies, and vice-versa. An AI that can effectively exploit vulnerabilities likely possesses the inherent knowledge to identify and patch those same weaknesses. However, the leader’s advantage in this benchmark was decidedly asymmetric. GPT-5.6 Sol’s fitted attack ability, at +0.46, was nearly identical to that of Claude Opus 5, which scored +0.45. The crucial
