clet.xyz

The Ethereum Foundation’s Protocol Security team has successfully deployed coordinated AI agents to identify vulnerabilities within critical protocol components, including systems software, cryptographic code, and smart contracts. This initiative has yielded tangible results, most notably the discovery of a remotely-triggerable panic in libp2p’s gossipsub, a fundamental element of the peer-to-peer layer utilized by Ethereum consensus clients. This vulnerability, now publicly disclosed as CVE-2026-34219, underscores the growing capability of AI in uncovering complex software flaws.

The true revelation for the Protocol Security team, however, was not the discovery of bugs itself, which was an expected outcome, but rather the disproportionate effort required to distinguish genuine vulnerabilities from false positives. This experience has led the team to refine their methodology, emphasizing the robust verification and meticulous triage of AI-generated findings. This approach is crucial for client teams and security researchers aiming to leverage similar AI-driven security auditing techniques.

This pioneering work aligns with broader industry trends. Companies like Anthropic, with their Frontier Red Team, have demonstrated AI agents capable of generating property-based tests that have uncovered real bugs across the Python ecosystem. Cloudflare has also reported utilizing advanced frontier models in a security research harness against their own systems. The common thread across these efforts is a cyclical process: deploying sophisticated AI models to probe codebases, initiating automated searches, and then rigorously vetting the outputs. The paramount challenge, therefore, is to navigate this deluge of confident-sounding, yet potentially inaccurate, claims.

It is important to note that the landscape of AI-driven auditing tools is evolving at an unprecedented pace. Specific technical configurations are likely to become outdated within weeks. Consequently, this report focuses on the persistent methodologies rather than ephemeral tooling. The complex topic of vulnerability disclosure is also a distinct area, slated for separate, in-depth coverage.

AI agents deployed against a codebase function primarily as sophisticated search tools, analogous to fuzzers. However, their output is significantly richer. While a fuzzer might present a crash and a stack trace, an AI agent provides a comprehensive report. This includes a detailed analysis, an assessment of the potential impact, a suggested severity level, and, critically, the artifacts necessary to reproduce the issue. The presence of a runnable proof-of-concept against the actual code is paramount, serving as the most compelling evidence of a genuine vulnerability. Therefore, the true measure of success lies not in the sheer volume of candidate findings an agent produces, but in the proportion that are validated as authentic issues.

The Operational Framework: Orchestrating AI Agents for Security Audits

The Ethereum Foundation’s Protocol Security team has adopted a distributed, parallel approach to AI agent deployment. Multiple agents are concurrently tasked with analyzing a single target codebase. Coordination is achieved through the repository itself, leveraging version control for shared state and eschewing a centralized process for task allocation. An agent records its assertion or discovery in a designated location within the repository, allowing other agents to observe its progress, and then proceeds with its assigned task, committing its findings upon completion.

This decentralized coordination model draws inspiration from Anthropic’s documented approach to building a C compiler with a fleet of AI agents. This methodology eliminates the need for a centralized coordinator, thereby reducing potential points of failure and simplifying maintenance.

The operational pipeline, as described, involves several distinct stages, mirroring the processes outlined by Cloudflare in their own security research efforts. These stages typically encompass:

• Reconnaissance: Initial analysis of the target codebase to understand its structure, dependencies, and potential attack surfaces.
• Parallel Hunting: Concurrent execution of multiple AI agents, each employing different strategies or focusing on specific code areas to discover potential vulnerabilities.
• Independent Validation: Rigorous testing of each candidate vulnerability by independent agents or human reviewers to confirm its existence and reproducibility.
• Deduplication: Identification and elimination of duplicate findings to avoid redundant efforts and streamline the reporting process.
• Reporting: Generation of comprehensive reports detailing the nature of the vulnerability, its impact, and proposed mitigation strategies.

A candidate vulnerability, before it is formally classified as a confirmed finding, must meet a stringent set of criteria:

• Target: Identification of the specific component and entry point that a real-world attacker could realistically exploit within a normal operational configuration.
• Invariant: A clear statement of the fundamental property or condition that must hold true for the system’s correct operation.
• Mechanism: A precise description of the specific method or sequence of actions by which the invariant might be violated.
• Success: Observable proof of the exploit, such as a system panic, a stall in execution, or the acceptance of invalid input that compromises integrity.
• Reproducer: A self-contained artifact, such as code or a configuration script, that reliably demonstrates the vulnerability when executed against the target system.
• Dedup Key: A unique identifier associated with the vulnerability to prevent other agents from pursuing the same issue.

This structured schema is deliberately designed to enforce specificity and establish a clear definition of "done." An agent that is required to document an observable proof of compromise is less likely to rely on subjective assessments such as "this code looks risky."

The Cornerstone of Trust: Reproducibility as the Ultimate Arbiter

Perhaps the most critical principle governing the validation of AI-generated findings is that "if it’s not reproducible, it didn’t happen." A candidate vulnerability is not considered a confirmed finding until a self-contained artifact exists that can reliably reproduce the failure within the actual code, and this artifact can be successfully executed by an independent party unfamiliar with its creation.

The reproducer acts as an objective arbiter, independent of the AI agent’s narrative or confidence level. It either works, demonstrating the vulnerability, or it does not. The true value of this rigorous reproducibility check lies in its ability to filter out false positives, which are common occurrences in AI-driven security audits. Over time, three recurring categories of false positives have been identified, each representing a scenario where an agent achieves a simulated "pass" for the wrong reasons:

• Invalid Input Acceptance: The agent identifies a scenario where the system accepts input that is technically malformed or unexpected but does not lead to a genuine security breach or functional failure. The system may simply reject the input or handle it gracefully, but the agent might misinterpret this as a vulnerability.
• Non-Exploitable State: The agent discovers a peculiar state within the system that deviates from expected behavior but cannot be practically exploited by an attacker to achieve a malicious outcome. This might involve subtle inconsistencies that have no tangible security implications.
• Inaccessible Code Paths: The agent identifies a potential vulnerability within a section of code that is unreachable under normal operational conditions or through standard user interactions. Such vulnerabilities are of theoretical interest but pose no immediate practical threat.

This phenomenon is not unique to AI; it mirrors the issue of a software test that passes not because the code is correct, but because the test itself is flawed and fails to adequately exercise the functionality it purports to validate. The novelty in the context of AI agents lies in the sheer volume and speed at which these flawed "tests" can be generated, often with an equally high degree of apparent confidence. This necessitates an automated, robust verification process, as relying on the AI agent to self-correct is an unreliable strategy.

Navigating the Signal-to-Noise Ratio: The Core of the Challenge

The overwhelming majority of AI-generated candidates are destined to be incorrect, duplicates of existing issues, or outside the defined scope of the audit. This is not an inherent flaw in the methodology but rather a fundamental characteristic of how it operates. The primary objective is to efficiently reject erroneous claims and to buttress genuine findings with irrefutable proof.

Each candidate that progresses beyond the initial filtering undergoes two independent validation checks. The first assesses whether a real-world attacker could realistically trigger the vulnerability in a typical system configuration. The second evaluates the cost-benefit analysis for an attacker: what resources would be required to execute the exploit, and what would be the network’s cost if the vulnerability were successfully leveraged? A bug that can be triggered by any single peer presents a vastly different risk profile than one requiring specialized access or substantial computational resources.

Furthermore, every candidate is cross-referenced against a continuously updated registry of known, previously disclosed, or rejected vulnerabilities. Without this crucial step, AI agents would repeatedly "rediscover" resolved issues, leading to wasted effort and inflated reporting.

The acceptance rate of AI-generated candidates varies significantly depending on the target system. Auditing mature, extensively scrutinized codebases typically results in a very low acceptance rate, a finding that is itself valuable. The conclusion, "extensive review yielded no novel significant vulnerabilities," is a legitimate and important security outcome. Conversely, when applied to less explored code or systems that have undergone formal verification (where machine-checked proofs guarantee correctness for a specific model, but the deployed implementation is assumed to match), a higher proportion of candidates may be deemed valid.

The observation that the triage process constitutes the most significant bottleneck is widely shared. Cloudflare’s primary insight was the strategic advantage of a narrowly defined audit scope over broad, indiscriminate scanning. Anthropic’s property-based testing agent, for instance, generated approximately one thousand candidate reports, subsequently refined through ranking and expert review to a top tier of findings that held up with an 86% success rate. The generation phase was comparatively effortless; the refinement was the true challenge. While specific numbers are not disclosed for the Ethereum Foundation’s efforts, it is understood that target-specific data would reveal more about the system under scrutiny than the efficacy of the methodology itself.

Understanding AI Capabilities and Limitations in Security Auditing

A balanced perspective on AI’s role in security auditing requires an understanding of both its strengths and its limitations. The current landscape is often characterized by exaggerated claims in both directions.

The performance of AI agents is not uniform across all tasks. Stanislav Fort’s research, testing various models against real-world vulnerabilities, describes this phenomenon as a "jagged frontier." An AI model that can successfully reconstruct a full exploit chain for one codebase might falter in performing basic data-flow tracing on another. This variability underscores the necessity of validating each candidate finding independently, as a single successful outcome does not guarantee consistent performance across different scenarios.

The final point in the table is particularly significant. A single AI agent session is adept at one-shot reasoning but struggles with vulnerabilities that manifest across a sequence of operations, where each individual step is legitimate, but their cumulative effect creates a flaw. In such cases, the AI’s role shifts from being the primary search tool to a facilitator. It can suggest which sequences of operations are most likely to yield interesting results, which are then subjected to rigorous testing using a stateful test harness. When employed in this capacity, AI agents are highly effective. However, if used as a complete replacement for such harnesses, they risk overlooking the most costly and insidious types of bugs—those that only become apparent through complex, multi-step interactions.

Maintaining Integrity: Essential Practices for Trustworthy Audits

A few fundamental practices are instrumental in ensuring the trustworthiness of findings generated by AI agents. These practices, while simple, are critical for the integrity of the audit process.

• Independent Verification: Every candidate finding must be independently validated by a separate entity or process, ideally a human security researcher, to confirm its exploitability and impact.
• Reproducibility: As emphasized, a candidate is only a finding if it can be reliably reproduced using the provided artifacts, without reliance on the original AI agent’s generated explanation.
• Contextual Analysis: Findings must be assessed within the context of real-world network conditions, attacker capabilities, and the system’s operational environment.
• Comprehensive Documentation: Detailed records of all findings, validation steps, and any mitigation efforts are essential for transparency and future reference.
• Continuous Learning: The process should incorporate feedback loops to refine AI models and improve the accuracy and efficiency of the detection and validation pipelines.

The Evolving Bottleneck: From Discovery to Verification

Artificial intelligence has not rendered the human security researcher obsolete. Instead, it has fundamentally shifted the nature of their work. The time previously allocated to formulating and pursuing hypotheses is now redirected towards evaluating those hypotheses at scale. This includes the intricate tasks of constructing robust validation oracles, meticulously executing triage procedures, maintaining comprehensive lists of known issues, and managing the complex process of vulnerability disclosure.

The bottleneck has not vanished; it has merely migrated. It has moved from the act of bug discovery to the more demanding challenge of ensuring the trustworthiness of the discovered results. This migration, while representing a more appropriate locus for human judgment, still constitutes a critical constraint. Ignoring this persistent bottleneck can lead to a false sense of security, where systems are declared "fine" based on incomplete or flawed verification.

The practices that underpin effective AI-driven security auditing are not novel. Reproducible failure reproduction, the establishment of reliable validation oracles, and diligent triage are the same foundational principles that have elevated fuzzing from an academic research topic to a standard practice in cybersecurity over the past fifteen years. The tools may be new, but the underlying methodologies remain consistent.

The rapid evolution of these AI tools presents an ongoing question. Nicholas Carlini, a respected figure in AI security and formerly a skeptic, argues that the exponential trajectory of AI capabilities warrants serious consideration, even while acknowledging the inherent uncertainties and wide error bars associated with such predictions. If the generation of potential vulnerabilities continues to accelerate at such a pace, the capacity for judgment and verification must keep pace. Failure to do so will inevitably widen the chasm between the volume of AI-generated claims and the actual number of verifiable security flaws.

For the critical systems upon which Ethereum depends, this capacity for rigorous judgment is paramount. AI agents enable a far broader and deeper exploration of codebases than was previously feasible through manual efforts. In return, they demand a more disciplined and nuanced approach to verification, applied across an ever-increasing volume of potentially misleading, yet confidently presented, assertions. This trade-off is undeniably advantageous, provided that the critical role of human judgment in the verification process is consistently recognized and prioritized. The true product of this endeavor is not the identification of a bug, but the assurance that the system is indeed secure.