clet.xyz

The Ethereum Foundation’s Protocol Security team has successfully deployed coordinated AI agents to identify critical bugs within the network’s core infrastructure, marking a significant advancement in the application of artificial intelligence to blockchain security. This pioneering effort, detailed in a recent internal report, highlights the growing capabilities of AI in probing complex software systems and underscores the evolving landscape of cybersecurity for decentralized technologies. One of the most notable outcomes of this initiative is the discovery and subsequent disclosure of a remotely-triggerable panic in libp2p’s gossipsub protocol, a foundational component for Ethereum’s consensus clients. This vulnerability has been assigned the identifier CVE-2026-34219 and has been patched by the libp2p development team, with the Ethereum Foundation’s Protocol Security team receiving due credit for their crucial discovery.

The surprise for the team was not the discovery of bugs, which was an anticipated outcome, but rather the efficiency with which these critical issues were unearthed and the substantial effort required to distinguish genuine vulnerabilities from false positives. This distinction between "signal" and "noise" has emerged as the primary challenge in leveraging AI for security audits. The team’s experience offers valuable insights for client development teams and independent security researchers looking to adopt similar AI-driven auditing methodologies.

This initiative aligns with a broader trend across the technology sector, where sophisticated AI models are increasingly being directed towards identifying software weaknesses. Companies like Anthropic, with its Frontier Red Team, have developed AI agents capable of generating property-based tests that have uncovered real-world bugs across the Python ecosystem. Similarly, Cloudflare has deployed advanced frontier models within a security research framework to scrutinize its own systems. The common thread across these efforts is a cyclical process: deploying a powerful AI model against a codebase, allowing it to explore potential vulnerabilities, and then meticulously triaging the findings. The core question for all these organizations is how to navigate the deluge of potential issues without being overwhelmed by plausible but ultimately incorrect assertions.

The rapid evolution of AI tooling means that specific technical configurations can become outdated quickly. Therefore, the Ethereum Foundation’s approach focuses on enduring methodologies rather than transient tools. The principles guiding their work, such as rigorous verification and a structured approach to bug reporting, are intended to remain relevant even as the underlying AI technologies advance.

The AI Agent Audit Pipeline: Organization and Methodology

At its heart, an AI agent deployed against a codebase functions as a sophisticated search tool, akin to a fuzzer but with a vastly more nuanced output. While a traditional fuzzer typically returns a crash and a stack trace, an AI agent can generate a comprehensive report. This report often includes a detailed explanation of the potential vulnerability, including the call chain, an assessment of its impact, a suggested severity rating, and, crucially, the artifacts necessary to reproduce the issue. This reproducibility is paramount, providing a tangible proof-of-concept that can be directly tested against the actual code. The ultimate measure of success for these AI agents is not the sheer volume of candidates they produce, but the number of those candidates that prove to be genuine vulnerabilities.

The Ethereum Foundation’s Protocol Security team employs a strategy of running multiple AI agents in parallel against a single target system. This coordination is managed directly within the code repository itself, leveraging version control for shared state rather than relying on a centralized process for task allocation. An agent will document its proposed vulnerability, perform the necessary work to validate it, and then commit its findings. This decentralized coordination model has been inspired by Anthropic’s approach to building a C compiler with a fleet of AI agents, which utilizes a similar repository-centric coordination mechanism. This design minimizes the need for a complex central coordinator and reduces potential points of failure.

The workflow is structured into distinct stages, mirroring a process that has been refined through the experiences of other leading organizations. Cloudflare’s description of their process, which includes reconnaissance, parallel hunting, independent validation, deduplication, and reporting, significantly shaped the Ethereum team’s own methodology.

Before a candidate vulnerability is recognized as a confirmed finding, it must meet a stringent set of criteria. These criteria are codified to ensure a specific, testable claim and a clear definition of "done." The essential components of a validated candidate include:

• Target: The specific component and entry point within the system that a real-world attacker could feasibly reach.
• Invariant: The fundamental property or condition that the system is expected to maintain.
• Mechanism: The precise method by which this invariant might be violated or broken.
• Success: Observable proof of the vulnerability, such as a system panic, a denial-of-service condition, or the acceptance of invalid input.
• Reproducer: A self-contained artifact, such as a script or test case, that reliably triggers the failure when executed against the live code.
• Dedup: A unique identifier to prevent multiple agents from investigating the same issue concurrently.

This structured schema is not merely bureaucratic; it compels the AI agent to articulate a precise, verifiable claim. An agent that is required to document observable proof cannot simply rely on subjective assessments like "this code looks risky."

The Ironclad Rule: Reproducibility is Non-Negotiable

The single most critical principle governing the validation of AI-generated findings is reproducibility. A candidate vulnerability is not considered a genuine finding until a self-contained artifact exists that can reliably reproduce the failure in the actual codebase. Furthermore, this reproducer must be executable by individuals other than the one who discovered it, ensuring its robustness and independence.

The reproducer acts as an objective arbiter, bypassing the narrative or perceived confidence of the AI model. It either works, demonstrating the vulnerability, or it fails, indicating a false positive. The true value of this requirement lies in its ability to filter out numerous false positives that commonly emerge from AI-driven audits. Three recurring categories of false positives highlight the ways in which an agent can incorrectly signal a vulnerability:

• Misinterpreted Control Flow: The agent might identify a code path that appears exploitable but is, in reality, unreachable under normal operating conditions or due to implicit system constraints.
• Flawed Success Condition: The agent might report a successful exploitation, but the "success" observed is not a true security failure. This could be a benign error message, a temporary state that self-corrects, or an outcome that does not impact system integrity or security.
• Incomplete or Incorrect Reproducer: The generated proof-of-concept might be flawed, incomplete, or dependent on specific, non-standard environmental conditions that are not representative of real-world scenarios. The reproducer may also fail to accurately reflect the agent’s initial hypothesis, leading to a disconnect between the claimed vulnerability and the provided evidence.

These scenarios mirror the familiar problem in traditional software testing where a test passes not because the code is correct, but because the test itself is inadequate and fails to exercise the critical logic. The challenge with AI agents is their capacity to generate these flawed "tests" at an unprecedented scale and with an equally high degree of apparent confidence. Consequently, the verification process must be highly automated to prevent the system from being overwhelmed by such misleading outputs. Human oversight is indispensable, but it must be applied to a rigorously pre-filtered set of candidates.

Navigating the Signal-to-Noise Ratio: The Triage Imperative

The overwhelming majority of AI-generated candidates are incorrect, duplicates of known issues, or fall outside the defined scope of the audit. This is not a flaw in the methodology but rather an inherent characteristic of how these systems operate. The objective is to swiftly reject invalid candidates and to back genuine vulnerabilities with irrefutable proof.

Each candidate that passes initial filtering undergoes two independent validation checks:

1. Reachability: Can a real-world attacker realistically access this vulnerability under typical system configurations? This involves assessing the necessary privileges, network access, and computational resources required to trigger the alleged flaw.
2. Cost-Benefit Analysis: What is the cost to the attacker to exploit the vulnerability, and what is the cost to the network if it is successfully exploited? A bug that any single peer can trigger with minimal effort is fundamentally different from one requiring sophisticated orchestration or significant resource investment.

Furthermore, all validated findings are cross-referenced against a continuously updated list of known issues, including previously disclosed vulnerabilities, fixed bugs, and rejected candidates. This prevents agents from repeatedly identifying and reporting issues that have already been addressed, a common pitfall in large-scale automated auditing.

The acceptance rate of candidates varies significantly depending on the target system. Auditing mature, heavily scrutinized codebases often results in very few surviving candidates, a finding that itself is valuable. The conclusion, "extensive investigation yielded no new critical vulnerabilities," is a significant security assurance. Conversely, when applied to less explored code or systems with formal verification, where machine-checked proofs cover specific models, more candidates may emerge.

The experience of other organizations corroborates the Ethereum team’s findings. Cloudflare’s key insight was the superiority of narrow, focused audits over broad, indiscriminate scanning. Anthropic’s property-based testing agent, for instance, generated approximately a thousand candidate reports, but subsequent ranking and expert review narrowed this down to a small set of highly reliable findings. The generation of potential issues was the less demanding phase; the critical work lay in the rigorous validation of those findings.

AI’s Strengths and Weaknesses in Security Auditing

The deployment of AI agents in security auditing is accompanied by both significant promise and inherent limitations. Understanding these nuances is crucial for effective implementation.

The effectiveness of AI agents is not uniform across all tasks. Security researcher Stanislav Fort’s analysis of various AI models found that while an agent might excel at reconstructing a full exploit chain for one codebase, it could falter on basic data-flow tracing in another. This "jagged frontier" effect necessitates that each candidate finding be evaluated independently.

A particularly important weakness lies in the agents’ proficiency with single-step reasoning versus their struggles with multi-step vulnerabilities. Bugs that manifest only after a specific sequence of seemingly valid operations are the most challenging to detect with current AI models. In such cases, the AI agent’s role shifts from being the primary search tool to an intelligent suggester, guiding human researchers toward sequences that warrant testing with a stateful test harness. When used as a replacement for such harnesses, AI agents risk overlooking the most insidious and costly vulnerabilities – those that emerge only through complex, stateful interactions.

Maintaining Integrity: Essential Practices for Trustworthy AI Audits

Several straightforward practices are essential for ensuring the trustworthiness of AI-generated security findings. These habits, while simple, form the bedrock of reliable automated security analysis:

• Reproducibility as a First Principle: As emphasized, every claimed vulnerability must be accompanied by a self-contained, independently verifiable reproducer. Without it, the claim remains unsubstantiated.
• Clear Definition of "Done": The criteria for a finding must be unambiguous, covering the target, invariant, mechanism, success condition, and the reproducer. This prevents subjective interpretations and ensures a consistent standard.
• Automated Validation and Triage: The process of checking candidates against existing known issues and performing initial reachability and impact assessments should be automated to handle the volume of outputs.
• Independent Verification: At least two independent checks are required for any candidate to be considered a finding. This reduces the risk of single points of failure or bias.
• Continuous Maintenance of Known Issues: An up-to-date database of previously identified, fixed, or rejected vulnerabilities is critical for preventing redundant effort and maintaining efficiency.
• Contextualized Reporting: Reports should clearly define the context in which a vulnerability can be exploited, including necessary preconditions and the attacker’s capabilities. This avoids overstating the risk.
• Transparency in Methodology: The process by which AI agents are used, their limitations, and the validation steps are clearly documented to foster trust and enable replication.

The Shifting Bottleneck: From Discovery to Verification

The advent of AI in security auditing has not rendered security researchers obsolete; rather, it has fundamentally shifted the nature of their work. The time and effort previously dedicated to formulating hypotheses and manually tracing potential exploit chains are now redirected towards evaluating these hypotheses at scale. This includes the crucial tasks of building the "oracle" for validation, managing the triage process, maintaining the repository of known issues, and overseeing the disclosure of vulnerabilities.

The bottleneck in the security process has not disappeared; it has merely moved. It has transitioned from the discovery phase to the verification and trustworthiness phase. This is a more appropriate location for human judgment, as it involves critical analysis and decision-making. However, it remains a bottleneck, and ignoring its existence can lead to flawed security assessments.

The practices that enable this new paradigm are not novel. Reproducible failures, robust validation mechanisms (oracles), and meticulous triage are the same principles that have transformed fuzzing from an academic research topic into a standard industry practice over the past fifteen years. The tools are new and rapidly evolving, but the underlying methodologies remain consistent.

The pace of change in AI tools presents an ongoing challenge. Nicholas Carlini, a respected figure in AI security, argues for taking the exponential growth projections seriously, even while acknowledging the inherent uncertainties and wide error margins. If the generation of potential vulnerabilities continues to accelerate at such a pace, the capacity for human judgment and verification must keep pace in parallel. Failure to do so will widen the gap between what AI can produce and what can be rigorously verified, potentially leading to the deployment of insecure systems.

For the critical infrastructure that underpins Ethereum, this enhanced judgment is paramount. AI agents enable a far more extensive exploration of the attack surface than manual methods alone could achieve. In return, they demand a more rigorous and scaled form of human judgment to sift through a vastly larger volume of confident-sounding claims. This trade-off is undeniably worthwhile, provided that the ultimate product of this endeavor—sound and trustworthy security verification—is never compromised.