Blockchain security auditing firm OpenZeppelin has conducted an independent review of the AI-based smart contract security benchmark EVMbench, launched through a collaboration between OpenAI and Paradigm, and identified two major critical issues: data contamination during training and at least four vulnerabilities labeled as “high risk” that are actually invalid forgeries.
EVMbench Data Contamination: A Critical Flaw in AI Training Cutoff Dates
EVMbench was released in mid-February 2026, aiming to evaluate different AI models’ ability to identify, fix, and exploit smart contract vulnerabilities. During testing, the AI agents’ internet access was cut off to prevent them from searching for answers online. However, OpenZeppelin’s audit revealed a structural flaw: the benchmark is based on vulnerabilities from 120 audits conducted between 2024 and mid-2025, and most top AI models’ knowledge cutoff dates are also set in mid-2025.
This means AI agents likely encountered EVMbench’s vulnerability reports during pretraining, and their memory may already contain answers to all the questions. OpenZeppelin stated, “The most important ability for AI security is to discover new vulnerabilities in code that the model has never seen before.” The limited size of the dataset further amplifies the impact of contamination on overall evaluation.
Key Issues Found in the EVMbench Audit
- Training Data Contamination: Pretraining of AI agents may have included EVMbench’s vulnerability reports, rendering “zero-knowledge discovery” tests meaningless.
- Invalid High-Risk Vulnerability Classifications: At least four vulnerabilities marked as high risk are actually unexploitable.
- Scoring System Flaws: EVMbench previously awarded points for AI discovering these fake vulnerabilities, indicating issues with the scoring basis.
- Limited Dataset Size: Further magnifies the impact of contamination on overall results.
- Current Leaderboard: Anthropic’s Claude 4.6 leads, followed by OpenAI’s OC-GPT-5.2 and Google’s Gemini 3 Pro.
Fake Vulnerability Crisis: At Least Four High-Risk Classifications Proven Invalid
Beyond data contamination, OpenZeppelin uncovered specific factual errors. They evaluated at least four vulnerabilities categorized as high risk by EVMbench and found that these vulnerabilities do not exist—more critically, the described exploit methods are fundamentally ineffective.
OpenZeppelin pointed out, “These are not subjective disagreements over severity; rather, the described exploit methods simply do not work.” If an AI agent “discovers” these fake vulnerabilities during testing, it indicates the scoring system rewards incorrect results.
OpenZeppelin emphasized that this audit does not negate AI’s potential in blockchain security: “The issue is not whether AI will change the security of smart contracts— it certainly will. The problem is whether the data and benchmarks we use to build and evaluate these tools are aligned with the standards of the contracts they aim to protect.”
Frequently Asked Questions
Q: What issues did OpenZeppelin find in their audit of EVMbench?
A: They identified two core problems: first, data contamination, as EVMbench’s test vulnerabilities come from audits conducted between 2024 and 2025, overlapping with AI models’ training cutoff dates, meaning models may have “seen” the answers during pretraining; second, at least four high-risk vulnerabilities are invalid forgeries, with exploit descriptions that are actually unexecutable.
Q: Why is data contamination so dangerous for AI security evaluation?
A: If AI models have already encountered the benchmark’s vulnerability reports during pretraining, they might answer questions based on memory rather than genuine vulnerability discovery ability. This invalidates the “zero-knowledge” test, making it impossible to accurately assess AI’s real security auditing capabilities against entirely new, unknown smart contracts.
Q: What is OpenZeppelin’s attitude toward AI’s future in blockchain security?
A: They believe AI will significantly impact smart contract security but emphasize that this influence must be based on trustworthy methodologies and accurate evaluation standards. They see the issues with EVMbench not as a rejection of AI’s potential but as an important warning to the industry standards.
Disclaimer: The information on this page may come from third parties and does not represent the views or opinions of Gate. The content displayed on this page is for reference only and does not constitute any financial, investment, or legal advice. Gate does not guarantee the accuracy or completeness of the information and shall not be liable for any losses arising from the use of this information. Virtual asset investments carry high risks and are subject to significant price volatility. You may lose all of your invested principal. Please fully understand the relevant risks and make prudent decisions based on your own financial situation and risk tolerance. For details, please refer to
Disclaimer.
Related Articles
Ethereum Poisoning Attacks: How to Protect Yourself From Scammers - U.Today
Address poisoning attacks on Ethereum users are increasingly automated, deceiving victims into sending money to fake wallets. Recent data reveals significant losses and highlights the economic incentives driving these attacks, emphasizing the need for caution among users.
UToday5h ago
China's State Network Information Center Releases OpenClaw Security Risk Warning, with Approximately 23,000 Active Assets in the Domestic Market
Data from China's National Internet and Information Security Notification Center shows that there are over 200,000 active OpenClaw internet assets worldwide, with about 23,000 located within China, primarily concentrated in areas with dense network resources. These assets are exposed to security risks, and the behavior of agents is difficult to control, which could result in serious consequences such as data deletion and information theft.
GateNews7h ago
Ethereum Fees Drop Triggers Surge in Scams? Address Poisoning Attacks Skyrocket, USDT Micro Transactions Spike 612%
As Ethereum transaction costs decline, address poisoning attacks are becoming increasingly frequent. Attackers create counterfeit similar addresses and conduct small-value transfers to trick users into sending funds to the wrong address. After the Fusaka upgrade, small-value transactions surged, causing massive losses. Although the success rate of attacks is low, attackers continue to carry out these schemes due to low costs. Users need to carefully verify addresses and remain vigilant against such risks.
GateNews10h ago
OpenClaw Goes Viral in China's AI Trading Circle: Retail Traders Use AI to Trade Cryptocurrencies, Some Earning Nearly $3000 in 48 Hours
OpenClaw has recently gained rapid popularity in China, attracting numerous developers and investors to experiment with AI-automated trading. Its impact has spread across various demographics, but it also faces security vulnerabilities and reliability concerns. While it has lowered the technical barriers to trading, investment decisions still require human oversight, and market sentiment gradually shifts amid volatility.
GateNews10h ago
Slowmist: ClawHub Has Backdoor Implantation Risk, 21% of Top 100 Skills Listed as High Risk
Slow Mist Technology warns that ClawHub poses security risks due to its reliance on GitHub one-click login, which makes it susceptible to credential theft for supply chain attacks. GoPlus scanned the top 100 Skills and discovered that 21% contain critical risks. Additionally, Tencent's SkillHub has sparked copyright controversy, with founders criticizing it for failing to provide support to open-source projects. Users are advised to carefully select Skills and adopt security measures to prevent potential attacks.
MarketWhisper11h ago
