Anthropic has presented the results of a large-scale study on the use of modern artificial intelligence models to find vulnerabilities in smart contracts. During the testing, the models Claude Sonnet 4.5, Claude Opus 4.5, and GPT-5 were employed to assess the reliability of contracts based on the SCONE-bench dataset. This dataset contained bugs and exploits in Ethereum and BNB Chain contracts that emerged between 2020 and 2025.
This is reported by Finway
Identified Vulnerabilities and Testing Results
During the testing, the AI models successfully simulated exploits for nearly half of the historical incidents recorded in the sample. The total value of assets in the hacked contracts at the time of the attacks exceeded $550 million. Additionally, the Anthropic team conducted an analysis of contracts that were hacked after March 2025—meaning after the AI models had “cut off” their knowledge of these events. In this sample, the AI was able to identify 19 vulnerabilities out of 34, which corresponds to approximately $4.6 million in hypothetical losses.
“These cases were not known to the models in advance and contained several new types of defects,” company representatives noted.
The best performance was shown by the Claude Opus 4.5 model, which was able to simulate exploits for 17 out of 34 cases, representing a hypothetical “revenue” of $4.5 million. The other models—Claude Sonnet 4.5 and GPT-5—combined with Opus 4.5 identified 55.8% of the vulnerabilities from the test set, estimated at approximately $4.6 million in assets.
The Importance of Open Benchmarking and Its Impact on Security
Anthropic also tested the AI’s ability to find vulnerabilities in new smart contracts that had not previously been analyzed. As a result, two zero-day vulnerabilities were discovered on new addresses, which experts believe demonstrates the potential of artificial intelligence to detect errors without prior signals or historical data.
The company emphasizes that the goal of the research is not to exploit vulnerabilities but to create tools for assessing the effectiveness of AI systems in identifying code defects. Anthropic plans to use SCONE-bench as an open standard for testing and comparing the capabilities of large language models (LLMs).
Researchers note that these models can be beneficial for developers and auditors of smart contracts, helping to find errors before deployment on the blockchain. However, the company warns that the study is not a comprehensive risk assessment, as the analysis is limited to historical contracts and a controlled environment. Moving forward, Anthropic plans to expand the benchmark and explore the potential use of AI tools to enhance the security of blockchain protocols.
