GPT Analyzer

Current auditing techniques for smart contracts predominantly focus on identifying standard vulnerabilities like reentrancy and integer overflows through the employment of automated analysis tools, such as Slither [1] and Mythril [2]. Yet, a recent study has uncovered that these tools detect a mere 20% of exploitable vulnerabilities in the wild, highlighting a critical deficiency in the automated vulnerability detection capabilities [3].

In the rapidly evolving landscape of software engineering, Large Language Models (LLMs) [4] have begun to play a transformative role by enhancing code generation, comprehension, and rectification processes [5]. Utilizing LLMs to augment the auditing of smart contracts offers a promising avenue for application.

Consequently, we envision the development of LLM-enhanced techniques for detecting vulnerabilities in smart contracts and introduce GPT Analyzer. This innovative tool incorporates the advanced capabilities of GPT, fine-tuned with an extensive dataset of real-world exploitable vulnerabilities, to detect a broad spectrum of sophisticated, polymorphistic logical vulnerabilities. GPT Analyzer provides several distinct advantages over traditional auditing methods:

1. Enhanced Generality. While traditional automated tools rely on expertly crafted detectors based on fixed patterns of control or data flows, GPT Analyzer utilized a dataset of over 6000 real-world vulnerabilities, all of high or medium severity, sourced from esteemed auditing platforms like Code4rena [6] and Immunefi [7]. This enables GPT Analyzer to mimic human-like code interpretation and reasoning, thereby identifying a broader range of vulnerabilities, including those not previously categorized, and offering potential code repair solutions.
2. Superior Assurance. The manual inspection of logical vulnerabilities often requires complex reasoning, heightening the risk of crucial vulnerabilities going unnoticed in production—a staggering 91.96% of compromised smart contracts had been thoroughly audited [8]. GPT Analyzer, however, is fine-tuned with a comprehensive dataset encompassing all seven categories of logical vulnerabilities identified by Zhang et al. [3], such as price oracle manipulation and erroneous accounting. This extensive knowledge base allows GPT Analyzer to surpass the analytical capabilities of original LLMs. Moreover, by utilizing Chain-of-Thought Prompting [9] to facilitate intermediate reasoning about detected vulnerabilities—highlighting crucial variables and functions—GPT Analyzer not only gains a deep understanding of the underlying causes of these vulnerabilities but also significantly enhances its precision by validating these findings with specific static analysis rules.

[1] Feist, Josselin, Gustavo Grieco, and Alex Groce. "Slither: a static analysis framework for smart contracts." 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB). IEEE, 2019.

[2] Mueller, Bernhard. "Smashing ethereum smart contracts for fun and real profit." HITB SECCONF Amsterdam 9 (2018): 54.

[3] Zhang, Zhuo, et al. "Demystifying exploitable bugs in smart contracts." 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023.

[4] "Large language model." [Online]. Available: https://en.wikipedia.org/wiki/Large_language_model

[5] Hou, Xinyi, et al. "Large Language Models for Software Engineering: A Systematic Literature Review." arXiv e-prints (2023): arXiv-2308.

[6] "Code4rena." [Online]. Available: https://code4rena.com/

[7] "Immunefi." [Online]. Available: https://immunefi.com/

[8] "Smart Contract Audits Have Failed: Can We Solve the $2.8 Billion Smart Contract Security Problem?." [Online]. Available: https://www.anchain.ai/blog/smart-contract-audits-failed

[9] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.