Researchers from the University of Sheffield's Department of Computer Science identified how Text-to-SQL systems – AIs designed to search databases using plain language queries – can be exploited in real-world cyber crimes. These findings shed light on how AI systems might be manipulated to access sensitive information, tamper with databases, or initiate Denial-of-Service attacks.

Out of the six commercial AI tools evaluated – ChatGPT, BAIDU-UNIT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE – all were found to have security vulnerabilities. By asking these platforms specific questions, researchers could get them to generate malicious code. When executed, this code could disrupt database services, leak confidential data, or even destroy the database. For instance, on Baidu-UNIT, a Chinese dialogue customization app, the team obtained confidential server configurations and even took a server node offline.

"At the moment, ChatGPT is receiving a lot of attention. It's a standalone system, so the risks to the service itself are minimal, but what we found is that it can be tricked into producing malicious code that can do serious harm to other services," says Xutan Peng, a Ph.D. student at the University of Sheffield and co-leader of the research, in a university release.

One key concern raised by the study is the use of AI tools like ChatGPT for productivity.