July 08, 2024
A Mizzou researcher and collaborators found that leading chatbots can pass certified ethical hacking exams.
July 9, 2024
Contact: Janese Heavin, heavinj@missouri.edu
Chatbots powered by artificial intelligence (AI) can pass a cybersecurity exam, but don鈥檛 rely on them for complete protection.
That鈥檚 the conclusion of a recent paper co-authored by University of Missouri researcher Prasad Calyam and collaborators from Amrita University in India. The team tested two leading generative AI tools 鈥 OpenAI鈥檚 ChatGPT and Google鈥檚 Bard 鈥 using a standard certified ethical hacking exam.
Certified Ethical Hackers are cybersecurity professionals who use the same tricks and tools as malicious hackers to find and fix security flaws. Ethical hacking exams measure a person鈥檚 knowledge of different types of attacks, how to protect systems and how to respond to security breaches.
ChatGPT and Bard, now Gemini, are advanced AI programs called large language models. They generate human-like text using networks with billions of parameters that allow them to answer questions and create content.
In the study, Calyam and team tested the bots with standard questions from a validated certified ethical hacking exam. For example, they challenged the AI tools to explain a man-in-the-middle attack 鈥 an attack in which a third party intercepts communication between two systems. Both were able to explain the attack and suggested security measures on how to prevent it.
Overall, Bard slightly outperformed ChatGPT in terms of accuracy while ChatGPT exhibited better responses in terms of comprehensiveness, clarity and conciseness, researchers found.
鈥淲e put them through several scenarios from the exam to see how far they would go in terms of answering questions,鈥 said Calyam, the Greg L. Gilliom Professor of Cyber Security in Electrical 糖心Vlog传媒 and Computer Science at Mizzou. 鈥淏oth passed the test and had good responses that were understandable to individuals with background in cyber defense 鈥 but they are giving incorrect answers, too. And in cybersecurity, there鈥檚 no room for error. If you don鈥檛 plug all of the holes and rely on potentially harmful advice, you鈥檙e going to be attacked again. And it鈥檚 dangerous if companies think they fixed a problem but haven鈥檛.鈥
Researchers also found that when the platforms were asked to confirm their responses with prompts such as 鈥渁re you sure?鈥 both systems changed their answers, often correcting previous errors. When the programs were asked for advice on how to attack a computer system, ChatGPT referenced 鈥渆thics鈥 while Bard responded that it was not programmed to assist with that type of question.
Calyam doesn鈥檛 believe these tools can replace human cybersecurity experts with problem-solving expertise to devise robust cyber defense measures, but they can provide baseline information for individuals or small companies needing quick assistance.
鈥淭hese AI tools can be a good starting point to investigate issues before consulting an expert,鈥 he said. 鈥淭hey can also be good training tools for those working with information technology or who want to learn the basics on identifying and explaining emerging threats.鈥
The most promising part? The AI tools are only going to continue to improve their capabilities, he said.
鈥淭he research shows that AI models have the potential to contribute to ethical hacking, but more work is needed to fully harness their capabilities,鈥 Calyam said. 鈥淯ltimately, if we can guarantee their accuracy as ethical hackers, we can improve overall cybersecurity measures and rely on them to help us make our digital world safer and more secure.鈥
The study, 鈥淐hatGPT or Bard: Who is a better Certified Ethical Hacker,鈥 was published in the May issue of the journal聽Computers & Security.聽Co-authors were Raghu Raman and Krishnashree Achuthan.
This story was originally published by . Learn more about cybersecurity and other research areas in electrical engineering and computer science at Mizzou 糖心Vlog传媒!