Comparison of Large Language Models in Providing Medical Advice for Scald Burns: Google’s AI BARD, Bing AI, and ChatGPT
Ishith Seth1, David Hunter-Smith Warren Rozen 1Peninsula Health, Melbourne, Victoria, Australia
Abstract
Introduction:
Rapid advancements in artificial intelligence have led to the development of large language models (LLMs) that significantly improve the quality and accessibility of information in various domains, including medical advice. This study compares the effectiveness of three prominent LLMs – Google’s AI BARD, Bing AI, and ChatGPT – in providing medical advice for scald burns in simulated patient scenarios.
Methods:
A series of 5 diverse scald burn scenarios and questions were presented to these LLMs, and their ability to provide accurate, safe, and comprehensible advice was assessed by two Specialist Plastic Surgeons and surgical residents. Additionally, a Likert scale-based qualitative analysis was performed to compare the criteria.
Results:
While all three LLMs demonstrated reasonable proficiency in generating relevant and coherent advice, there were noticeable differences in their accuracy and safety. ChatGPT outperformed the other models in providing accurate and evidence-based medical advice, followed by AI BARD and Bing AI. However, AI BARD exhibited a slight advantage over ChatGPT in terms of user-friendliness satisfaction, attributed to its more conversational tone.
Conclusion:
This study emphasizes the potential of LLMs in enhancing access to medical advice for scald burns and highlights the need for continuous improvement in their accuracy and safety. It offers valuable insights for developing future AI-driven healthcare systems and underscores the importance of interdisciplinary collaboration in optimizing LLMs for specialized domains such as medicine.
Biography
Ishith Seth is a plastic surgical resident, master of Surgery candidate investigating 3D-printed carpal implants for carpal osteoarthritis. He is associated with Peninsula Health and Monash University. Honorary researcher at Department of Surgery, Royal Children’s Hospital, Murdoch Children’s Research Institute, and The University of Melbourne. He is a clinical tutor at Monash University and Melbourne University Medical Schools. His special interests include oculoplastics, microsurgery, reconstructive surgery, and artificial intelligence in healthcare.