Assessing the Reliability of AI-Powered Burns First Aid Advice: A Study of ChatGPT’s Accuracy

James Doherty¹, Bishoy Soliman²,
¹Prince Of Wales Hospital, Sydney , New South Wales, Australia
²Royal North Shore Hospital, St Leonards, Sydney, NSW, Australia

Abstract

We face a changing internet search landscape as the next generation of search engines incorporate conversational Artificial Intelligence (AI) to provide information specific to each enquiry – this is a shift away from the traditional list of linked resources ranked by relevancy. The end result is a computer generated plain text answer with an opaque origin of the referenced information, the validity of which has not been confirmed by any clinician

Aims: This study aims to assess the accuracy of burns first aid advice given by ChatGPT, a large language model trained by OpenAI, based on the GPT-3.5 architecture.

Methods: A sample of 20 burn-related questions was devised in plain language, and ChatGPT’s responses were evaluated based on the current first aid guidelines and compared with the performance of Google Snippets – the current standard in web search.

Results: Qualitative analysis identified several common areas where ChatGPT provided inaccurate or incomplete information, including the appropriate duration for cooling the burn or decontaminating a chemical burn. Furthermore the accuracy of the response changed with the length of response requested. Responses were often influenced by ambiguous or incomplete information in the question prompt. In contrast the performance of the Google Snippet feature was more accurate as it in most cases paraphrased guidelines.

Conclusion: Overall, the study suggests that ChatGPT’s burns first aid advice is can be accurate, but there is room for improvement in the specifics.
The study highlights the need for ensuring accurate information is available for ongoing patient education and public awareness of burns first aid. Stakeholder engagement with those training the AI models that will be deployed in the next generation of search engines is required to ensure we are providing clear and unambiguous information when seeking first aid advice from any other conversational AI Search engine.

Biography

Dr James Doherty is a post exam final year plastics trainee who’s previous life as a Mechanical engineer has led him to investigate the interface between surgery and technology.