-Original-

Performance of a Large Language Model on Japanese Emergency Medicine Board Certification Examinations

Yutaka Igarashi¹, Kyoichi Nakahara¹, Tatsuya Norii², Nodoka Miyake¹, Takashi Tagami³ and Shoji Yokobori¹

¹Department of Emergency and Critical Care Medicine, Nippon Medical School, Tokyo, Japan
²Department of Emergency Medicine, University of New Mexico, NM, United States of America
³Department of Emergency and Critical Care Medicine, Nippon Medical School Musashi Kosugi Hospital, Kanagawa, Japan

Background: Emergency physicians need a broad range of knowledge and skills to address critical medical, traumatic, and environmental conditions. Artificial intelligence (AI), including large language models (LLMs), has potential applications in healthcare settings; however, the performance of LLMs in emergency medicine remains unclear.
Methods: To evaluate the reliability of information provided by ChatGPT, an LLM was given the questions set by the Japanese Association of Acute Medicine in its board certification examinations over a period of 5 years (2018-2022) and programmed to answer them twice. Statistical analysis was used to assess agreement of the two responses.
Results: The LLM successfully answered 465 of the 475 text-based questions, achieving an overall correct response rate of 62.3%. For questions without images, the rate of correct answers was 65.9%. For questions with images that were not explained to the LLM, the rate of correct answers was only 52.0%. The annual rates of correct answers to questions without images ranged from 56.3% to 78.8%. Accuracy was better for scenario-based questions (69.1%) than for stand-alone questions (62.1%). Agreement between the two responses was substantial (kappa = 0.70). Factual error accounted for 82% of the incorrectly answered questions.
Conclusion: An LLM performed satisfactorily on an emergency medicine board certification examination in Japanese and without images. However, factual errors in the responses highlight the need for physician oversight when using LLMs.

J Nippon Med Sch 2024; 91: 155-161

Keywords
artificial intelligence, emergency medicine, language, medicine, specialty boards

Correspondence to
Yutaka Igarashi, MD, PhD, Department of Emergency and Critical Care Medicine, Nippon Medical School, 1-1-5 Sendagi, Bunkyo-ku, Tokyo 113-8603, Japan
igarashiy@nms.ac.jp

Received, August 17, 2023
Accepted, October 4, 2023