Large language models for diabetes training: a prospective study

Li H 1 , Jiang Z 2 , Guan Z 3 , Bao Y 3 , Liu Y 3 , Hu T 3 Show all authors , Li J 4 , Liu R 4 , Wu L 3 , Cheng D 3 , Ji H 5 , Wang Y 6 , Wang YX 7 , Cheung CY 8 , Zheng Y 9 , Wang J 1 , Li Z 1 , Wu W 1 , Lim CC 10 , Bee YM 11 , Tan HC 11 , Ekinci EI 12 , Klonoff DC 13 , Echouffo-Tcheugui JB 14 , Mathioudakis N 15 , Corsino L 16 , Simó R 17 , Sabanayagam C 18 , Tan GSW 19 , Cheng CY 20 , Wong TY 21 , Li H 22 , Cai C 23 , Mao L 24 , Lim LL 25 , Tham YC 26 , Sheng B 27 , Jia W 28

Affiliations 

  • 1 Shanghai University of Sport, Shanghai 200438, China
  • 2 School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Tsinghua Medicine, Tsinghua University, Beijing 100084, China; Beijing Visual Science and Translational Eye Research Institute (BERI), Tsinghua Medicine, Tsinghua University, Beijing 100084, China
  • 3 Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
  • 4 Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China; MoE Key Lab of Artificial Intelligence, Artificial Intelligence Institute, Shanghai Jiao Tong University, Shanghai 200240, China
  • 5 Department of Cardiology, the Affiliated Hospital of Qingdao University, Qingdao 266011, China
  • 6 CEMS, NCMIS, HCMS, MADIS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematics, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Beijing 100049, China; Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China
  • 7 Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Medical University, Beijing Ophthalmology and Visual Sciences Key Laboratory, Beijing 100730, China
  • 8 Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong, China
  • 9 State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou 510060, China
  • 10 Department of Renal Medicine, Singapore General Hospital, SingHealth-Duke Academic Medical Centre, Singapore, 169608, Singapore
  • 11 Department of Endocrinology, Singapore General Hospital, Singapore, 169608, Singapore
  • 12 Department of Endocrinology, Austin Health, Melbourne, Heidelberg Victoria 3084, Australia; Department of Medicine, The University of Melbourne (Austin Health), Melbourne, Parkville Victoria 3010, Australia; Australian Centre for Accelerating Diabetes Innovations (ACADI), The University of Melbourne, Melbourne, Parkville Victoria, 3010, Australia
  • 13 Diabetes Research Institute, Mills-Peninsula Medical Center, San Mateo, CA 94010, USA
  • 14 Department of Medicine, Division of Endocrinology, Diabetes and Metabolism, Johns Hopkins School of Medicine, Baltimore, MD 21211, USA
  • 15 Division of Endocrinology, Diabetes, & Metabolism, Johns Hopkins University School of Medicine, Baltimore, MD 21211, USA
  • 16 Department of Medicine, Division of Endocrinology, Metabolism, and Nutrition, Duke University School of Medicine, Durham, NC 27710, USA
  • 17 Centro de Investigación Biomédica en Red de Diabetes y Enfermedades Metabólicas Asociadas (CIBERDEM), Instituto de Salud Carlos III, Madrid, 08035, Spain; Diabetes and Metabolism Research Unit, Vall d'Hebron Research Institut, Autonomous Univeristy of Barcelona, Barcelona, 08035, Spain
  • 18 Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, 168751, Singapore; Ophthalmology and Visual Science Academic Clinical Program, Duke-NUS Medical School, Singapore, 169857, Singapore
  • 19 Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, 168751, Singapore
  • 20 Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, 168751, Singapore; Ophthalmology and Visual Science Academic Clinical Program, Duke-NUS Medical School, Singapore, 169857, Singapore; Centre for Innovation and Precision Eye Health; and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, 119228, Singapore
  • 21 Beijing Visual Science and Translational Eye Research Institute (BERI), Tsinghua Medicine, Tsinghua University, Beijing 100084, China; Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, 168751, Singapore
  • 22 Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China. Electronic address: huarting99@sjtu.edu.cn
  • 23 Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China. Electronic address: 1540422623@qq.com
  • 24 Shanghai University of Sport, Shanghai 200438, China. Electronic address: maolijuan@sus.edu.cn
  • 25 Department of Medicine, Faculty of Medicine, Universiti Malaya, Kuala Lumpur, 50603, Malaysia; Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Hong Kong, China; Asia Diabetes Foundation, Hong Kong, China. Electronic address: leeling.lim@ummc.edu.my
  • 26 Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, 168751, Singapore; Ophthalmology and Visual Science Academic Clinical Program, Duke-NUS Medical School, Singapore, 169857, Singapore; Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, 119077, Singapore; Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, 119077, Singapore. Electronic address: thamyc@nus.edu.sg
  • 27 Shanghai University of Sport, Shanghai 200438, China; Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China; MoE Key Lab of Artificial Intelligence, Artificial Intelligence Institute, Shanghai Jiao Tong University, Shanghai 200240, China. Electronic address: shengbin@sjtu.edu.cn
  • 28 Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China. Electronic address: wpjia@sjtu.edu.cn
Sci Bull (Beijing), 2025 Jan 27.
PMID: 39947986 DOI: 10.1016/j.scib.2025.01.034

Abstract

Diabetes poses a considerable global health challenge, with varying levels of diabetes knowledge among healthcare professionals, highlighting the importance of diabetes training. Large Language Models (LLMs) provide new insights into diabetes training, but their performance in diabetes-related queries remains uncertain, especially outside the English language like Chinese. We first evaluated the performance of ten LLMs: ChatGPT-3.5, ChatGPT-4.0, Google Bard, LlaMA-7B, LlaMA2-7B, Baidu ERNIE Bot, Ali Tongyi Qianwen, MedGPT, HuatuoGPT, and Chinese LlaMA2-7B on diabetes-related queries, based on the Chinese National Certificate Examination for Primary Diabetes Care in China (NCE-CPDC) and the English Specialty Certificate Examination in Endocrinology and Diabetes of Membership of the Royal College of Physicians of the United Kingdom. Second, we assessed the training of primary care physicians (PCPs) without and with the assistance of ChatGPT-4.0 in the NCE-CPDC examination to ascertain the reliability of LLMs as medical assistants. We found that ChatGPT-4.0 outperformed other LLMs in the English examination, achieving a passing accuracy of 62.50%, which was significantly higher than that of Google Bard, LlaMA-7B, and LlaMA2-7B. For the NCE-CPFC examination, ChatGPT-4.0, Ali Tongyi Qianwen, Baidu ERNIE Bot, Google Bard, MedGPT, and ChatGPT-3.5 successfully passed, whereas LlaMA2-7B, HuatuoGPT, Chinese LLaMA2-7B, and LlaMA-7B failed. ChatGPT-4.0 (84.82%) surpassed all PCPs and assisted most PCPs in the NCE-CPDC examination (improving by 1 %-6.13%). In summary, LLMs demonstrated outstanding competence for diabetes-related questions in both the Chinese and English language, and hold great potential to assist future diabetes training for physicians globally.

* Title and MeSH Headings from MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine.

Similar publications