Letter 2 regarding “Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma”

Article information

Clin Mol Hepatol. 2024;30(1):113-117

Publication date (electronic) : 2023 November 10

doi : https://doi.org/10.3350/cmh.2023.0440

Yiwen Zhang^1,*, Liwei Wu^2,*, Zepeng Mu^1,*, Linlin Ren^3,#, Ying Chen^4,#, Hanyun Liu^5,#, Lili Xu^1,#, Yangang Wang^1,#, Yaxing Wang^6,#, Susan Cheng^7,#, Yih Chung Tham^8,9,#, Bin Sheng^10,#, Tien Yin Wong^,9,11, Hongwei Ji^,11,12

¹Department of Endocrinology and Metabolic Hepatology, The Affiliated Hospital of Qingdao University, Qingdao, China

²Department of Gastroenterology and Hepatology, Shanghai East Hospital, Tongji University, Shanghai, China

³Department of Gastroenterology and Hepatology, The Affiliated Hospital of Qingdao University, Qingdao, China

⁴Shandong Provincial Key Laboratory of Metabolic Diseases and Qingdao Key Laboratory of Gout, the Affiliated Hospital of Qingdao University, Qingdao, China

⁵Department of Infectious Disease and Hepatology, The Affiliated Hospital of Qingdao University, Qingdao, China

⁶Beijing Key Laboratory of Ophthalmology and Visual Sciences, Beijing Tongren Eye Center, Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Medical University, Beijing, China

⁷Department of Cardiology, Smidt Heart Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA

⁸Yong Loo Lin School of Medicine, National University of Singapore, Singapore

⁹Singapore Eye Research Institute, Singapore National Eye Center, Singapore

¹⁰Department of Computer Science and Engineering, Shanghai JiaoTong University, Shanghai, China

¹¹Tsinghua Medicine, Tsinghua University, Beijing, China

¹²Department of Internal Medicine, Beijing Tsinghua Changgung Hospital, Beijing, China

Corresponding author : Tien Yin Wong Tsinghua Medicine, Tsinghua University, Haidian District, Beijing, 100084, China Tel: +86-10-62782835, Fax: +86-10-62782835, E-mail: wongtienyin@tsinghua.edu.cn

Hongwei Ji Tsinghua Medicine, Tsinghua University, Haidian District, Beijing, 100084, China Tel: +86-10-62782835, Fax: +86-10-62782835, E-mail: hongweijicn@gmail.com

*Contributed equally.

#LLM-Liver Investigators.

Editor: Ji Won Han, Catholic University of Korea, Korea

Received 2023 October 29; Revised 2023 November 8; Accepted 2023 November 10.

Dear Editor,

We read with great interest the recently published research analyzing the performances of ChatGPT with respect to the management of cirrhosis and hepatocellular carcinoma [1]. In addition to these advanced liver diseases, steatotic liver disease (SLD) also represents a considerable burden on global health, as it affects one-third of the worldwide population [2]. SLD requires long-term self-management and continuous support. This stems from its slow progression, the emphasis on lifestyle changes, and the constant need for regular patient-physician interactions. Therefore, for patients diagnosed with SLD, education plays a pivotal role in understanding, managing, and possibly reversing their condition. In our evolving digital era, large language models (LLMs), which are sophisticated generative AI systems trained on vast volumes of data that are capable of producing human-like textual responses, have emerged as promising aids for patient education [3], particularly in facilitating interactions through natural language dialogues [4]. However, given that the efficacy of LLMs in advancing SLD patient education might vary, it is imperative to compare their performances. Therefore, we conducted a comparative evaluation study to assess the performance of five leading LLMs in responding to SLD-related queries.

Our study was performed between Sep 8th and 28th, 2023. We curated 30 common SLD-related queries spanning domains such as risk factors, clinical test and diagnosis, treatment, follow-up, and prognosis based on guideline-based topics and our clinical experience (Table 1) [5,6]. As a separate and independent prompt, each query was posed to five LLMs: ChatGPT-3.5, ChatGPT-4, Google Bard, Meta Llama2 and Anthropic Claude2, which yielded a total of 30 responses per LLM-chatbot. The generated responses were then randomly ordered within each set of questions and stripped of revealing information (e.g., statements such as “I’m not a doctor” from ChatGPT) to blind reviewers to the LLM-specific response identity. Three seasoned attending-level physicians independently graded the responses as either “appropriate” or “inappropriate” over five rounds, each on a separate day, with an overnight washout interval in between to mitigate memory bias (Supplementary Fig. 1). Specifically, the responses were graded as “appropriate” when they were free from errors and “inappropriate” when they contained potential factual errors that could harm or mislead the average patient. The final grade for each chatbot response was determined using a majority consensus approach, based on the grade most often assigned by the three expert graders.

Table 1.

Performance of large language models in addressing patient queries regarding steatotic liver disease

We assessed the performances of the five LLMs in responding to SLD-related queries. As shown in Table 1, ChatGPT-4 provided 29 of 30 (96.7%) appropriate responses, followed by Bard and Llama2 with 27 of 30 (90.0%), and ChatGPT-3.5 and Claude2 both with 24 of 30 (80.0%), Chi-square test χ²=6.17, P=0.18. A notable area of concern was the frequent misclassification of fatty liver disease as synonymous with nonalcoholic fatty liver disease (NAFLD). This oversimplification can lead to inaccuracies. For example, ChatGPT-3.5 replied to the question “Are there different stages of fatty liver disease, and how do they differ?” with the following response: “Yes, there are different stages of fatty liver disease, which is also known as nonalcoholic fatty liver disease (NAFLD). …. The stages of NAFLD are typically categorized as follows: 1. Simple Steatosis (Fatty Liver): ….2. Nonalcoholic Steatohepatitis (NASH): .... 3. Fibrosis: …. 4. Cirrhosis: ….”

This rigorous evaluation study revealed that, among five state-of-the-art LLMs, ChatGPT-4 could generate largely appropriate responses to patient queries regarding SLD, boasting an impressive appropriateness rate of 96.7%. Other LLMs provided 80% to 90% appropriate responses. Health literacy—commonly defined as the degree to which individuals have the skills and abilities to obtain, process, and utilize health-related information—has emerged as a critical priority in reducing inequities among patients, including those with SLD [7,8]. Our findings underscore the varied potential of LLM chatbots to provide professional yet patient-friendly health literacy guidance to SLD patients [3]. Whereas prior investigations predominantly focused on ChatGPT3.5 [1], our study offers a comprehensive assessment of popular LLMs, namely ChatGPT-3.5, ChatGPT-4, Bard, Llama2 and Claude2, and we specifically evaluated their proficiency in addressing typical SLD-related patient queries. Notably, one in five responses from ChatGPT-3.5 and Claude2 was inappropriate, thus highlighting the need for further iterations and probably domain-specific fine-tuning. Although the exact parameters of ChatGPT-4 remain undisclosed, its impressive performance may result from the large parameter set, extensive user feedback, advanced reasoning abilities, and the integration of insights from previous models into the system [9]. This study derived benefits from implementing a robust study design with proper randomization, wash-out periods and a majority consensus grading process. However, there are also limitations. These sample queries may represent only a small proportion of real-world scenarios. In addition, as the field of LLM evolves at an unprecedented speed, future research is needed to confirm whether LLMs are adapting to new nomenclatures, such as metabolic dysfunction associated steatotic liver disease (MASLD). Generative AI with LLMs—especially ChatGPT-4—may offer yet further valuable insights into opportunities for patient education about SLDs.