Reappraisal of sepsis-3 and CLIF-SOFA as predictors of mortality in patients with cirrhosis and infection presenting to the emergency department: A multicenter study

Ji Hyun Kim; Baek Gyu Jun; Minjong Lee; Hye Ah Lee; Tae Suk Kim; Jeong Won Heo; Da Hye Moon; Seong Hee Kang; Ki Tae Suk; Moon Young Kim; Young Don Kim; Gab Jin Cheon; Soon Koo Baik; Dong Joon Kim; Dae Hee Choi

doi:10.3350/cmh.2021.0169

Clin Mol Hepatol > Volume 28(3); 2022 > Article

Kim, Jun, Lee, Lee, Kim, Heo, Moon, Kang, Suk, Kim, Kim, Cheon, Baik, Kim, and Choi: Reappraisal of sepsis-3 and CLIF-SOFA as predictors of mortality in patients with cirrhosis and infection presenting to the emergency department: A multicenter study

Original Article

Clin Mol Hepatol. 2022; 28(3): 540-552.

Published online: May 6, 2022

DOI: https://doi.org/10.3350/cmh.2021.0169

Reappraisal of sepsis-3 and CLIF-SOFA as predictors of mortality in patients with cirrhosis and infection presenting to the emergency department: A multicenter study

Ji Hyun Kim^1,^2,^*, Baek Gyu Jun^3,^*, Minjong Lee^4,⁵

, Hye Ah Lee⁶, Tae Suk Kim^1,², Jeong Won Heo¹, Da Hye Moon¹, Seong Hee Kang³, Ki Tae Suk⁷, Moon Young Kim⁸, Young Don Kim⁹, Gab Jin Cheon⁹, Soon Koo Baik⁸, Dong Joon Kim⁷, Dae Hee Choi^1,²

Corresponding author : Minjong Lee Department of Internal Medicine, Ewha Womans University Seoul Hospital, Ewha Womans University College of Medicine, 260 Gonghang-daero, Gangseo-gu, Seoul 07804, Korea
Tel: +82-2-6986-1761, Fax: +82-2-6986-5576, E-mail: minjonglee2@naver.com or minjonglee2@ewha.ac.kr

Dae Hee Choi Department of Internal Medicine, Kangwon National University School of Medicine, 156 Baengnyeong-ro, Chuncheon 24289, Korea
Tel: +82-33-258-9470, Fax: +82-33-258-2146, E-mail: dhchoi-md@kangwon.ac.kr

^*These two authors contributed equally to this work.

Editor: Salvatore Piano, University of Padova Faculty of Medicine and Surgery, Italy

Received June 14, 2021 Revised May 2, 2022 Accepted May 3, 2022

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

ABSTRACT

Background/Aims

Sepsis-3 criteria and quick Sequential Organ Failure Assessment (qSOFA) have been advocated to be used in defining sepsis in the general population. We aimed to compare the Sepsis-3 criteria and Chronic Liver Failure-SOFA (CLIF-SOFA) scores as predictors of in-hospital mortality in cirrhotic patients admitted to the emergency department (ED) for infections.

Methods

A total of 1,622 cirrhosis patients admitted at the ED for infections were assessed retrospectively. We analyzed their demographic, laboratory, and microbiological data upon diagnosis of the infection. The primary endpoint was inhospital mortality rate. The predictive performances of baseline CLIF-SOFA, Sepsis-3, and qSOFA scores for in-hospital mortality were evaluated.

Results

The CLIF-SOFA score proved to be significantly better in predicting in-hospital mortality (area under the receiver operating characteristic curve [AUROC], 0.80; 95% confidence interval [CI], 0.78–0.82) than the Sepsis-3 (AUROC, 0.75; 95% CI, 0.72–0.77, P<0.001) and qSOFA (AUROC, 0.67; 95% CI, 0.64–0.70; P<0.001) score. The CLIF-SOFA, CLIF-C-AD scores, Sepsis-3 criteria, septic shock, and qSOFA positivity were significantly associated with in-hospital mortality (adjusted hazard ratio [aHR], 1.24; 95% CI, 1.19–1.28; aHR, 1.13; 95% CI, 1.09–1.17; aHR, 1.19; 95% CI, 1.15–1.24; aHR, 1.88; 95% CI, 1.42–2.48; aHR, 2.06; 95% CI, 1.55–2.72; respectively; all P<0.001). For CLIF-SOFA scores ≥6, in-hospital mortality was >10%; this is the cutoff point for the definition of sepsis.

Conclusions

Among cirrhosis patients presenting with infections at the ED, CLIF-SOFA scores showed a better predictive performance for mortality than both Sepsis-3 criteria and qSOFA scores, and can be a useful tool of risk stratification in cirrhotic patients requiring timely intervention for infection.

Keywords: Liver cirrhosis; Sepsis; Hospital mortality; Bacterial infections

Study Highlights

• CLIF-SOFA is a simple tool used to predict organ dysfunction in cirrhotic patients with infection.

• Compared to Sepsis-3 and qSOFA score, CLIF-SOFA score is better at predicting in-hospital mortality of cirrhotic patients with infection.

• Higher CLIF-SOFA score is associated with higher in-hospital mortality in cirrhotic patients with infection.

Graphical Abstract

INTRODUCTION

Patients with liver cirrhosis are at a high risk of developing bacterial infections and sepsis [1,2], and have a four-fold increase in mortality compared to patients without infection [3]. Consequently, it is critical to accurately diagnose sepsis and stratify its mortality risk. A panel of experts has recently revised the consensus definitions for sepsis and septic shock (Sepsis-3) in the general population, focusing on organ dysfunction rather than systemic inflammation [4]. According to the Sepsis-3 criteria, sepsis is defined as an acute change (≥2 points) in the Sequential Organ Failure Assessment (SOFA) score, which is significantly associated with an in-hospital mortality rate greater than 10% [4]. In addition, the quick SOFA (qSOFA) score has been proposed as a simple bedside check for infection in adult patients with poor expected outcomes [5]. Both approaches have been validated within and outside intensive care units (ICUs) in the general population on a large-scale; however, they do not focus on cirrhosis patients [4,6,7]. Furthermore, the scores have neither been validated nor studied specifically in the emergency department (ED). For the Sepsis-3 criteria to be globally endorsed, external validation is essential in cirrhosis patients visiting the ED for infections.

To apply the Sepsis-3 criteria to define sepsis, baseline SOFA scores before infection are required. If information on the patient’s baseline SOFA scores are unavailable, a default score of 0 is attributed when they present at the ED for infection. Such assumptions may be wrong in cirrhosis patients since most of these patients would have at least a baseline score of 1 (not 0) due to low platelet counts, high bilirubin levels, or some degrees of hepatic encephalopathy.

To date, the Chronic Liver Failure-SOFA (CLIF-SOFA) score has been used and validated in cirrhosis patients. It helps to diagnose acute-on-chronic liver failure (ACLF) and to assess the prognosis [8-15]. Since the CLIF-SOFA score focuses on organ dysfunction, and infection is directly implicated in ACLF, this score may be useful in defining sepsis in cirrhosis patients. Besides, the CLIF-SOFA score is easily applicable and does not require baseline SOFA scores. However, the CLIF-SOFA score has yet to be assessed alongside the Sepsis-3 and qSOFA scores in a large cohort of cirrhosis patients who visit the ED for infections. The main purpose of this study was to assess the external validity of the Sepsis-3 criteria in cirrhosis patients presenting to the ED. Furthermore, we aimed to compare the predictive performances of the Sepsis-3 and CLIF-SOFA scores in predicting in-hospital mortality, and evaluate whether there is a significant association between the CLIF-SOFA score and in-hospital mortality in this population.

MATERIALS AND METHODS

Study population

For this retrospective analysis of a multicenter cohort, we assessed 1,622 consecutive cirrhosis patients with bacterial or fungal infections who visited the ED of four tertiary hospitals (Kangwon National University Hospital, Gangneung Asan Hospital, Wonju Severance Christian Hospital, and Chuncheon Sacred Heart Hospital) between January 2010 and December 2018. The inclusion criteria were as follows: (1) diagnostic features of cirrhosis including platelet count <150,000/mL, ultrasound features of cirrhosis (i.e., blunt liver edges and nodularity, with splenomegaly [>12 cm]), clinical signs of portal hypertension (ascites, esophageal or gastric varices), and hepatic encephalopathy [16]; (2) diagnosis of bacterial and/or fungal infection at the ED visit (detailed definitions of bacterial infections are shown in Supplementary Methods 1); (3) empirical initiation of antibiotics, and culture studies obtained at EDs for bacterial and/or fungal infections; (4) age ≥18 years; and (5) main reason for consultation being infection. We excluded the following: (1) patients aged <18 years; (2) patients who did not receive standard care based on the guidelines [17] (such as inappropriate antibiotics or no culture study); (3) patients who received prophylactic antibiotics; and (4) patients who were lost to follow-up (those for whom we were unable to assess mortality). Regarding the in-hospital mortality, we used medical records to confirm the date of death in each hospital. Regarding the mortality after hospital discharge, we used mortality data from the Korean National Health Insurance Service database, which covers the entire Korean population and provides information on the vital status of patients. Patient transfers (ward vs. ICU) were determined by the attending physicians based on the severity of organ dysfunction. The study protocol complied with the ethical guidelines of the World Medical Association Declaration of Helsinki, and was approved by the Institutional Review Boards of the participating hospitals.

Management of infection

Bacterial/fungal infections were characterized as community-acquired (CA) or healthcare-associated (HCA), as previously shown [18]. Once appropriate cultures were obtained, empiric antibiotic treatment was initiated based on the available international recommendations [17,19,20] and local epidemiological factors (high prevalence of resistance to quinolones, extended spectrum beta-lactamase-producing Enterobacteriaceae and Enterococci). In the event of positive cultures, antibiotic regimens were adjusted to align with the susceptibility testing. Septic shock resuscitation required the use of plasma expanders and vasopressors.

Endpoints and definitions

The primary endpoint was the in-hospital mortality rate. The secondary endpoints were the mortality rates at 1 and 3 months. We also evaluated whether the CLIF-SOFA score was an independent risk factor associated with in-hospital mortality. Index dates were the dates of initial patient presentation at the EDs with infection. Detailed definitions of bacterial infections are summarized in the Supplementary Methods 1, 2 [17,21-24]. Since the Sepsis-3 criteria defines sepsis in patients with infection when the in-hospital mortality rates are greater than 10% and septic shock is defined when in-hospital mortality rates are greater than 40% [5], we assessed the Sepsis-3 and CLIF-SOFA scores to evaluate the two cut-off values of in-hospital mortality rates (>10% for sepsis and >40% for septic shock) in cirrhosis patients with infections. The CLIF-SOFA scores and ACLF grading were calculated using the European Association for the Study of the Liver-Chronic Liver Failure Consortium [8]. The definitions of the Sepsis-3 criteria and qSOFA are explained in Supplementary Methods 2 [5]. For patients who acquired second infections, and/or for new admissions for infections during the study period, only the first episode of infection qualified for the analysis.

Statistical analysis

Numerical data were expressed as median (or mean) with interquartile range (or standard deviation), and categorical data as number (%). To compare the parameters, Student’s t-test or Mann-Whitney’s U test was used as needed for continuous variables, and chi-squared test was applied for categorical variables. Univariate and multivariate Cox regression analyses were performed to assess significant associations between the models and in-hospital mortality.

We assessed the performances of the CLIF-SOFA, Sepsis-3, and qSOFA scores as predictors of in-hospital, 1-month, and 3-month mortalities. Discrimination and calibration measures were also examined to assess the predictive performances of the models. Accuracy in predicting mortality was evaluated using the area under the receiver operating characteristic (AUROC) curve at each point in time (such as at hospital discharge, 1 month, and 3 months). The AUROC curves were then compared using the DeLong method to assess the association of each prognostic model with in-hospital mortality [25]. We included all variables of the baseline characteristics for univariable and multivariable analyses of in-hospital mortality. To avoid multicollinearity between variables and prognostic models, variables such as age, respiratory rate, body temperature, ACLF grade, median model for end-stage liver disease (MELD) score, Child-Pugh score, serum total bilirubin levels, serum creatinine levels, and platelet counts, included in each prognostic model were excluded from multivariable analyses of in-hospital mortality. Furthermore, it was difficult to perform multivariable analyses for all prognostic models simultaneously, since the models shared similar variables. Thus, we separately performed multivariable analyses of each prognostic model for in-hospital mortality. The Hosmer-Lemeshow goodness-of-fit test was used to assess calibrations of the CLIF-SOFA score, measuring the differences between the expected and observed outcomes (in-hospital mortality) over deciles of risk [26]. To define the optimal cut-off values for maximizing sensitivity and specificity, Youden’s index method was used [27]. Kaplan-Meier curves served as estimates of survival in each risk group according to the scores. The locally weighted scatterplot smoothing curve method was used to fit a smooth curve between two variables (CLIF-SOFA or Sepsis-3 score and in-hospital mortality; span =0.80) [28]. All tests were two-tailed and powered by a standard software (R freeware v3.3.3; R Foundation for Statistical Computing, Vienna, Austria), with statistical significance set at P<0.05.

RESULTS

Baseline characteristics

A total of 1,622 cirrhosis patients who presented to the ED for bacterial or fungal infections were enrolled; men were predominant (74.1%). The baseline characteristics of the study population are shown in Table 1. The mean age was 60 years. The causes of cirrhosis were primarily alcohol-related, followed by hepatitis B virus infection; and 37.1% of the patients had ACLF. The median MELD, SOFA, and CLIF-SOFA scores were 15, 4, and 5, respectively. Baseline SOFA scores were available for 1,316 patients (81.1%), and the median score was 2 (interquartile range [IQR], 1–4). The median serum albumin level was 2.9 g/dL, and the median platelet count was 103×10⁹/L. Increased C-reactive protein (CRP) levels were evident in most patients (median, 23 mg/L [upper normal value, 5 mg/L]).

During hospital stay, we recorded significantly lower mean arterial pressures and platelet counts in patients who died than in survivors. Patients who died during hospital stay were also more prone to have ascites and hepatic encephalopathy, as well as higher ACLF grades, heart rates, blood leucocyte counts, serum CRP level, MELD, Child-Pugh, SOFA, and CLIF-SOFA scores, compared to the survivors (Table 1).

The median time from the baseline SOFA score to the onset of bacterial infection (ED visit) was 85 days (IQR, 40–168 days). Sepsis, as defined by the Sepsis-3 criteria, was significantly more common in patients without (272 of 306 patients) than in those with (888 of 1,316 patients) baseline SOFA scores (88.9% vs. 67.5%; P<0.001). It was also more common in patients with (236 of 260 patients) than in those without (924 of 1,362 patients) qSOFA positivity (90.8% vs. 67.8%; P<0.001).

Characteristics of bacterial infections

Bacterial infections were CA in 1,466 patients (90.4%) and HCA in 156 patients (9.6%). The most common site of bacterial infection was the abdomen (61.3%), followed by the lungs (15.2%), urinary tract (11.8%), and skin or soft tissues (4.3%) (Table 2). At least one microbial culture was positive in 399 patients (24.6%), showing either gram-negative (72.2%) or gram-positive (27.3%) bacteria. The most common isolate was Escherichia coli (42.6%), followed by Klebsiella pneumoniae (16.5%) and Staphylococcus aureus (12.3%). Ninety-nine patients (24.8%) harbored infections due to multidrug-resistant (MDR) bacteria, and only 0.5% experienced fungal infections (Aspergillus and Candida species).

At the time of diagnosis of the infection, 231 patients (14.2%) showed qSOFA positivity. According to the Sepsis-3 criteria, 1,160 patients (71.5%) had sepsis and 87 (5.4%) had septic shock. When defining sepsis as 10% in-hospital mortality according to current guidelines (patients with CLIF-SOFA scores ≥6 in this study) [5], 663 patients (40.9%) had an in-hospital mortality >10%.

Predictive performances of the Sepsis-3, qSOFA, and CLIF-SOFA scores for in-hospital, 1-month, and 3-month mortalities

During in-hospital stay (median, 8 days; IQR, 5–15 days), the in-hospital mortality rate was 15.0% (244 of 1,622 patients). The median duration of follow-up for the study period (in-hospital duration + duration of follow-up after discharge) was 11.2 months (IQR, 2.7–27.9 months). One-month and 3-month mortality rates were 17.0% (n=275) and 25.2% (n=408), respectively. The AUROCs of the Sepsis-3, qSOFA, and CLIF-SOFA scores as predictors of in-hospital, 1-month, and 3-month mortalities are shown in Table 3 and Fig. 1. The CLIF-SOFA score (AUROC, 0.80; 95% CI, 0.78–0.82) was a better predictor of in-hospital mortality than the Sepsis-3 (AUROC, 0.75; 95% CI, 0.72–0.77; P<0.001) and qSOFA (AUROC, 0.67; 95% CI, 0.64–0.70; P<0.001) scores (Fig. 1A). Likewise, regarding 1-month mortality, the CLIF-SOFA score (AUROC, 0.77; 95% CI, 0.75–0.80) performed significantly better than did the Sepsis-3 (AUROC, 0.69; 95% CI, 0.66–0.71; P<0.001) and qSOFA (AUROC, 0.63; 95% CI, 0.61–0.66; P<0.001) scores (Fig. 1B). The predictive performance of the CLIF-SOFA scores for 3-month mortality was also better than that of the Sepsis-3 and qSOFA scores (all P<0.001; Fig. 1C). The predictive performances of the CLIF-SOFA score for in-hospital, 1-month, and 3-month mortality were significantly higher than those of the static SOFA score at the time of diagnosis of the infection (Table 3, P<0.001). The predictive performance of the CLIF-SOFA for in-hospital mortality (AUROC, 0.80; 95% CI, 0.78–0.82) was also significantly better than those of the CLIF-C ACLF (AUROC, 0.77; 95% CI, 0.75–0.79; P=0.01) and CLIF-C-AD (AUROC, 0.74; 95% CI, 0.72–0.76; P<0.001) scores. Therefore, the CLIF-SOFA scores had the highest performance for predicting the in-hospital, 1-month, and 3-month mortalities when compared to the Sepsis-3 and qSOFA scores. Regarding the calibration of the CLIF-SOFA score, the probabilities of in-hospital, 1-month, and 3-months mortalities predicted by the CLIF-SOFA scores were not significantly different from the observed mortality rates (P=0.28 for in-hospital mortality [Supplementary Fig. 1]; P=0.23 for 1-month mortality, and P=0.81 for 3-month mortality).

Based on the CLIF-SOFA score, Sepsis-3 criteria, and qSOFA positivity, the in-hospital mortality rates of patients were 29.5%, 18.5%, and 35.5%, respectively (Table 4). The CLIF-SOFA score showed a significant linear relationship with in-hospital mortality (r=0.97; 95% CI, 0.94–0.99; P<0.001) (Fig. 2A). According to the Sepsis-3 criteria, in-hospital mortality rates for sepsis and septic shock were 10% and 40% respectively [5], whereas the CLIF-SOFA cut-off values were 6 for sepsis and 10 for septic shock (Fig. 2A). Although the Sepsis-3 criteria were significantly correlated with in-hospital mortality (r=0.90, 95% CI, 0.78–0.96; P<0.001), the observed relationship was not linear (Fig. 2B). The cut-off SOFA score was 4 for in-hospital mortality of >10% (sepsis) and 9 for in-hospital mortality of >40% (septic shock).

In multivariable Cox regression analyses, the CLIF-SOFA score, CLIF-C-AD score, Sepsis-3 criteria, septic shock, and qSOFA positivity were significantly associated with in-hospital mortality (adjusted hazard ratio [aHR], 1.24; 95% CI, 1.19–1.28; aHR, 1.13; 95% CI, 1.09–1.17; aHR, 1.19; 95% CI, 1.15–1.24; aHR, 1.88; 95% CI, 1.42–2.48; aHR, 2.06; 95% CI, 1.55–2.72; respectively, all P<0.001; Supplementary Tables 1-5).

Predictive performances of Sepsis-3, qSOFA, and CLIF-SOFA scores in patients who had positive cultures

Among patients who had positive culture studies (n=399), the CLIF-SOFA score was a better predictor of in-hospital mortality (AUROC, 0.82; 95% CI, 0.77–0.86) compared to the Sepsis-3 (AUROC, 0.75; 95% CI, 0.70–0.80; P=0.003) and qSOFA (AUROC, 0.65; 95% CI, 0.60–0.71; P<0.001; Supplementary Table 6) scores. Regarding the 1-month mortality, the CLIF-SOFA score (AUROC, 0.81; 95% CI, 0.76–0.86) significantly outperformed the Sepsis-3 (AUROC, 0.75; 95% CI, 0.70–0.80; P=0.01) and qSOFA (AUROC, 0.67; 95% CI, 0.61–0.72; P<0.001) scores. The predictive performance of CLIF-SOFA scores for 3-month mortality (AUROC, 0.78; 95% CI, 0.72–0.82) was also better than that of the Sepsis-3 (AUROC, 0.72; 95% CI, 0.66–0.77; P=0.01) and qSOFA (AUROC, 0.64; 95% CI, 0.58–0.70; P<0.001) scores. Therefore, in patients with confirmed infections on culture studies, the CLIF-SOFA scores had the highest performance for predicting the in-hospital, 1-month, and 3-month mortalities compared to the Sepsis-3 and qSOFA scores.

Risk stratification of survival according to the Sepsis-3, septic shock, and CLIF-SOFA scores

The cumulative 28-day mortality differed significantly with respect to the Sepsis-3 criteria as follows: absence of sepsis (7.6%, 35 out of 462 patients), positive Sepsis-3 criteria (17.6%, 186 out of 1,055 patients), and septic shock (51.4%, 54 out of 105 patients; all P<0.001 using the log-rank test) (Fig. 3A). Compared to patients without sepsis, the in-hospital mortality risk was significantly higher in patients with septic shock (HR, 8.18; 95% CI, 5.22–12.83; P<0.001) and those with positive Sepsis-3 criteria (HR, 2.14; 95% CI, 1.44–3.17; P<0.001).

Based on the CLIF-SOFA scores, the cumulative 28-day mortality differed significantly for scores <6 (6.0%, 56 out of 937 patients), scores ≥6 but <10 (23.3%, 115 out of 493 patients), and scores ≥10 (54.2%, 104 out of 192 patients; all P<0.001 by using the log-rank test) (Fig. 3B). In-hospital mortality risk was significantly greater at CLIF-SOFA scores ≥6 but <10 (HR, 3.53; 95% CI, 2.44–5.09; P<0.001) and at scores ≥10 (HR, 10.98; 95% CI, 7.67–15.70; P<0.001), when compared to scores <6.

DISCUSSION

Accurate diagnosis and prognostication of sepsis in the ED are important for timely management of cirrhosis patients with bacterial infections. In this study, we found that the Sepsis-3 criteria in cirrhosis patients who visited the ED for infections had limitations with respect to acute diagnosis of sepsis and risk stratification. When applying the Sepsis-3 criteria in these patients, the prevalence of sepsis was quite high (71.5% of our study cohort). The high prevalence of “sepsis” in this study was in line with the results of previous studies [29,30]. The proportion of patients who met the Sepsis-3 criteria and died during hospital stay was 18.5% (215 out of 1,160 patients), which was higher than the usual in-hospital mortality rate of 10% suggested in the Sepsis-3 criteria [5].

Another issue is that 18.9% of the total population in this study lacked baseline SOFA scores. In such cases, the baseline SOFA score was defaultly taken to be 0, as recommended in the guidelines [5]. However, this may not be the case for cirrhosis patients with low platelet counts and high serum bilirubin levels. Furthermore, among the patients with available baseline SOFA scores, 50.2% had baseline SOFA scores ≥2. Therefore, a default value of 0 may be an improper assumption in patients with cirrhosis.

Interestingly, sepsis, as defined by the Sepsis-3 criteria, was significantly more frequent in patients without an available baseline SOFA score (88.9%) than in those with a baseline SOFA score (67.5%). When cirrhosis patients did not have a baseline SOFA score, sepsis could be overestimated. Furthermore, although there was a 2.8-month interval between the baseline SOFA scoring and the SOFA scores generated at the time of infection in this study, the criteria are vague (with respect to the appropriate time frames for baseline score calculations) when assessing SOFA disparities. In real practice, it may be complicated to accurately define sepsis using the Sepsis-3 criteria due to the dual scoring at baseline and ED visit, inappropriate assumption of baseline SOFA score, and no recommended time interval for dual scoring in cirrhosis patients with bacterial infections.

The findings of the CANONIC study clearly indicate that organ dysfunction is a strong predictor of mortality in patients with acute decompensation of cirrhosis [8], and the CLIF-SOFA score has been validated as a predictor of ACLF-related mortality. Given that bacterial infection is one of the pivotal precipitating events in ACLF, prompt diagnosis of sepsis in the ED is crucial. In this study, the CLIF-SOFA scores showed a significant linear relationship with in-hospital mortality in cirrhosis patients with infections, significantly outperforming both the Sepsis-3 and qSOFA scores as predictors of in-hospital mortality. Regarding the in-hospital mortality cut-offs of Sepsis-3 (sepsis=10%; septic shock=40%) for patients with infection, we also determined the corresponding optimal CLIF-SOFA cut-offs to be 6 and 10, respectively, for risk stratification during in-hospital stays. In this study, in-hospital mortality risk was significantly 3.5 times higher at CLIF-SOFA scores ≥6 and 11.0 times higher at scores ≥10 than at scores <6.

Current guidelines require the following three steps for evaluating cirrhosis patients with infections: 1) evaluating the SOFA score before infection to evaluate the underlying organ dysfunction, 2) re-evaluating the SOFA score at the onset of infection to evaluate the present organ dysfunction, and 3) evaluating the qSOFA score for rapid prognostication [19]. However, given the excellent predictive/prognostic performance of CLIF-SOFA scores in this study, this algorithm may effectively be reduced to the following (using a one-time CLIF-SOFA score evaluation): CLIF-SOFA score <6, no sepsis; CLIF-SOFA score ≥6 but <10, sepsis; and CLIF-SOFA score ≥10, septic shock. After adopting a CLIF-SOFA score >6 to define sepsis, the proportion of patients with sepsis was 41%, which is lower than the 66% generated using the Sepsis-3 criteria. Therefore, CLIF-SOFA score evaluation may help to accurately identify patients at high risk for sepsis who are in need of intensive care, regardless of the availability of the baseline SOFA scores.

This study had several limitations that should be acknowledged. First, there may be a possibility of selection bias. The study only included patients with CA infections visiting the ED, and not those with nosocomial infections. In addition, the majority of infections were abdominal (61.3%), though the distribution of infections can differ based on epidemiological variations [22,31]. This could also account for selection bias, and may limit the generalizability of this study. However, a recent study reported that the most common type of infection in Asian patients with cirrhosis was spontaneous bacterial peritonitis (SBP) [21]. Prevalence of SBP in Asian patients with cirrhosis and infection was 41%. In this study, 44.8% of the cirrhosis patients (n=726) had SBP, and 16.5% (n=268) had either infectious colitis or colonic diverticulitis. Thus, the prevalence of SBP as the main type of infection in this study was similar to the results of the previous study [21]. In addition, there might be possibilities of selection bias originating from the exclusion criteria of this study. The findings of this study should be interpreted with caution as it excluded patients younger than 18 years of age, those not receiving standard care based on the guidelines, and those treated with prophylactic antibiotics. Second, there was a lack of geographical heterogeneity, which can explain the difference in the prevalence of ACLF and death from bacterial infection obtained from different regions [21]. This study only included Asian patients. Further studies are needed to fully validate the predictability of the CLIF-SOFA score for in-hospital mortality compared to that of the Sepsis-3 criteria used for Caucasian patients with cirrhosis and infection in Western countries. However, the outcomes of an earlier Western study involving a smaller cohort already suggested that the CLIF-SOFA score in cirrhosis patients with bacterial infection was a better predictor of 30-day mortality than the Sepsis-3 criteria [29]. Finally, other precipitating events other than bacterial infection have not been well considered. Prior to the onset of the bacterial infections, various factors may have influenced ACLF and mortality. In real practice, it may be difficult to precisely identify the precipitating factors other than infection in patients with cirrhosis and infection. Even in such cases, the CLIF-SOFA scores (reflecting acute decompensation from other precipitating factors) could be more helpful in stratifying high-risk patients than the Sepsis-3 criteria.

In conclusion, the CLIF-SOFA score showed better predictability for mortality than the Sepsis-3 criteria and the qSOFA score in cirrhosis patients who visited the ED for infections. The CLIF-SOFA score can be a simple and useful one-time tool for accurate risk stratification in patients with cirrhosis who need timely intervention for infection at the ED.

ACKNOWLEDGMENTS

This study has been worked with the support of a research grant of the Kangwon National University in 2017. This study was also supported by Young Investigator Grant from Department of Internal Medicine, Ewha University Medical Center, and supported by a grant from Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (grant number: 2017R1D1A1B03031499), and a grant and supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science and ICT) (grant number: 2020R1C1C1004112).

FOOTNOTES

Authors’ contributions

Study concept and design: Minjong Lee

Acquisition of data: Minjong Lee, Ji Hyun Kim, Young Don Kim, Baek Gyu Jun, Tae Suk Kim, Ki Tae Suk, Seong Hee Kang, Moon Young Kim, Gab Jin Cheon, Dong Joon Kim, Soon Koo Baik, Dae Hee Choi, Jeong Won Heo, Da Hye Moon

Analysis and interpretation of data: Minjong Lee, Ji Hyun Kim, Hye Ah Lee

Drafting of the manuscript: Minjong Lee, Baek Gyu Jun, Ji Hyun Kim, Dae Hee Choi

Critical revision of the manuscript for important intellectual content: Minjong Lee, Ji Hyun Kim, Young Don Kim, Baek Gyu Jun, Tae Suk Kim, Ki Tae Suk, Seong Hee Kang, Moon Young Kim, Gab Jin Cheon, Dong Joon Kim, Soon Koo Baik, Dae Hee Choi

Obtained funding: Minjong Lee

Administrative, technical, or material support: Minjong Lee, Ji Hyun Kim, Young Don Kim, Baek Gyu Jun, Tae Suk Kim, Ki Tae Suk, Seong Hee Kang, Moon Young Kim, Gab Jin Cheon, Dong Joon Kim, Soon Koo Baik, Dae Hee Choi

Study supervision: Minjong Lee, Dae Hee Choi

Conflicts of Interest

The authors have no conflicts to disclose.

SUPPLEMENTAL MATERIAL

Supplementary material is available at Clinical and Molecular Hepatology website (http://www.e-cmh.org).

Supplementary Methods 1.

Definition of bacterial infections

cmh-2021-0169-suppl1.pdf

Supplementary Methods 2.

Definition of Sepsis-3 criteria and qSOFA

cmh-2021-0169-suppl2.pdf

Supplementary Table 1.

Univariable and multivariable analyses for in-hospital mortality of CLIF-SOFA score^*

cmh-2021-0169-suppl3.pdf

Supplementary Table 2.

Univariable and multivariable analyses for in-hospital mortality of CLIF-C-AD score^*

cmh-2021-0169-suppl4.pdf

Supplementary Table 3.

Univariable and multivariable analyses for in-hospital mortality of Sepsis-3 criteria^*

cmh-2021-0169-suppl5.pdf

Supplementary Table 4.

Univariable and multivariable analyses for in-hospital mortality of septic shock^*

cmh-2021-0169-suppl6.pdf

Supplementary Table 5.

Univariable and multivariable analyses for in-hospital mortality of qSOFA positivity^*

cmh-2021-0169-suppl7.pdf

Supplementary Table 6.

Comparison of predictive performance for mortality in patients with positive culture results (n=399)

cmh-2021-0169-suppl8.pdf

Supplementary Figure 1.

Calibration plot of CLIF-SOFA scores. Diagonal line is the line of perfect calibration. Dots represent the intersects of observed and expected values for the risk of mortality, and bars represent the 95% confidence interval of the observed probability. Blue line indicates the calibration trend; since this line is close to the diagonal line, it indicates that the CLIF-SOFA fits the data well. CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment.

cmh-2021-0169-suppl9.pdf

Figure 1.

Receiver operating characteristic curve for the CLIF-SOFA scores, Sepsis-3, and qSOFA as predictors of (A) in-hospital mortality; (B) 1-month mortality; and (C) 3-month mortality, CLIF-SOFA significantly surpassing all other methods at all time-points (P<0.001 by the DeLong test) in patients with cirrhosis and infection CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment.

Figure 2.

Correlation of (A) CLIF-SOFA scores and (B) Sepsis-3 scores with in-hospital mortality in patients with cirrhosis and infection. Given that the in-hospital mortality rate was 10% for diagnosis of sepsis and 40% for diagnosis of septic shock according to the definition of sepsis, the cut-off point of CLIF-SOFA score was 6 for sepsis and 10 for septic shock. The cut-off point of SOFA score was 4 for sepsis and 9 for septic shock. CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment.

Figure 3.

Kaplan-Meier estimate of 28-day survival in patients with cirrhosis and infection: (A) significantly lower survival in the presence of septic shock or sepsis based on Sepsis-3 criteria compared to the absence of sepsis; and (B) significantly lower survival at CLIF-SOFA scores ≥6 but <10 or CLIF-SOFA scores ≥10 compared to CLIF-SOFA scores <6. CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment.

Table 1.

Baseline characteristics

Variable	Total population (n=1,622)	In-hospital mortality (-) (n=1,378)	In-hospital mortality (+) (n=244)	P-value^‡
Age (years)	60±13	60±13	62±14	0.04
Gender, male	1,202 (74.1)	1,012 (73.4)	190 (77.9)	0.15
Etiology of cirrhosis				0.40
Alcohol	949 (58.5)	809 (58.7)	140 (57.4)
HBV	338 (20.8)	294 (18.1)	44 (18.0)
HCV	91 (5.6)	74 (5.4)	17 (7.0)
NAFLD	218 (13.4)	181 (13.1)	37 (15.2)
Others^*	26 (1.6)	20 (1.5)	6 (2.5)
Diabetes	491 (30.3)	429 (31.1)	62 (25.4)	0.08
Body mass index (kg/m²)	22.8 (20.6–25.4)	22.9 (20.6–25.4)	22.3 (20.4–25.0)	0.21
Mean arterial pressure (mmHg)	87 (74–101)	89 (76–102)	79 (65–94)	<0.001
Heart rate (beat/minute)	95 (81–111)	94 (80–110)	100 (84–116)	0.002
Body temperature (℃)	36.8 (36.2–37.5)	36.8 (36.3–37.5)	36.5 (35.9–37.2)	<0.001
Respiratory rate (breath/min)	20 (18–20)	20 (18–20)	20 (19–22)	<0.001
Ascites	734 (45.3)	607 (44.0)	127 (52.0)	0.02
Hepatic encephalopathy	330 (20.3)	239 (17.3)	91 (37.3)	<0.001
ACLF	601 (37.1)	411 (29.8)	190 (77.9)	<0.001
ACLF grade^†				<0.001
No ACLF	1,021 (62.9)	967 (70.2)	54 (22.1)
Grade 1	401 (24.7)	314 (22.8)	87 (35.7)
Grade 2	132 (8.1)	76 (5.5)	56 (23.0)
Grade 3	68 (4.2)	21 (1.5)	47 (19.3)
INR	1.4 (1.2–1.6)	1.3 (1.2–1.6)	1.7 (1.4–2.3)	<0.001
Total bilirubin (mg/dL)	2.1 (1.1–4.3)	1.9 (1.0–3.8)	3.8 (1.8–8.6)	<0.001
Albumin (g/dL)	2.9 (2.5–3.4)	3.0 (2.6–3.5)	2.4 (2.0–2.9)	<0.001
Creatinine (mg/dL)	1.0 (0.8–1.6)	1.0 (0.7–1.4)	1.8 (1.0–2.9)	<0.001
Sodium (mEq/L)	136 (131–139)	136 (132–139)	134 (128–138)	<0.001
Platelet (×10⁹/L)	103 (69–159)	104 (69–158)	92 (60–160)	0.04
Leukocyte (×10⁹/L)	8.0 (5.5–12.0)	7.8 (5.4–11.3)	10.0 (6.0–14.7)	<0.001
C-reactive protein (mg/L)	23 (8–73)	20 (8–68)	47 (13–106)	<0.001
MELD score	15 (11–21)	14 (11–19)	23 (18–31)	<0.001
Child-Pugh score	8 (6–9)	7 (6–9)	9 (8–11)	<0.001
SOFA score before ER visit^§	2 (1–4)	1 (1–4)	2 (1–4)	0.006
SOFA score	4 (3–6)	4 (2–6)	7 (5–11)	<0.001
Sepsis-3 criteria (increase in SOFA ≥2)	1,160 (71.5)	945 (68.6)	215 (88.1)	<0.001
qSOFA ≥2	231 (14.2)	146 (10.6)	85 (34.8)	<0.001
CLIF-SOFA score^†	5 (3–7)	4 (3–6)	9 (6–12)	<0.001
CLIF-C-AD score^†	56.1 (50.0–63.6)	54.9 (49.3–61.6)	64.9 (57.1–73.1)	<0.001

Values are presented as mean±standard deviation, median (interquartile range), or number (%).

HBV, hepatitis B virus; HCV, hepatitis C virus; NAFLD, nonalcoholic fatty liver disease; ACLF, acute-on-chronic liver failure; INR, international normalized ratio for prothrombin time; MELD, model for end-stage liver disease; SOFA, Sequential Organ Failure Assessment; ER, emergency room; qSOFA, quick Sequential Organ Failure Assessment; CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment; CLIF-C-AD, Chronic Liver Failure Consortium Acute Decompensation score.

^* Others included the causes of autoimmune hepatitis, primary biliary cholangitis, and cryptogenic cirrhosis.

^† ACLF defined according to the European Association for the Study of the Liver-Chronic Liver Failure Consortium (EASL-CLIF).

^‡ P-values estimated by chi-squared test or Fisher’s exact test for categorical variables, and by Mann-Whitney U test or Kruskal-Wallis test for continuous variables: a comparison of baseline characteristics of survivors versus non-survivors.

^§ Available in 1,316 patients.

Table 2.

Clinical and microbiological characteristics of bacterial infections

Variable	Value (n=1,622)
Site of infection
Abdominal	994 (61.3)
Pulmonary	247 (15.2)
Urinary tract	192 (11.8)
Skin and soft tissue	70 (4.3)
Catheter-related	5 (0.3)
Others	114 (7.0)
Type of infection
Community-acquired	1,466 (90.4)
Healthcare-associated	156 (9.6)
Positive qSOFA score	231 (14.2)
Sepsis according to Sepsis-3 criteria	1,160 (71.5)
Septic shock	87 (5.4)
Sepsis^* according to in-hospital mortality of 10%	663 (40.9)
MELD score	19 (13–26)
Positive microbiological cultures	399 (24.6)
Type of strains isolated
Gram-positive	109 (27.3)
Gram-negative	288 (72.2)
Fungi	2 (0.5)
Type of bacteria isolated
Escherichia coli	170 (42.6)
Klebsiella pneumoniae	66 (16.5)
Enterococcus faecium	3 (0.8)
Enterococcus faecalis	13 (3.3)
Staphylococcus aureus	49 (12.3)
Pseudomonas aeruginosa	10 (2.5)
Other Enterobacteriaceae	10 (2.5)
Other Streptococci	39 (9.8)
Multimicrobial	16 (4.0)
Others	26 (6.5)
Multidrug-resistant bacteria	99 (24.8)

Values are presented as median (interquartile range) or number (%).

qSOFA, quick Sequential Organ Failure Assessment; MELD, model for end-stage liver disease; CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment.

^* Sepsis was defined when in-hospital mortality rates were greater than 10%, not based on the Sepsis-3 criteria. CLIF-SOFA score was 6 when in-hospital mortality rates were greater than 10%.

Table 3.

Comparison of predictive performance for mortality (n=1,622)

Prediction model	AUROC (95% CI)	P-value^*	Sensitivity (%)	Specificity (%)
In-hospital mortality
CLIF-SOFA (cut-off=6)	0.80 (0.78–0.82)	Reference	66.0	78.7
Sepsis-3 (cut-off=2)	0.75 (0.72–0.77)	<0.001	64.2	74.8
qSOFA (cut-off=2)	0.67 (0.64–0.70)	<0.001	39.6	86.7
Static SOFA (cut-off=3)	0.78 (0.74–0.71)	<0.001	69.7	73.6
1-month mortality
CLIF-SOFA (cut-off=6)	0.77 (0.75–0.80)	Reference	61.3	78.8
Sepsis-3 (cut-off=2)	0.69 (0.66–0.71)	<0.001	57.1	74.1
qSOFA (cut-off=2)	0.63 (0.61–0.66)	<0.001	71.3	48.2
Static SOFA (cut-off=3)	0.75 (0.72–0.79)	<0.001	65.5	73.7
3-month mortality
CLIF-SOFA (cut-off=6)	0.75 (0.72–0.77)	Reference	66.1	69.9
Sepsis-3 (cut-off=2)	0.66 (0.63–0.69)	<0.001	51.6	75.8
qSOFA (cut-off=2)	0.60 (0.57–0.63)	<0.001	66.4	48.6
Static SOFA (cut-off=3)	0.71 (0.68–0.74)	<0.001	57.6	75.4

AUROC, area under the receiver operating characteristics; CI, confidence interval; CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment; SOFA, Sequential Organ Failure Assessment.

^* P-values estimated by the Delong test.

Table 4.

Mortality rates according to CLIF-SOFA, Sepsis-3 criteria, and qSOFA

Prognostic model	In-hospital mortality	1-month mortality	3-month mortality
CLIF-SOFA	202 (29.5)	219 (32.0)	290 (42.3)
Sepsis-3 criteria	215 (18.5)	240 (20.7)	345 (29.7)
qSOFA	85 (35.5)	93 (40.3)	104 (45.0)

Values are presented as number (%).

CLIF-SOFA, Chronic Liver Failure-Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment.

Abbreviations

ACLF

acute-on-chronic liver failure

aHR

adjusted hazard ratio

AUROC

area under the receiver operating characteristic

community-acquired

confidence interval

CLIF-C-AD

Chronic Liver Failure Consortium Acute Decompensation score

CLIF-SOFA

Chronic Liver Failure-Sequential Organ Failure Assessment

CRP

C-reactive protein

emergency department

HCA

healthcare-associated community-acquired

ICU

intensive care unit

IQR

interquartile range

MDR

multidrug-resistant

MELD

model for end-stage liver disease

qSOFA

quick Sequential Organ Failure Assessment

SBP

spontaneous bacterial peritonitis

SOFA

Sequential Organ Failure Assessment

REFERENCES

1. Gustot T, Durand F, Lebrec D, Vincent JL, Moreau R. Severe sepsis in cirrhosis. Hepatology 2009;50:2022-2033.

2. Piano S, Tonon M, Angeli P. Changes in the epidemiology and management of bacterial infections in cirrhosis. Clin Mol Hepatol 2021;27:437-445.

3. Arvaniti V, D’Amico G, Fede G, Manousou P, Tsochatzis E, Pleguezuelo M, et al. Infections in patients with cirrhosis increase mortality four-fold and should be used in determining prognosis. Gastroenterology 2010;139:1246-1256.e12565.

4. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016;315:762-774.

5. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016;315:801-810.

6. Freund Y, Lemachatti N, Krastinova E, Van Laer M, Claessens YE, Avondo A, et al. Prognostic accuracy of sepsis-3 criteria for inhospital mortality among patients with suspected infection presenting to the emergency department. JAMA 2017;317:301-308.

7. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA 2017;317:290-300.

8. Moreau R, Jalan R, Gines P, Pavesi M, Angeli P, Cordoba J, et al. Acute-on-chronic liver failure is a distinct syndrome that develops in patients with acute decompensation of cirrhosis. Gastroenterology 2013;144:1426-1437.e14379.

9. Pan HC, Jenq CC, Tsai MH, Fan PC, Chang CH, Chang MY, et al. Scoring systems for 6-month mortality in critically ill cirrhotic patients: a prospective analysis of chronic liver failure - sequential organ failure assessment score (CLIF-SOFA). Aliment Pharmacol Ther 2014;40:1056-1065.

10. Lee M, Lee JH, Oh S, Jang Y, Lee W, Lee HJ, et al. CLIF-SOFA scoring system accurately predicts short-term mortality in acutely decompensated patients with alcoholic cirrhosis: a retrospective analysis. Liver Int 2015;35:46-57.

11. McPhail MJ, Shawcross DL, Abeles RD, Chang A, Patel V, Lee GH, et al. Increased survival for patients with cirrhosis and organ failure in liver intensive care and validation of the chronic liver failure-sequential organ failure scoring system. Clin Gastroenterol Hepatol 2015;13:1353-1360.e8.

12. Choudhury A, Jindal A, Maiwall R, Sharma MK, Sharma BC, Pamecha V, et al. Liver failure determines the outcome in patients of acute-on-chronic liver failure (ACLF): comparison of APASL ACLF research consortium (AARC) and CLIF-SOFA models. Hepatol Int 2017;11:461-471.

13. Heo NY. Acute-on-chronic liver failure: a predictor of poor prognosis in patients with variceal bleeding or a risk factor for variceal bleeding? Clin Mol Hepatol 2020;26:487-488.

14. López-Sánchez GN, Dóminguez-Pérez M, Uribe M, Nuño-Lámbarri N. The fibrogenic process and the unleashing of acute-on-chronic liver failure. Clin Mol Hepatol 2020;26:7-15.

15. Shin J, Yu JH, Jin YJ, Yim HJ, Jung YK, Yang JM, et al. Acute-on-chronic liver failure as a major predictive factor for mortality in patients with variceal bleeding. Clin Mol Hepatol 2020;26:540-553.

16. Korean Association for the Study of the Liver (KASL). KASL clinical practice guidelines for liver cirrhosis: varices, hepatic encephalopathy, and related complications. Clin Mol Hepatol 2020;26:83-127.

17. Jalan R, Fernandez J, Wiest R, Schnabl B, Moreau R, Angeli P, et al. Bacterial infections in cirrhosis: a position statement based on the EASL Special Conference 2013. J Hepatol 2014;60:1310-1324.

18. Fernández J, Acevedo J, Castro M, Garcia O, de Lope CR, Roca D, et al. Prevalence and risk factors of infections by multiresistant bacteria in cirrhosis: a prospective study. Hepatology 2012;55:1551-1561.

19. European Association for the Study of the Liver. EASL clinical practice guidelines for the management of patients with decompensated cirrhosis. J Hepatol 2018;69:406-460.

20. European Association for the Study of the Liver. EASL clinical practice guidelines on the management of ascites, spontaneous bacterial peritonitis, and hepatorenal syndrome in cirrhosis. J Hepatol 2010;53:397-417.

21. Wong F, Piano S, Singh V, Bartoletti M, Maiwall R, Alessandria C, et al. Clinical features and evolution of bacterial infection-related acute-on-chronic liver failure. J Hepatol 2021;74:330-339.

22. Piano S, Singh V, Caraceni P, Maiwall R, Alessandria C, Fernandez J, et al. Epidemiology and effects of bacterial infections in patients with cirrhosis worldwide. Gastroenterology 2019;156:1368-1380.e10.

23. Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008;36:309-332.

24. Rimola A, García-Tsao G, Navasa M, Piddock LJ, Planas R, Bernard B, et al. Diagnosis, treatment and prophylaxis of spontaneous bacterial peritonitis: a consensus document. International Ascites Club. J Hepatol 2000;32:142-153.

25. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837-845.

26. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17-23.

27. Schisterman EF, Perkins NJ, Liu A, Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples. Epidemiology 2005;16:73-81.

28. Cleveland WS, Devlin SJ. Locally weighted regression: an approach to regression analysis by local fitting. J Am Stat Assoc 1988;83:596-610.

29. Augustinho FC, Zocche TL, Borgonovo A, Maggi DC, Rateke ECM, Matiollo C, et al. Applicability of sepsis-3 criteria and quick sequential organ failure assessment in patients with cirrhosis hospitalised for bacterial infections. Liver Int 2019;39:307-315.

30. Piano S, Bartoletti M, Tonon M, Baldassarre M, Chies G, Romano A, et al. Assessment of sepsis-3 criteria and quick SOFA in patients with cirrhosis and bacterial infections. Gut 2018;67:1892-1899.

31. Fernández J, Acevedo J, Wiest R, Gustot T, Amoros A, Deulofeu C, et al. Bacterial and fungal infections in acute-on-chronic liver failure: prevalence, characteristics and impact on prognosis. Gut 2018;67:1870-1880.