Portuguese Society of Intensive Care Score for Predicting SARS-CoV-2 Infection Applied to Inpatients with Pneumonia: A Reliable Tool? ()
1. Introduction
Coronavirus induced-disease 2019 (COVID-19) was first reported on the 31st of December of 2019 [1]. Since then, it became a pandemic and it brought a heavy burden for the health system of several countries [2]. Symptoms of COVID-19 are non-specific, and its presentation can range from no symptoms to severe acute respiratory syndrome and death [3]. Widespread testing is essential. However, a single negative test does not exclude COVID-19, especially in highly suspected patients [2] [4]. For a negative test, there are two key factors: pretest probability and test sensitivity. The sensitivity rate of the rRT-PCR is estimated to be 66% - 80% [2] [4]. Pretest probability depends on several factors, including local COVID-19 prevalence, exposure history and symptoms [4]. Prediction models that combine several of these features to estimate the risk of people being infected could assist medical staff [2]. Efficient diagnosis tools are necessary to help triage patients when allocating limited healthcare resources [2].
In March 2020, the Portuguese Society of Intensive Care (PSIC) proposed a risk score based on 10 variables to stratify patients with pneumonia regarding Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2) infection probability [5]. It confers a relative score to 10 variables, including demographic, clinical, analytical and imaging data (Table 1). It is considered indicative of a
![]()
Table 1. Portuguese Society of Intensive Care (PSIC) risk score for COVID-19.
Abbreviation: CT, Computed tomography; SARS-CoV-2, Severe Acute Respiratory Syndrome-Coronavírus-2; COPD, Chronic Obstructive Pulmonary Disease; LDH, lactate dehydrogenase.
probable COVID-19 case when a patient with lower airway infection has a total punctuation ≥10 [5]. If a patient’s first rRT-PCR test is negative but the score is ≥10, the PSIC recommends considering the patient COVID-19 suspect until a second test (preferably in lower airway samples) is performed 24 to 48 hours later [5].
In Portugal, this tool was frequently used at admission increasing SARS-CoV-2 infection clinical suspicion. Since few data is available on validated scores predictive of SARS-CoV-2 infection, the aim of this study was to validate the PSIC clinical risk score, applied to inpatients with pneumonia, in predicting a higher probability of COVID-19, based on a single-center department cohort, while assessing the significance of the independent PSIC score variables on the probability of a positive SARS-CoV-2 rRT-PCR test.
2. Methods
This retrospective study was conducted in a COVID-19 dedicated Department, at Centro Hospitalar de Setúbal, E.P.E. (CHS), in a community hospital in Setúbal, Portugal, with the approval of the hospital institutional review board. As only retrospective, non-identifying patient data were collected, informed consent was not applied. Access to clinical file was performed through hospital intranet software namely SClinic.
All patients with pneumonia admitted to the ward between 20/03 and 20/05/2020 were included. Patients without all the required variables to the score calculus were excluded.
Each case of the data set is characterized by 12 independent/predictor variables (including age, gender, comorbidities, symptoms, laboratory and imaging findings as well as exposure as described in the PSIC score), 10 of which are categorical variables with two classes (Yes = 1/No = 0). The dependent variable is a binary variable coded as 0 (Test Negative) and 1 (Test Positive). The independent categorical variables are the 10 score variables proposed by the PSIC codified in binary values considering the presence or absence of male gender, comorbidities (diabetes, COPD, cerebrovascular, oncologic and/or cardiovascular disease such as hypertension or ischemic cardiomyopathy), exposure to a positive SARS-CoV-2 patient, fever and cough at presentation, lymphopenia (<1200/µL), reactive C-protein (CRP) > 5 mg/dL, procalcitonin (PCT) < 0.5 ng/mL, negative respiratory viruses other than SARS-CoV-2 test as well as Legionella pneumophila and Streptococcus pneumoniae urinary antigen tests, lactate dehydrogenase (LDH) > 250 U/L and typical chest CT features. Typical CT features were defined as uni-/bilateral ground glass opacities/consolidation or crazy-paving patterns [6]. Clinical characteristics outside the scope of the PSIC score were not considered. The total PSIC score was calculated for each patient.
All patients were tested for SARS-CoV-2 infection by rRT-PCR assay using nasal and oropharyngeal swab specimens. The number of samples collected during admission was variable, based on individual clinical characteristics. Patients were then classified as COVID-19 positive or negative.
Data were collected based on patients’ clinical file and were analyzed in May 2020. A team of experienced respiratory clinicians reviewed, abstracted and cross-checked the data.
Statistical Methods
The logistic regression modeling approach was selected considering the binary nature of the dependent variable. The selection of significant independent variables, with predictive power, was done by using the Forward Stepwise (Likelihood Ratio) method. The coefficients adjustment of the retained independent variables was done by maximizing the log likelihood function using a nonlinear optimization algorithm. The use of the logistic regression model as a classifier tool (Positive group versus Negative group result) requires a cut value for the computed probability. The best cut value was obtained with the help of the Receiver Operating Characteristic (ROC) curve together with the evaluation of the discriminatory power of the model through the Area Under the Curve (AUC). Since the total PSIC score was calculated for each patient and this was also included on our database, in our study two candidate logistic regression models were developed. The probability of a positive SARS-CoV-2 rRT-PCR test was computed. Model I considered the total PSIC score of each patient and Model II considered all the PSIC score variables regardless total score.
Descriptive statistics were performed for the basic analysis of data. Hypothesis tests of significance of the differences between proportions and means of two groups were also performed, assuming, respectively, Bernoulli and Normal populations with unknown equal variances. All the analysis was carried out using the software IBM SPSS Statistics 25.
3. Results
A total of 207 patients with pneumonia were admitted to the COVID-19 Department. The validation cohort included 145 patients, and 62 (30.0%) were excluded due to lack of information on PSIC score variables such as: chest CT (n = 20), procalcitonin level (n = 40), urinary antigen tests (n = 15), LDH level (n = 3) and information regarding exposure to a positive SARS-CoV-2 patient (n = 1).
The mean age was 69.90 ± 17.83 years and 81 (55.9%) were males. Thirty-seven (25.5%) patients were active or ex-smokers and most (87.6%) presented comorbidities, including cardiovascular disease (n = 108, 74.5%), diabetes (n = 45, 31.0%), COPD (n = 34, 23.4%), oncologic (n = 24, 16.6%) and/or cerebrovascular disease (n = 21, 14.5%).
Ninety-eight (67.6%) patients presented a CRP > 5 mg/dL, 93 (64.1%) PCT < 0.5 ng/mL, 77 (53.1%) lymphopenia (<1200/uL), and 74 (51.0%) LDH level > 250 U/L. Sixty-one (42.1%) patients presented a PSIC score ≥ 10.
Twenty-three (15.9%) patients tested positive in the SARS-CoV-2 rRT-PCR assay, 20 of whom (86.9%) obtained a PSIC score ≥ 10. Of note, 3 of the 62 excluded patients tested positive in the SARS-CoV-2 rRT-PCR assay. Two of these patients did not perform chest CT and one did not include information in the clinical file regarding exposure status to a positive SARS-CoV-2 patient. The total PSIC score of these patients were 8, 10 and 13, respectively.
Demographics and clinical characteristics of patients in the validation cohort are provided in Table 2.
PSIC score variables and mean total PSIC scores were compared between the positive and negative COVID-19 groups (Table 3). Typical chest CT features, exposure to a positive COVID-19 patient and PCT < 0.5 ng/mL were significantly more prevalent in the positive SARS-CoV-2 rRT-PCR group (p < 0.0001, <0.0001 and 0.044, respectively). The mean PSIC score was also significantly higher in the positive group (13.17 ± 3.72 vs 8.39 ± 3.67, p < 0.0001).
For Model I, the probability to be positive as a function of the PSIC score is given by the relation
(1)
and, for a cut value of 0.20, the calculated sensitivity and specificity achieved a value of 78.3% and 73.8%, respectively, with an overall predictive accuracy of 74.5%.
The most significant independent predictive variables of Model II were typical chest CT features (OR 12.16, 95% CI, 3.32 - 44.49) and exposure to a positive COVID-19 patient (OR 6.56, 95% CI, 1.33 - 32.30). For Model II, the probability to be positive as a function of two most significant independent variables is given
![]()
Table 2. Demographics and clinical characteristics of the validation cohort.
Abbreviation: SARS-CoV-2, Severe Acute Respiratory Syndrome-Coronavírus-2; LDH, lactate dehydrogenase; CT, Computed tomography. *Comorbidities included all those considered in the PSIC score such as: diabetes, COPD, cerebrovascular, oncologic and/or cardiovascular disease such as hypertension or ischemic cardiomyopathy.
![]()
Table 3. PSIC score variables frequency and total PSIC score according to SARS-Co-V-2 rRT-PCR result and groups’ comparison.
Abbreviation: CT, Computed tomography; LDH, lactate dehydrogenase; SARS-CoV-2, Severe Acute Respiratory Syndrome-Coronavírus-2. *Comorbidities included all those considered in the PSIC score such as: Diabetes, COPD, cerebrovascular, oncologic and/or cardiovascular disease such as hypertension or ischemic cardiomyopathy; +Including: Legionella pneumophila and Streptococcus pneumoniae urinary antigen tests.
by the relation
(2)
and for a cut value between 0.20 and 0.24, sensitivity and specificity were, 87.0% and 69.7%, respectively, with an overall classification performance of 72.4%. For all models, the AUC was greater than 0.8, meaning that their discriminating power can be considered as good.
Further analysis revealed that the best cut-off value for the PSIC score is 10 or 11. The selection of each one depends on what is more relevant—sensitivity (10) or specificity (11) (Figure 1 and Figure 2).
4. Discussion
Less expensive and complex COVID-19 pneumonia diagnostic methods are urgently needed to facilitate timely intervention. Validated clinical prediction models to estimate the risk of SARS-CoV-2 infection are of value in managing limited hospital resources and aid in infection control. However, according to our research, various scores to estimate the probability of SARS-CoV-2 infection in inpatients with pneumonia have been proposed, but a small number have
![]()
Figure 1. Graphics (a) and (b) are informative on how the best values simultaneously for Sensitivity and Specificity, around 80%, were found, as function of the cut value of the probability to be positive. The selection of the best cut values of the probability providing the highest classification performance was based on the ROC curve (Graphic (c)), representing Sensitivity versus (1-Specificity).
![]()
Figure 2. Probability to be Positive versus total PSIC score. Two horizontal lines, representing two possible best cut values (0.10 and 0.15) for the probability are shown. The corresponding value for the total PSIC score is between 10 and 11.
been validated.
L. Wynants et al. [2] published a metanalysis with critical approach to models published to support the diagnosis of COVID-19 in patients with suspected infection. One group of authors [7] developed the model “Suspected COVID-19 pneumonia Diagnosis Aid System”, which aims at predicting SARS-CoV-2 infection in patients admitted with fever. The authors developed diagnosis aid models based on machine-learning algorithms and clinical data derived from demographic, clinical signs and symptoms and laboratory tests. However, chest CT features were not included. On the other hand, another group [8] developed a diagnostic model to quickly identify COVID-19 pneumonia which mainly relies on chest CT findings, as they defend that according to clinical symptoms and signs and laboratory examinations, COVID-19 pneumonia is difficult to distinguish from other viral pneumonia.
The PSIC proposed a risk model that confers a relative score (grade) to ten variables, including demographic, clinical, analytical and imaging data. The score’s main objective is to determine which patients with lower airway infection present a higher probability of SARS-CoV-2 infection. However, this score has not yet been validated. This study aimed at validating this clinical risk score for SARS-CoV-2 infection, applicable to inpatients with pneumonia. Most patients admitted for pneumonia were males, with a mean age of 69.90 ± 17.83 years old, and most (87.6%) presented comorbidities which is concordant with the epidemiological COVID-19 studies’ findings, suggesting a higher incidence of hospital admission in this subgroup of patients [9] [10] [11]. However, other respiratory pathogens can also present a higher incidence of severe disease in the elderly subpopulations with comorbidities [12] [13]. Indeed, when comparing positive and negative SARS-CoV-2 rRT-PCR subgroups, male gender and comorbidities were equally found in both conditions.
According to recent publications, the most common comorbidities in COVID-19 patients are hypertension, obesity and diabetes [9] [14] [15], which is also in agreement with our findings. Of note, obesity was not analyzed in this study since it was not included in the PSIC score.
When comparing positive and negative SARS-CoV-2 rRT-PCR subgroups, typical chest CT features, exposure to a positive SARS-CoV-2 patient and PCT < 0.5 ng/mL were significantly more prevalent in the positive SARS-CoV-2 rRT-PCR subgroup.
The logistic regression analysis revealed that typical chest CT features and contact with a positive case were the most significant independent variables in predicting a positive rRT-PCR-SARS-CoV-2 test, which partially agrees with the relative scoring system proposed by the PSIC. This is also concordant with other studies’ findings, which have demonstrated that typical chest CT features present higher sensitivity in the early detection of COVID-19 pneumonia [6]. Interestingly, 37 (30.3%) patients in the negative subgroup presented typical CT features.
Twenty-one percent of patients presented with fever and dry cough and when comparing subgroups these symptoms were more prevalent in the positive group (34.8% vs 18.0%, p = 0.069), with a p-value close to the threshold of statistical significance suggesting these might play a role in the prediction a positive test in wider sample size studies.
Lymphopenia has been associated with severe coronavirus disease [16] [17]. Nevertheless, its role in predicting infection may be somewhat limited since in this study, the presence of lymphopenia (<1200/uL) didn’t determine a significantly increased risk of a positive SARS-CoV-2 test.
The aged study population and the development of lymphopenia in other viral infections and medical conditions may have accounted for these findings since ≥50% of patients in both groups presented this condition. This suggests that the relative scoring system proposed by the PSIC might need adjustment in the weight of this variable for this subpopulation.
Only a small percentage of patients tested positive for other respiratory viruses and presented positive urinary antigen tests and the prevalence of negative results was similar between the positive and negative rRT-PCR-SARS-CoV-2 subgroups, which seems to preclude these variables from being good infection discriminators. However, these can be useful tools for diagnosing co-infection, which may imply different treatment strategies. Importantly, in our study, co-infection was found in 39.0% of patients, including other respiratory viruses and bacteria. This finding may have also contributed to the non-significant results in CRP, PCT and LDH levels.
According to our results, the best score cut-off value was between 10 and 11, which is in accordance with the one proposed by the original authors. In this study, most (86.9%) of the positive SARS-CoV-2 patients presented a score ≥ 10 and the mean PSIC score was significantly higher in the confirmed COVID-19 subgroup.
The ROC curve analysis illustrated that the PSIC score can accurately stratify patients with pneumonia in different COVID-19 risk categories.
Although this model has been shown to be a useful risk assessment tool for COVID-19 pneumonia, it may miss-label some patients. Indeed, forty-one (33.6%) patients of the negative COVID-19 subgroup presented a PSIC score ≥ 10, but the meaning of these findings may be somewhat misleading considering the sensitivity of the rRT-PCR test.
Furthermore, the PSIC score was proposed with no description of the methods used for the determination of score variables, grade scoring system and score cut-off, probably due to the need of fast interventions in the setting of a public health emergency.
This study presented several limitations. First, it was a retrospective analysis. Secondly, it included a limited number of patients reflecting the reality of a single-center department. Not all patients admitted to the ward were included in the validation cohort. Most patients were excluded due to lack of chest CT images since not all were considered to present criteria for the performance of a CT-scan when the clinical picture and the chest radiography were sufficiently informative and the CT radiation risks and costs were considered to outcome its benefits. Other patients were excluded due to a lack of information on other variables such as PCT and urinary antigen tests. However, it should be noted that the study was developed during the initial phase of COVID-19 in Portugal when diagnostic protocols were still being implemented and the exams were requested according to clinical presentation. Results in this study rely on the SARS-CoV-2 rRT-PCR test for the confirmation of positive COVID-19 cases. However, this test presents a sensitivity of approximately 66% - 80% [2] [4], which means that a considerable number of positive cases may have not been identified. Moreover, collected samples for the SARS-CoV-2 test were from Naso and oropharyngeal swabs, which present a lower diagnostic yield compared to lower airway samples [10]. The authors hypothesized whether some of these patients could have been good candidates for the serologic monitoring during the acute phase and follow-up, particularly the ones with typical CT features. Regardless of the limitations, this is an attempt to validate a useful tool in the clinical practice with encouraging results, particularly when allocating patients at admission.
In this study, the PSIC score was a reliable and valid tool for assessing the probability of a positive SARS-CoV-2 rt-PCR test in inpatients with pneumonia. Findings suggest that slight adjustments of the PSIC score might be considered in this subpopulation. When dealing with a highly contagious respiratory virus, the use of diagnostic clinical risk scores coupled with laboratory tests at admission may facilitate hospital patient allocation while under observation promoting intra-hospital infection control. To validate the proposed risk score at a broader level and to improve it, it should be applied at a multicentric scale. So far, proposed models are poorly reported and at high risk of bias. Hence, they cannot be recommended in current practice.
Contributions
VC conceptualized the study. AA collected and organized the data. FD, VD, AA and DN performed the statistical analysis. AA, DN, VD and VC wrote the manuscript. All authors reviewed the manuscript.
Acknowledgements
We would like to thank all the medical team that provided care to the patients admitted to the COVID dedicated department of Centro Hospitalar de Setúbal E.P.E.. We also would like to thank Ivone Fernandes M.D. whose hard work contributed to the continuous care of other respiratory patients while the team was divided.