A Validation Pilot Study Comparing Telemedicine Images to a Face-to-Face Airway Exam for Conducting the Anesthesia Preoperative Airway Evaluation ()
1. Introduction
Telemedicine is a branch of healthcare that uses communication technology, such as smartphones, video conferencing, and photographs, to provide patients and healthcare providers with medical information and services [1] [2]. Population growth has outpaced the availability of healthcare providers in the United States. Telemedicine has been deemed a useful tool for patient care in remote locations, management of chronic and perioperative conditions, and monitoring of special needs patients and intensive care units [3] [4] [5]. As the COVID-19 pandemic continues globally, the burden on both the inpatient and outpatient healthcare systems has reached unprecedented levels; thus, noninvasive preoperative diagnostics have been successfully adopted to limit this burden and ensure the safety of patients and providers [6].
Preoperative anesthetic assessment is achieved via thorough examination of patients including medical history, physical examination, airway assessment, laboratory workup, and perioperative risk assessment [7]. A review of the medical literature shows evidence that current telemedical technologies can be used to obtain portions of the preoperative assessment [2] [8] [9] [10]. It has been well-described in the past about how telemedicine is a useful tool in obtaining patients’ medical history, provide preoperative instructions, and explaining the anesthetic plan. Telemedicine’s growing role and potential future in anesthesia as a form of expert consultations in emergent situations has been discussed with the growth of technology through newer smartphones, tablets, and 5G networks to ensure seamlessness in communication between patients and providers [11]. The camera quality in smartphones continues to grow rapidly each year; consequently, capturing images and videos in 4K quality is allowing viewers of these media to immerse themselves into experiences that are becoming increasingly lifelike. Even further, virtual reality technology is rapidly developing and has shown various uses throughout medicine and healthcare; this may be another route for telemedicine in the future. Anesthesia telemedicine studies in the past have shown increased patient satisfaction, decreased patient cost in both time and money, and no significant difference in case cancellation rates with telemedical preoperative visits [9] [12] [13] [14] [15]. Studies have even shown increased patient and provider satisfaction, specifically after inclusion of still images into the electronic medical record to supplement the in-person airway exam, in lieu of the traditional written assessment of patients’ airway exams [16]. Despite this burgeoning growth in technology, there is little statistical evidence to date that supports the use of telemedicine as a reliable substitute for an in-person preoperative airway exam using objective parameters.
Currently, patients need to schedule an appointment to meet with an anesthesia provider to have an airway exam performed. This may present hardships for those patients with limited mobility, living in rural areas, undergoing economic hardship, and/or with special needs. As an example of the latter, some individuals may become uncooperative when exposed to the stresses of visiting a large medical center. Patients who cannot receive an airway exam prior to the day of surgery may have increased perioperative morbidity and mortality. Having a validated tool for remote assessment such as a telemedical airway exam will eliminate these barriers to safe and thorough preoperative care. While there have been studies done in the past supporting the use of still images in the preoperative airway examination, conclusions have been limited to suggest incorporating these as a supplement for providers to review. No study has shown a remote assessment alone to be noninferior to an in-person examination.
The goal of this study was to determine whether telemedicine was a reliable and consistent alternative to in-person exams for evaluation of airway parameters by anesthesia providers. Still, images were compared to a live, face-to-face evaluation by three independent, board-certified anesthesiologists. Mallampati score, neck extension, ability to prognath, and thyromental distance, were evaluated for the airway examination. These four parameters were selected because the available literature indicates that, when combined, they are effective in predicting successful intubation [17].
2. Materials and Methods
2.1. Study Design and Subjects
This prospective blinded study was approved by the Institutional Review Board (Pro2012002417) of the New Jersey Medical School, Rutgers University, Newark, New Jersey, USA, and registered at clincaltrials.gov (code NCT0193750). Informed consent was obtained from all participants in the study. The primary aim was to demonstrate that telemedicine can be used with equal efficacy to obtain airway parameters for preoperative airway evaluation.
Volunteer subjects (n = 48) were evaluated for four airway parameters (Mallampati score, neck extension, ability to prognath, and thyromental distance) by telemedicine (still images) and by live examination by three independent board-certified anesthesiologists. Each subject’s telemedical and live office scores were plotted to determine the degree of correlation between telemedicine and face-to-face examination of the study subject’s airway. The still images were taken at a standard distance and height, of mouth opening, neck extension, prognath, and thyromental distance.
2.2. Data Capture and Measurements
A room with four divided spaces was used for each of the three independent evaluators. Each volunteer visited three stations for face-to-face examination. Mallampati score was evaluated in accordance with the I-V classification system. Neck extension was scored between 0 - 4 which corresponded to a range of motion spanning 0 - 60 degrees. Ability or inability to prognath was scored as either a 1 or 0, respectively. Thyromental distance was scored by the approximate number of fingerbreadths needed to bridge the distance. At the fourth station, an iPhone 6 (Apple, Inc., Cupertino, California) was mounted on a tripod. The built-in flash was used on all photos and automatic photo exposure was activated. Additional light was provided by an adjustable table lamp with a 60-watt incandescent bulb. A picture was taken for each of the Mallampati view, neck extension, ability to prognath, and thyromental distance.
Each viewing station had an adjustable chair to ensure proper height and a giant protractor to measure neck extension. Study subjects were instructed on how to place their fingers under their chin to demonstrate their thyromental distance. The pictures were deidentified. The images were sent to the same three independent anesthesiologists one week later for evaluation of Mallampati score, neck extension, ability to prognath and thyromental distance using the above-mentioned scales.
2.3. Statistical Analysis
Data from the telemedical and face-to-face images was entered into an Excel (version 16.29; Microsoft Corporation, Redmond, Washington) spreadsheet for analysis. Comparisons were performed using Wilcoxon signed-rank test to analyze the two study groups. Inter-rater agreement among the three independent anesthesiologists was calculated using free-marginal multirater kappa methodology found using an online Kappa calculator: https://www.justusrandolph.net/kappa/. Free-marginal multi-rater kappa is an extension of the bi-rater free-marginal kappa and a variation of Fleiss’ fixed-marginal kappa [18]. It can take values from 1 to −1. Values from 0 to 1 indicate agreement better than chance, a value of 0 indicates a level of agreement that could have been expected by chance and values from −1 to 0 indicate levels of agreement that are worse than chance. Differences between the average live airway assessment scores and the average telemedical evaluations were considered to be statistically significant if the p-value was ≤0.05. All the material was stored in a safe computer which could be accessed by the study team members only.
3. Results
This prospective pilot study enrolled 48 study subjects to undergo an airway exam via telemedical images and face-to-face evaluation. The average (mean) Mallampati scores for live vs. telemedicine were 1.73 vs. 2.54, respectively. The average neck extension scores were 3.77 vs. 3.60, respectively. The average thyromental distances were 2.95 vs. 2.92 finger breadths, respectively. Finally, the average prognath scores were 0.97 vs. 0.94, respectively. Full results are shown in Table 1 and Figure 1.
![]()
![]()
Table 1. Average independent in-person examiner and telemedical image scores (n = 3).
aext = extension; fb = finger breadths; MP = Mallampati; TMD = thyromental distance.
![]()
Figure 1. Comparison of average scores and standard error for live versus telemedical evaluations. The four parameters studied were (A) mallampati scores; (B) neck extension scores; (C) prognath ability; and (D) thyromental distance. Assessments were performed by three independent anesthesiologists.
Results regarding the objective measurement parameters used in preoperative airway grading were mixed, in terms of differences between face-to-face and telemedical evaluation. Despite the small sample size (n = 48), there were no statistically significant differences (CI 95% and p-value ≤ 0.05) between the study groups for ability to prognath and thyromental distance (Table 2). However, there was a statistically significant difference found between the Mallampati scores and neck extension, based on the Wilcoxon signed-rank tests (Table 2).
To assess inter-rater reliability among the three independent anesthesiologists for the four different outcome measures assessed in this study, free-marginal multirater kappa methodology was applied (Table 3). Degree of agreement based on kappa score was determined to fall into one of six categories: very good (0.81 - 1.00), good (0.61 - 0.80), moderate (0.41 - 0.60), fair (0.21 - 0.40), slight (0.01 - 0.20), or poor (≤0). For Mallampati scores, agreement was fair (κ = 0.30) for in-person examination and slight (κ = 0.15) for telemedical evaluation. For neck extension, agreement was moderate (κ = 0.66) for in-person examination and fair (κ = 0.44) for telemedical evaluation. For ability to prognath, agreement was very good for both in-person (κ = 0.83) and telemedical (κ = 0.83) evaluation. For thyromental distance, agreement was also very good for both in-person (κ = 0.83) and telemedical (κ = 0.85) evaluation. None of these kappa values were statistically different from each other, when comparing live exams to the telemedicine approach (Figure 2 and Table 3).
![]()
Table 2. Comparison of live and telemedical exams for preoperative airway evaluation. Comparisons with statistical significance (p ≤ 0.05), using Wilcoxson signed-rank test, are in bold.
bAvg = Average; ext = extension; fb = finger breadths; MP = Mallampati; TMD = thyromental distance.
![]()
Table 3. Inter-rater reliability between three independent, board-certified anesthesiologists for four airway examination parameters. Agreement was assessed using free-marginal multirater kappa measurements with 95% confidence intervals.
cCI = confidence interval; ext = extension; MP = Mallampati; TMD = thyromental distance.
![]()
Figure 2. Comparison of free-marginal multirater kappa scores with 95% confidence intervals for live and telemedical exams with Mallampati score, neck extension, prognath ability, and thyromental distance. Assessments were performed by three independent anesthesiologists.
4. Discussion
The successful and thorough examination of a patient’s airway is critical to an anesthesiologist. Airway management is one of the most important aspects of a successful anesthetic, and proper preparation depends on the preoperative examination. The ability of an anesthesia provider to evaluate a patient’s airway utilizing telemedical images expands health care delivery to a variety of patients with limited health care access. Obtaining airway exam data prior to the date of surgery increases patient safety, decreases same day surgery cancellation, and allows anesthesia providers to prepare for a difficult airway.
This pilot study was designed to determine if a telemedical airway exam can substitute for a face-to-face evaluation. Our data shows that the exam methods are comparable for ability to prognath and thyromental distance (Figure 1 and Table 2). However, differences in the assessment of study subject’s Mallampati score and neck extension were found to be large when comparing still images to a live exam (Figure 1 and Table 2).
There are several likely explanations for the differences found in Mallampati scores and neck extension. First and foremost, adequate lighting was a significant issue. All three examiners noted that lighting was problematic when scoring the telemedical Mallampati images. They stated that it was difficult to obtain an accurate assessment because the study subjects’ oropharynx was too dark in the photos. This issue will likely be resolved as camera quality rises and technology continues to improve. Other options include adding light clips to cellphones or other devices used to capture the images. For example, there are many commercially available “ring lights” that come with a stand and cellphone holder that can be powered with a remote control and have adjustable lighting settings. Setting these up with a camera in a standardized manner in offices to capture images of the airway may help obtain better views of the oropharynx. A second reason for the difference in Mallampati scores between the two groups is that the photo captures the airway for only an instant, whereas a live assessment allows the examiner to view the oropharynx during mouth opening, tongue movement, and even phonation (although discouraged). This limitation may lead to differing Mallampati scores for telemedical assessments. Previous work suggests that the Mallampati exam should be performed two times to avoid a false positive or false negative result [19]. Our study did not have the patient repeat the Mallampati test, so this may be another reason for the lack of correlation between the face-to-face score and the telemedical score.
Regarding the apparent differences in neck extension grades, extension of the neck is a dynamic motion and obtaining a true understanding of a patient’s motor abilities will understandably be more difficult with a still image. Our study did not include a ruler or protractor for reference with the still images, which may have made it more difficult to gauge the angle of neck extension of the subjects for the anesthesiologists evaluating the images. Future studies may also want to consider the use of videos, either live or prerecorded, in examining the airway for a more dynamic remote examination. Prerecorded videos may be advantageous for patients to be able to perform at their own convenience. However, while live videos may require an anesthesiologist to be present the whole time, they also ensure that the video that is captured is adequate for comprehensive preoperative evaluation. This may possibly be a more robust method of mimicking a live assessment, provided there is a convenient, HIPAA-compliant method of documenting and sharing video files.
Of note, the percent agreement and free-marginal kappa statistic for both the Mallampati score and neck extension score were considerably lower than those for the thyromental distance and ability to prognath, both for live exams and telemedicine evaluations (Table 3). The small sample size of this study (n = 48) and the lower inter-rater agreement rates for Mallampati score and neck extension highlight that it may be difficult to obtain consistent ratings for these parameters in daily practice. These kappa values and agreements rates were not different from each other when comparing live exams with telemedicine, signifying that raters had similar reliability with both approaches in all four measurements (Figure 2). However, this level of reliability was considerably lower with Mallampati and neck extension scores. Thus, the statistically significant differences found from the Wilcoxon signed-rank tests should be viewed in the greater context of decreased inter-rater reliability, as compared to the other outcome measures in this study. Larger scale studies are needed to further evaluate if there are true differences between telemedicine exams and in-person exams for these outcome measures.
The feasibility of “remote evaluations” and actual implementation is a topic that must be addressed. One option for capturing images for a remote airway evaluation would be setting up a camera with adequate lighting in outpatient surgery offices where a patient must travel to regardless to discuss treatment with his or her surgeon. This would ensure that there is no burden on patients to have the adequate equipment or technical skills to take a good photograph of their airway by themselves, and instead allow medical technologists to oversee the process in a controlled setting, like what was done in this study. Another option would be giving patients detailed instructions on how to set up their phone or camera in an area with adequate lighting in their own homes for a true remote evaluation. Advances in camera technology may make this more feasible soon, but certain patients may be limited in their functional capacity to be able to get high quality images. In communities of a generally lower socioeconomic status, access to the proper instrumentation, environment, and the literacy required to complete the tasks needed to properly record adequate images or videos of the airway may be limited.
As this is a pilot study where n = 48, confidence intervals of 95% (p-values ≤ 0.05) applied to the data do not indicate validity of the study. Rather, trends in the data are more representative and provide credibility for the results. Confidence intervals such as 75% or 85% are often employed to provide more accurate estimation of hypothesis testing in pilot studies [20]. This is a limitation of the study, and future studies with larger sample sizes must be done for a stronger diagnostic test validation analysis.
This study’s data shows that a telemedical airway exam is a reliable tool that enables anesthesia providers to obtain essential patient information prior to the day of surgery and without having a patient visit the preadmissions testing clinic. Moreover, in the setting of the global COVID-19 pandemic, tele-anesthesia affords providers a safe and accurate method of performing airway exams while minimizing contact with patients who are positive or suspected to be positive for the virus and require urgent or semi-urgent surgery. Further studies on this topic with larger sample sizes and more advanced smartphones are warranted to validate telemedicine for airway exams and other applications in the practice of anesthesia.
Acknowledgements
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The authors declare no competing interests.