Evaluation of the Usability of a Telehealth System for COVID-19 According to the Perception of the User Professional ()
1. Introduction
The year 2020 was marked by the discovery of an infection originating from a modification of the virus called SARS-Cov-2, which had an impact of sufficient size to the point of being considered a Pandemic. Near ubiquity and cross-species transmission are specific features of coronaviruses. Still, the rapid dispersion, the varied incubation period, not yet fully identified forms of transmission, asymptomatic carriers, absence of herd immunity and high morbidity and mortality related to SARS-CoV-2 are some of the factors that concern physicians and scientists worldwide [1].
It is known whether the main route of transmission of the virus is by inhalation of contaminated droplets (aerosols) or by contact with contaminated hands and nasal, oral and ocular mucosa [2]. Faced with the growing pandemic chaos, defense and prevention measures involve, in addition to early diagnosis, isolation of the infected patient, monitoring of their contacts, as well as suspected and confirmed cases, social isolation and quarantine [3].
The confinement, combined with the impact of the crisis on health systems and the risks inherent to exposure to the virus, demanded behavioral adaptations never seen in history, demanding almost instantaneous changes, and still ongoing, in this new pandemic scenario.
In this context, telehealth and telemedicine gain a prominent role in meeting many demands in the health area, contributing to the reduction of the risk of transmission of the coronavirus in different scenarios in the world [4] [5] [6] [7] [8]. Given the scenario, the Ministry of Health of Brazil listed some measures to respond to the fight against the pandemic in the country. In order to support the population in the adoption of preventive practices, guarantee assistance for mild cases and coordinate care for the most severe cases, it was essential to use remote assistance strategies, placing Telehealth and Telemedicine as central points in this strategy.
In partnership with a private company that won the bidding process for the execution of the proposed services, the telehealth service of the Unified Health System called TeleSUS was structured in record time, an ecosystem based on the intensive use of information technology involving automated mechanisms and a central personalized service at a distance. Through Telehealth, there was teleservice by qualified and trained nursing professionals for remote care and clinical triage of the patient, in the presence of symptoms of COVID-19, all guided by clinical algorithms referenced and based on scientific literature.
According to each identified risk situation, different guidelines were offered to the patient as well as promotion and prevention guidelines, home isolation for cases and household contacts, with provision of self-declaration, guidance on the use of medication, referral to the most appropriate health service, health, among others, in addition to returning calls for all severe symptomatic cases.
During the period of services operation (April to July 2020) more than 970 thousand individuals throughout the national territory were assisted by TeleSUS and more than 85% of those assisted did not need to seek a face-to-face health service to solve their problem of health, staying at home under the program’s self-care guidelines.
In addition to constant evaluations carried out in the service as a public health strategy, at clinical and epidemiological levels, the team involved in the project, of which the present author was part, was also concerned with evaluating the system developed to enable the operation of remote care, from the conception of the organization of health actions to the technological development of the digital health tool. To this end, among other strategies, the researcher used the context as a pilot test of the guidelines developed in her Doctoral research for the registration of clinical trials containing digital health tools, yet to be published.
Previous research indicates that the telehealth strategy makes it possible to provide specialized assistance to the population and is highlighted in translational research as a remote access technology for health databases [9] [10] [11]. In the scenario of the COVID-19 pandemic in Brazil, it is possible to identify the emergence of several telemedicine and telehealth initiatives developed by the federal and state government spheres, with a focus on medical assistance and also on the training of health professionals. In addition, self-assessment tools were developed to act in an integrated way in the diagnosis of patients infected with the COVID-19 virus, aiming to reduce the exposure of individuals to forms of contagion and, consequently, reducing the spread of the new coronavírus [12] [13] [14] [15].
Countries such as China have also made significant advances in the field of telemedicine, focusing on remote patient diagnosis for COVID-19 treatment and playing a key role in the response to the pandemic [16]. The United Kingdom, on the other hand, has developed strategies such as an app to connect people remotely, through artificial intelligence, reducing social and mental damage caused by the period of isolation [17].
The United States, in turn, invested in health care focused on outpatient care in a home stay regime, hospital outbreaks of the new coronavirus and treatment of confirmed cases of the disease. Latin American countries such as Argentina, Uruguay, Peru and Ecuador have developed applications aimed at communication for decision-making in the health of users, self-care newsletters, specialized service guides, synchronous free medical advice, among others [18].
It is known that health information systems are not limited to the technology they use, but to the care environment in which they are inserted, the profile of system users and the type of data that must be stored [19].
In order to design systems considered to be of good usability, it is necessary to understand factors of different natures such as psychological, organizational, ergonomic and social that will determine how people use the systems effectively [20]. From there, the development of tools must translate this interaction, reaching the pillars of efficiency, effectiveness and safety when using them.
To evaluate the system, the instrument chosen was the System Usability Scale (SUS) [20] with ten questions that measure the usability of products and services, applicable to mobile and clinical systems, hardware, websites, among others. The instrument is quick to apply and generates a single score on an intuitive and easy-to-understand scale, in addition to its good reliability and references that support its interpretation.
In this sense, considering the importance of the system for public health services in Brazil and for helping managers and professionals in health decision-making, the objective of this study was to carry out an evaluation of a telehealth system, measuring the degree of satisfaction of user health professionals regarding its usability and identifying factors that positively or negatively influence the evaluation.
2. Methods
It is a cross-sectional analytical descriptive research with a quantitative and qualitative approach. The main objective of cross-sectional studies is to obtain reliable data that allow the elaboration of reliable, robust conclusions, in addition to generating new hypotheses that can be investigated later. Both in analytical studies, which seek the association between two or more phenomena, and in descriptive studies, whose scope is the detailed and organized description of one or more phenomena, the quality of the necessary data and the standardization of the information collection methodology must be guaranteed.
The project was approved by the Research Ethics Committee (CEP) of the Faculty of Medicine of Ribeirão Preto, University of São Paulo—FMRP/USP (CAAE: 14671019.2.0000.5440). Data were analyzed in MS Excel 2019 16.0.11929.20198 and statistical tests of correlation of variables were applied, when necessary.
2.1. Selection of the Study Population
Ten professional health and information technology specialists involved in the development of TeleSUS, with more than five years of academic training and a minimum Master’s degree, were selected for the study. Participants were invited to fill out the electronic instrument (System Usability Scale) and registered in the system with a login and password to fill in the data, consult and/or correct it, if necessary. Data completion was carried out between January and April 2022.
2.2. System Usability Scale Instrument
The System Usability Scale instrument has the premise of numerically verifying the usability level of a system. The numerical usability scale, created by John Brooke in 1986, can be used to evaluate software, hardware, websites, applications, services, among others, and its popularity is because it has scientific validity and is an instrument of easy and quick application.
The criteria that the instrument evaluates are the effectiveness of the system, that is, how much users are able to complete their objectives when using it, the efficiency related to how much effort and resources are needed to fulfill such objectives and the user’s satisfaction in the usage experience. The application of the instrument took place at the end of the usability test, after users performed a certain group of tasks using the website/application [20].
The questionnaire was divided into two stages, the first containing demographic data such as gender, age, education and professional area, and the second containing the ten questions of the System Usability Scale instrument, graded using a 5-point Likert scale: 1—“strongly disagree”; 2—“disagree”; 3—“neutral/neither agree nor disagree”; 4—“agree” and 5—“strongly agree”.
To find the answer value, a single number, it is necessary to use the test calculation methodology: for odd questions (numbers 1, 3, 5, 7 and 9) it is necessary to subtract 1 from the score that the user answered. For even questions (number 2, 4, 6, 8 and 10) you must subtract 5 from the score that the user answered. Then, add the values of the ten questions and multiply by 2.5. Questionnaire results can range from 0 to 100. The system is then rated according to the score received as <20.5 (worst imaginable); from 21 to 38.5 (poor); 39 to 52.5 (median); 53 to 73.5 (good); 74 to 85.5 (excellent) and 86 to 100 (best imaginable/possible).
3. Results
All users selected for the study answered the questionnaire (n = 10). The profile of the participants was mostly female (70%), 40% with a Master’s degree and 60% with a Doctor’s degree and an average age of 46.2 years, being a user (10%) in the age group of 30 to 39 years, seven (70%) in the age group from 40 to 49 years and two (20%) in the age group from 50 to 59 years.
Regarding the time of professional activity, it was observed that 40% of the specialists had between five and ten years of professional practice, another 40% between eleven and twenty years of experience and 20% between twenty-one and 30 years of career. Regarding the area of activity, most were distributed between Medical Assistance (30%) and Scientific Research (30%) and the rest in Health Management (20%) and Higher Education Teaching (20%).
In the final score obtained by the System Usability Scale instrument, the mean total score was 84.6, with a standard deviation of 14.2, a minimum value of 42 and a maximum of 100, which classifies it as excellent in usability by the metric established in the construct.
The association analysis of variables was also performed using the Spearman coefficient and the Kruskal-Wallis test. The Spearman Coefficient method aims to measure the strength of the association between classified variables and can have a value between +1 and −1, where the value of +1 in ρ indicates a perfect association and the value of −1 means a negative association between the intervals. More clearly, it means that the greater the absolute value of the coefficient, the stronger the relationship between the variables studied.
The Kruskal-Wallis Test is used in situations where there is a need to compare more than two independent groups of the same size or not, indicating whether there is a difference between at least two of them. For the application of the test, the numerical values transformed into rank and grouped as a data set are used. The groups are then compared using the average of the posts created. In summary, the test is indicated to test the hypothesis that three or more populations have equal distribution or not.
In the analysis of the association by the Spearman coefficient and the Kruskal-Wallis test of the instrument’s score with the variables education, age group and professional area, no statistically significant correlations were found (p-value = 0.201; p = 0.613; p = 0.511, respectively).
Considering the relative values obtained as answers to the instrument’s questions it was possible to verify that the system developed for TeleSUS presents the usability attributes recommended by Nielsen, which are the ease of learning, efficiency in use, ease of memorization, low error rate and subjective user satisfaction [19] [20] [21].
In addition to the usability assessment carried out through the SUS instrument, the researcher verified, along with the telehealth service system development process, items referring to the objectives, needs and expectations of the professionals who use the system through its developed checklist (SACI protocol, to be published). The delay in correcting errors in filling in patient data in the system was identified as a point of attention once the call was opened to the development team. A highlight raised by the participants was the user support offered by the system on a 24-hour help platform, with direct communication to support in a quick and resolute way and with an intuitive interface for filling.
4. Conclusions
Throughout the technology development process, concepts and theories relevant to telehealth [22] [23] were analyzed, comparing them at different stages of the life cycle of a solution for the purpose of remote health care. Since its conception, the stages of technological development, implementation, integration and operation were permeated based on real world needs and the intended outcomes for patients using the system, causing a positive impact in the face of the COVID-19 pandemic scenario experienced in the entire world.
The system was evaluated with excellent usability by the user professionals for observing fundamental issues related to the Telehealth Model built for the provision of services and included seven essential domains in its composition: the identification of the population’s health needs, the definition of the technologies used with the choosing the solution that best meets the presented scenario, the legal and ethical responsibility with health data, the acceptability of the patient and the professional user of the technology, the constant monitoring and auditing of the system, indicators and political aspects [23] [24].
It is expected, in the future, that new assessments will be carried out with a larger number of populations and that services and systems like this will be increasingly present in the reality of countries like Brazil, where the geographic distance limits the offer of services in basic and specialized health in a large part of the national territory.
Acknowledgements
The lead author would like to thank all the experts who contributed to the evaluation of the system in this project, the support of the Universities involved for the publication of the article and all the expert co-authors equally involved in the construction of this research and data analysis.