SACI Protocol: Preliminary Guidelines for the Development of Digital Health Technologies for Clinical Trials in Brazil ()
1. Introduction
Mankind, throughout history, records its knowledge and evolution as a species using various techniques from cave painting to the most standardized technical documents currently available. The scientist man is no different. Unfortunately, research carried out without standardization in its reports causes difficulties in the correct interpretation of the results and in their replication by other researchers in other scenarios.
Often essential elements of study methodology are omitted or poorly described, which limits the value of established guidelines for clinical practice. Another main quest of mankind has been the creation of machines equipped with artificial intelligence that approach and even surpass the capacity of the human brain. This very ambitious search has captivated, mainly, professionals linked to the health area, who envisioned in the computing the aid in carrying out diagnoses and making decisions regarding treatments for their patients, functions far beyond the first uses of the computer for storage and data processing and knowledge.
In the context of the pandemic in which we have been living since 2020, digital technologies are increasingly being used to support public health decision-making and response around the world, including epidemiological surveillance, case diagnosis and contact tracing, in addition to assisting in the evaluation of remote interventions based on health data collected from the population. The trend is for health to become increasingly digital and for the use of e-health tools to assist in the process of formulating regulation, evaluation, and use of digital technologies to strengthen public and supplementary management in the face of crisis situations in health [1] [2].
Although several studies have been carried out to test digital technologies as part of clinical intervention in health treatments [3] - [18] there is no record of a specific proposal for guidelines for research and implementation of scientific investigations—such as clinical trials—with e-health tools in Brazil.
The scientific community has, to date, the CONSORT E-HEALTH [19] , the CONSORT-AI EXTENSION [20] and the SPIRIT-AI [21] as guide protocols for interventions that use artificial intelligence in health, established in contexts different from those experienced in our country in relation to the development of technologies in digital health.
Given the context, the objective of this work is to propose preliminary guidelines for the development and implementation of digital health tools for clinical trials in Brazil. Despite its apparent simplicity, what differentiates the protocol proposed in this article from the guidelines already published internationally is its development designed for the health technology scenario in Brazil and its implementation carried out in record time in the service called TeleSUS, a telehealth service of the Brazilian Unified Health System (SUS) developed in response to the COVID-19 pandemic in Brazil, in March 2020 [22].
2. Summary
This article discusses the process of creating a protocol to standardize the registry for the development of digital health tools in Brazil.
The first section brings a brief introduction, with important findings from the scientific literature on the subject; the methods section, in turn, presents the step-by-step development of the research, going through the stages of ethical approval, theoretical conception, and validation of the instrument by specialists in the area.
The results section presents the protocol ready with the complete checklist and finally, the concluding remarks section highlights the main discussion points of the research and the value of the standardized record of the development of technologies for the health area.
3. Methods
The methodological study describes the development of a rigorous and systematic method of obtaining and organizing data and conducting the research. The approach chosen was qualitative and quantitative.
The construction of the preliminary guidelines, or here also called “protocol”, was based on existing instruments in the literature that systematize guidelines and recommendations for conducting clinical trials, although only one of these instruments [20] specify important methodological topics for software research in healthcare. The main instruments used were CONSORT Statement [23] and its extensions STRICTA [24] , E-HEALTH [19] and CONSORT-AI EXTENSION [20] , SPIRIT Statement [25] and its extension SPIRIT-AI [21] , TIDIER [26] , QUOROM [27] and STARD BLCM [28].
Clinical trial registration platforms certified by the World Health Organization (WHO) and the International Committee of Medical Journal Editors (https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html) were also consulted, highlighting the ReBec—Brazilian Registry of Clinical Trials (http://www.ensaiosclinicos.gov.br/) and the ISRCTN Registry (http://www.isrctn.com/), in order to verify which registry aspects they contemplated. After collecting and analyzing the synthesized knowledge, the development of the standardized protocol was divided into five steps, whose implementation and evaluation steps will be explored in another publication.
The first step was to carry out a systematic review in the PUBMED/MEDLINE, WEB OF SCIENCE, SCOPUS, LILACS SciELO databases, using the descriptors previously selected to create the protocol. The methodology used to carry out the systematic review is described in detail in an article already published [29].
The second step was the refinement of the topics/issues to be addressed in the protocol statement. After the delimitation of these topics, the third step was the validation of the protocol, ensuring the review/evaluation of the statement established by peers of experts. The fourth step was implementation, including the pilot test site.
Finally, the fifth and final step, called Assessment, consisted of analyzing the results of using the protocol and its quality as a teaching instrument.
3.1. Ethical Approval
This study was approved by the Research Ethics Committee of the Faculty of Medicine of Ribeirão Preto, University of São Paulo (CAAE: 14671019.2.0000.5440). All the ethical requirements prescribed by the resolution n˚ 510/2016 of the Brazilian National Health Council and its complementary were fulfilled in the development of this work. All the necessary information about the study was provided to participants electronically prior to survey completion. Delphi participants provided electronic informed consent.
3.2. Validation of Preliminary Guidelines
Among the available methods for validating protocol content, the Delphi Method [30] was chosen which aims to obtain as much agreement as possible among a group of experts on a topic, when there is no unanimity of opinion or contradictory information recorded in scientific literature. In this research, the method was adapted to the online modality, via the internet, and with a pre-established occurrence of a maximum of three rounds and a consensus meeting in case there were still disagreements. This facilitated the interaction between the study participants and the responsible researcher, regardless of the geographic location.
The evaluation criteria included Scope and purpose, involvement of Stakeholders (or interested parties), rigor for development, clarity in presentation, applicability, and editorial Independence [31]. Stakeholders included healthcare professionals, computer scientists, industry representatives, policy makers, health informaticists, patients and funders.
Ten expert judges in the areas of health and/or information technology in Brazil, with professional experience of over five Years and minimum master’s degree, were selected by means of an intentional non-probabilistic sample. The researcher contacted the selected experts by e-mail, inviting them to participate and sending a link to the form to be completed, via an electronic questionnaire (Google Docs), accompanied by the first version of the operational protocol, defining the study objectives, instructions regarding the completion and importance of the evaluation of the instrument by the specialists. To gain access to the form containing the survey items to be evaluated, the participant was instructed to click on the link sent by email and to complete the evaluation within 20 days.
By clicking on the link, the participant was directed to the form with immediate opening of the Free and Informed Consent Term—TCLE. Only after the mandatory completion of the TCLE indicating the consent to participate in the research, the following evaluation screens were made available to the participant. Once the questionnaires were applied electronically, the responses were recorded directly in a database, avoiding errors associated with the researcher’s intervention in recording the responses provided. Two rounds were needed for expert consensus and Figure 1 presents the flowchart of the process of validating the proposed guidelines using the Delphi Method.
The profile of the expert judges participating in the validation of the guidelines was balanced in relation to the degree of master and doctor, 40% and 60%, respectively, and the average age of the experts was 46.2 years, with one (10%) being in the age group from 30 to 39 years old, seven (70%) in the age group from 40 to 49 years old and two (20%) in the age group from 50 to 59 years old. Regarding the time of professional activity, it was observed that 40% of the specialists had between five and ten years of professional practice, another 40% between eleven and twenty years of experience and 20% had between twenty-one and 30 years of career.
Regarding the field of activity of the participants, it was observed that the specialists were distributed in 20% in the field of Higher Education, 30% in Scientific and Institutional Research, 30% in Medical Assistance and 20% in Public and Private Management in Health. It is important to highlight the number of specialists who work daily involved with telehealth services aimed at Primary Health Care, totaling half of the sample of judges consulted in this research.
The domains evaluated by the experts in the validation rounds of the guidelines proposed in this research and the degree of agreement between the judges in the process are shown in the following table (Table 1).
Statistical analyzes were performed using the IBM SPSS Statistics program, version 22.0.0.0. The agreement between the judges was evaluated by the Content Validity Index (CVI) [32] and reliability verification was performed by Cronbach’s Alpha Coefficient. The proportional result of experts who judged the item as valid for the protocol by the total number of experts in the sample was considered as agreement/disagreement.
For consensus, a CVI greater than 0.80 was considered [33] and for the calculation of the CVI, on a seven-point scale, responses from 5 to 7, where 5 and 6
![]()
Figure 1. Flowchart of the validation process of preliminary guidelines using Delphi Method.
![]()
Table 1. Results of CVI and Cronbach’s α in each domain for the 1st and 2nd Delphi rounds.
(“important, but not critical”) and 7 (“important and critical”) will be considered as agreement, and responses from 1 to 4 (“not important/necessary”) will be considered as disagreement.
In relation to Cronbach’s Alpha, a coefficient that measures the relationship between responses in a questionnaire through the analysis of the profile of the responses made by the respondents, in the first round, a Cronbach’s alpha of 0.90 was obtained and in the second 0.93, indicating optimal internal consistency.
Before starting the protocol implementation stage, the researcher chose not to carry out any type of individual training of the study participants, with clarification of doubts and/or examples of practical application of the protocol to avoid conditioning the way in which each one would use it and, also, to extract more reliable results in terms of clarity, intuition, practicality, and easy understanding of the proposed instrument, primarily for use in teaching. The pilot test carried out in the TelesSUS operational environment served as a guide to ensure the most accurate verification of the efficiency of the protocol in practice and the identification of the necessary adjustments for a new test.
The intention was also to measure, through a semi-structured questionnaire designed especially for this research, how much the protocol helps the professionals involved and simplifies the construction of a process of developing a digital health tool to be used as a health intervention.
It is important to say that the protocol proposal is not intended to automate health techniques and procedures, nor to replace the specialist professional, but to complement and better manage patient care, streamline care, reduce costs, and provide critical reflection to patients, patients with respect to health care delivery standards.
4. Results: The SACI Protocol
The instrument was named SACI Protocol, acronym for “Software as Clinical Intervention”, friendly to the teaching of Brazilian students and professionals alluding to a well-known character in Brazilian folklore—Saci Perêrê [34]. The intention was to develop the protocol in the form of text with tables, illustrated with figures and flowcharts and divided into sessions as shown in the figure below (Figure 2).
The first section is dedicated to essential data referring to the research project developed. The second section is dedicated to important data for the initial meeting of the multidisciplinary team involved in the research project. The third
![]()
Figure 2. Distribution of protocol items into four sections.
section is dedicated to data on the development of the digital health tool and, finally, the fourth section is dedicated to data for project management, including the results of the patient intervention carried out with e-health.
The protocol checklist was designed in a table format (with a title involving the name and description of the checklist) consisting of the indication of each of the protocol sections and six columns, listed below and represented in Table 2.
1) The column called “Item”, where the study item for consideration must be referenced;
2) The column called “Subitem”, where the composition of sub-items for each item presented in the previous column must be referenced, when applicable;
3) The column called “Description”, where the description referring to the first two columns must be recorded in an essay form;
4) The column called “Reported (Yes/No)”, where it must be registered if there was an indication, referring to the previous columns, of the necessary information at the time of the research execution;
5) The column called “Status (Completed/under discussion)”, where the status of the items and/or sub-items referenced in the previous columns must be indicated, according to the moment in which the research was carried out.
6) The column called “Space to write down the respective information”, where pertinent information regarding the research must be described, for each of the items addressed.
During the construction of the protocol, some highlights were raised. It is necessary to pay attention to the fact that in interventions carried out with software, it is the patient/health professional who has control over the use of the technology, in terms of frequency and time of use. For this reason, it is extremely difficult to assess the actual results of these interventions and interpret the findings on their effectiveness, as many participants do not adhere to the proposed technologies and are not followed up for a long period of time.
Researchers, through metrics such as the number of logins in the software and the time spent in each session, can raise more reliable analyzes of the effectiveness of interventions in these trials, but it is important to provide additional and detailed information about these parameters, preventing, for example, that the user only logs in to the software but does not actually use it (establishing, for example, a maximum time for automatic logout), increasing the accuracy of the analyses. Primary and secondary outcomes of the targeted intervention should also be described in detail, including the methodology for extracting the results.
![]()
Table 2. Organization of SACI checklist.
Here, the importance of the analysis called ITT from the English “Intention to treat” or Analysis by intention to treat is highlighted. This analysis of results is based on the attribution of initial treatment and not on the one eventually received, precisely because of the difficulty in adhering to the interventions proposed throughout the studies. The ITT analysis recommends that participants be included in the analyzes even if they have not fully adhered to the research protocol, those who deviated from the protocol due to non-adherence or being removed from the treatment should still be kept in the analysis. The justification for this is that there is a need to estimate the allocation effects of an intervention during practice and not just the effects on the subgroup of participants who adhere to the first one [35].
Another highly recommended aspect is the creation of flow diagrams during the construction of the research and a friction diagram [19] [20] facilitating the visualization of participants’ behaviors and the influence of interventions with software in the patient care process.
The SACI protocol (PSACI) shown in Table 3 was lovingly conceived and improved to serve as a guide for researchers, mainly in the national territory, to conduct their studies under the theme of clinical trials with e-health. It is believed that these guidelines, as well as the few previous international publications with the same desire can have a positive impact on the quality of research reports carried out in Brazil and in the world with interventions based on new technologies, facilitating the learning of students and professionals, the replicability of studies, and the translation of the knowledge produced.
The first steps in clinical trials involve research registration, where the researcher must provide important data for the identification of his work. With that in mind, the first section of the protocol includes items on data that characterize the research project. In the second section of the PSACI, the items set out aim to include the fundamental information for the understanding of the study by all team members and for the alignment of expectations and possibilities during the formulation and execution of the work. Here, important issues are highlighted for understanding the scenario where the research is developed and the clinical algorithms to be considered.
For multidisciplinary teams composed of health professionals and IT professionals to be able to communicate more efficiently, it is important to raise questions about the functioning of the environment where the software will be implemented as an intervention, the protocols followed for patient care in this environment, the modus operandi of the professionals in the team in which the intervention will be applied and their expectations regarding the intervention’s outcomes for each phase of the patient’s treatment in which it is used. After extensive dialogues with students and professionals from both areas covered in this study, it is believed that these are the initial points of interest to clarify and align the work of multidisciplinary teams.
In the third section of the SACI protocol, the items provided aim to include fundamental information for the process of technological development of the
software that will be used as a clinical intervention in the treatment of the patient. In addition to the aspects directly related to the development of the tool that must be described with precision, respecting the organization and quality of the final product, what distinguishes the protocol created here from other existing theories of software development is the genuine concern both in relation to its possible effects such as health intervention and the way in which information from the entire process of creation and application of the tools is recorded, so that their reproducibility by other researchers is possible, facilitating the learning process and the dissemination of acquired knowledge.
It is understood that in the documentation phase of scientific research and in the phase of publication of the results, it is essential that the readers of the works can understand step by step how each stage of the development of new technologies was carried out, whether they are familiar with the topic or not. Adopting this attitude in scientific research contributes to the transparency of the work process and to the increase/assurance of quality both in terms of methodological aspects and in relation to the software itself.
In view of this scenario, we realized that in order to enable the creation of a software for health, as mentioned above, it is necessary to hold meetings with focus groups involving professionals specializing in the subject and to research information with key informants belonging to the place where the study is intended to be carried out. Using these strategies, it is possible to understand more clearly the needs of software users and to arrive more easily at the determination of requirements, also recognizing the importance of systematizing clinical and administrative processes and representing the workflow of professionals, including behavioral and environmental variables, in providing patient care. To this end, it is recommended that the first contact of students/professionals in the informatics and computing area for the development of tools is with patient care planning information.
When planning the intervention, the multidisciplinary team must consider the health situation of the patient involved with the implementation of the technology, observing physiological aspects and other clinical procedures to which he is already submitted in traditional care. With these aspects in mind, the third section of the protocol includes a checklist to verify that the important aspects were considered in the process of technological development of the digital health tool.
The fourth and final section of the SACI protocol is aimed at aspects of research project management, with the greatest emphasis being given to managing the digital health tool development process, involving domains of communication, management, implementation and project completion.
5. Final Considerations: The Value of Registration and Information
We believe that the guidelines suggested here, as well as those that already exist in international literature and that contribute so much to the quality of scientific research in health, will promote a healthy exercise of reflection regarding the importance of a well-prepared record of scientific information and compliance in manuscripts submitted to the available journals.
The guidelines were conceived mainly as a teaching tool, as a guide for novice researchers to build their way in the development of research, focusing on the quality of the product and the well-being of the patient involved in the study.
The judges participating in this study and the authors of this study believe that the constituent items of the SACI protocol are useful in this objective and start a good path for future discussions in the national literature.
The authors are aware that the construction of a list of minimum items is not enough to improve the completeness of scientific reports, but they believe that the initiative is a starting point for the creation of more specific instruments in the national literature to encourage more transparent reports for teaching, increasing the possibilities of reproducibility of higher quality research and medical-scientific evidence.
Acknowledgements
The lead author would like to thank all the experts who contributed to the creation and validation of the protocol, the support of the Universities involved for the publication of the article and all the expert co-authors equally involved in the construction of this research and data analysis.