Adopting Healthcare Information Exchange among Organizations, Regions, and Hospital Systems toward Quality, Sustainability, and Effectiveness

Abstract

Adopting data and information integration and exchange between two or more organizations, regions, communities, health insurance companies, and hospital systems has gradually become a requirement to improve health care quality, sustainability in management of work processes, and efficiency and effectiveness in financial and administrative management through cost monitoring and improved clinical protocols. The central aim of this study was to demonstrate how the use of Health Information Exchange (HIE) models helps increase quality, sustainability, efficiency, and effectiveness through data and information integration and exchange. Data collection were performed through an assessment of the literature in the timeframe between 2002 and 2016 referring to academic or non-academic organizations. The analysis of the literature considered Brazilian, American, and European articles. The results from the research on electronic databases were not satisfactory for the Brazilian scenario, where this theme is still very recent. Results from statistical, quantitative, and qualitative analyses showing data, indices, and the use of ICTs through dedicated software showed that health data and information exchange is capable of improving: 1) the quality of care given to patients and employees of the institution; 2) sustainability in management of work processes and activities pertaining to the relationship between providers, users, and service offer; 3) effectiveness and efficiency in operational cost monitoring and service protocols, giving the health institutions a prominent position in their fields in the eyes of external investors. Regarding the appropriate, legally- and policy-compliant use of data and information exchange and integration, organizations must obey legal requirements in conformity with each country’s current legislation. Suggestions for future studies are given at the end.

Share and Cite:

Salomi, M. J. A., & Claro, P. B. (2020) Adopting Healthcare Information Exchange among Organizations, Regions, and Hospital Systems toward Quality, Sustainability, and Effectiveness. Technology and Investment, 11, 58-97. doi: 10.4236/ti.2020.113005.

1. Introduction

With the advance of today’s information society, health organizations and systems increasingly use and demand Information and Communication Technology (ICT) resources to integrate and exchange data and information among organizations, regions, communities, service providers, and hospital systems.

The use of ICT is largely recognized as essential to meet national goals and reduce care costs, going beyond mere data processing to influence administrative and healthcare areas such as Telemedicine, Big Data, Business Intelligence (BI), etc. This advance is improving the quality and security of patient clinical data, as well as their availability on websites to health professionals. It is also helping eliminate duplication of services, reduce errors, and promote individualized patient care. Fundamental changes in the quality of healthcare delivery and the sustainability of detailed information on processes and results for a given population can now be supported, creating a basis to promote public health initiatives, control and monitor diseases, reduce costs, and increase effectiveness and efficiency.

One of the most utilized integrated management systems is Enterprise Resource Planning (ERP), whose main advantage is to offer a comprehensive overview of the business process to different organizational areas so as to increase their profitability and competitiveness.

Another advantage is the issue of quality in health care. Institute of Medicine Committee on Quality of Health Care in America (2001) defines quality as:

[...] the degree to which health services aimed to care for individual patients or general populations increase the chance to yield desired results and are consistent with current professional knowledge.

This means much more than just ensuring patient safety through drug interaction checks, electronic prescriptions, and medication checks. Quality dashboards can measure whether all patients are getting the right care at the right time. Clinical vigilance software can constantly scan Electronic Health Record (EHR) data to detect emergency signals and alert care providers about serious conditions such as sepsis and CAUTI, among others (Chassin & Galvin, 1998).

Business Intelligence (BI) solutions can also support practically all of a company’s processes by supplying an enormous amount of data and information, sharing them rapidly and meticulously, with accurate analytical precision to help improve decision making. Adopting BI: a) brings new opportunities; b) deepens the view on patient care, potentially increasing satisfaction with in/outpatient care; c) helps budgets by potentially increasing sales and reducing costs, thus promoting new business opportunities based on scenario analyses and efficient crossing of data and information on care, seasonality, and financial flows. This contributes to organizational sustainability and optimizes business intelligence about stakeholders.

Advancements in ICTs have also enabled the advent of Telemedicine and its subdivisions, bringing reductions in operational costs that result in increased effectiveness and efficiency, and faster diagnosis. By enabling medical activities at a distance, consulting, getting a second medical opinion, and exchanging data and information, telemedicine offers higher quality and sustainability, as well as technological advancements to the whole chain of medical care.

Furthermore, systems that combine various types of images to get the best from each one and then exchange these digital medical images permit to visualize exam results through a website, which, in turn, improves quality of care. Other interesting results are lower deadlines for test results, process automation, better reports, and better control of materials to avoid waste, resulting in more accurate projections, historical analyses, and reports. The result is higher-quality health care, more sustainable work processes, and effectiveness and efficiency in cost management.

In 2004 the Office of National Coordinator for Health Information Technology (ONC-HIT) created the Nationwide Health Information Network (NHIN) to establish standards, services, and policies for Health Information Exchange (HIE). Federal agencies and healthcare providers agreed to adopt the NHIN standards for HIE security at local and national levels, which was named eHealth Exchange in 2012.

HIE is a HIPAA-compliant software that enables healthcare providers to digitally exchange clinical data and information with other participant organizations. Patient data are combined without personal identification in compliance with local laws and policies. Therefore, the HIE system aims at safe, proper access and exchange of information on patient health to improve health care costs, quality, safety, and speed, increasing interoperability through EHRs held by doctors and organizations. This way, this system can improve diagnosis and results and avoid: 1) re-admissions; 2) medication errors; and 3) duplicated tests and exams.

In turn, HIMSS Analytics collects, analyses, and distributes basic health-related data on products, costs, metrics, trends, and purchase decisions, supplying adequate information and analytical expertise for healthcare organizations, pharmaceutical companies, government entities, etc.

Regarding sustainability, the World Commission on Environment and Development created by the UN (1987) defines sustainable development as:

(...) development capable of meeting the needs of the present generation without compromising the ability of future generations to meet their own needs, comprising three dimensions: the social, the environmental, and the economic dimensions ( ONU Brazil, 2020 ).

In 2012, this vision was reinforced in the final document from RIO + 20, paragraph 138:

(...) we recognize that healthiness is a previous condition, a result, and an indicator of the three dimensions of sustainable development. It is our conviction that measures toward social and environmental health determinants, both for poor, vulnerable populations and for the general population, are important to create inclusive, equitable, economically productive and healthy societies according to Corvalan (2014) .

Garde et al. (2007) mention the need to define and measure sustainability, which may be one of the main business drivers for developing a Health Information System (HIS). A HIS is a high-level structure, whose main enablers and inhibitors to sustainability consist of four pillars: clinical, technical, socio-technical, and political/business.

However, a frequent discussion is that health systems are not sustainable. Coiera (2007) , for example, posits that: “The health system at present is one that consumes enormous resource, and generates enormous waste, and would not meet any criterion of sustainability”. But when organizations exchange data and information, such as lab test results, history of in/out patient care, clinical case reports, and medication lists, through HIEs that comply with nationally- and internationally-recognized standards, improved quality, sustainability, health care efficiency and effectiveness, and results are expected.

Therefore, information, knowledge management, and communication technologies have become crucial drivers of change across health systems. These drivers are expected to play a vital role in sustainability as they transform health care systems and avoid failures. Information is a vital resource for the success of organizations, for more competitive companies will be those that better explore and use information to generate knowledge and develop new business opportunities ( Jussawalla, 1989 ).

With constant increases in healthcare expenditures, adequate management and the adoption of high-tech solutions have become necessary to help this sector balance budgets ( Hannah et al., 2009 ).

The current Brazilian scenario has not generally been favorable to these advances. However, technological management of hospital systems plays an important role in helping organizations establish a new dynamic in their relationship with other market players. In the last few months, the General Law for Data Protection (LGPD, in its Portuguese acronym) has stressed the demand from organizations to conform treatment of patient data and information to specific regulations. Healthcare institutions such as laboratories, hospitals, or clinics have higher prerogatives to deal with patient data and information, but are not exempt from adapting their procedures.

After the LGPD was sanctioned in August, 2018, some alterations have been made through the Provisional Measure 869/2018, which created the National Authority for Data Protection (ANPD, in the Portuguese acronym), responsible for enforcing data protection by corporations.

Considering the potential and relevance of data and information exchange among healthcare institutions, the aim of this study is to offer healthcare professionals a deeper understanding on how this technology can be used.

2. Methodology

Aiming to explore knowledge about the adoption of integration and exchange of health-related data and information in articles, websites, and books, we carried out a review of the literature focusing on quality, sustainability, and process effectiveness and efficiency.

The question formulated to guide the present study was: “Which are the benefits of adopting the exchange of health-related data and information aiming at quality, sustainability, effectiveness, and efficiency?”

The review of the literature was carried out on the following databases: PubMed, National Library of Medicine (NLM), National Institutes of Health, National Center for Biotechnology Information (NCBI) at the US, Elsevier, Harvard University, Stanford University and Articles of Health Affairs for works in English and Spanish. For works in Portuguese, we assessed the digital database Biblioteca Virtual em Saúde (BVS). We used the following key words: “Sustainability”, “Economy”, “Efficiency”, “Effectiveness”, “Quality”, “Electronic Health Record”, “Health Information Exchange” and “Electronic Patient Record”.

This study is descriptive in nature as it presents models for using Health Information Exchange (HIE) to exchange data and information among two or more health institutions, service providers, regions, governmental and non-governmental organizations. This study is also observational as, by means of quantitative, qualitative, and statistical data, we sought to give evidence of quality, sustainability, effectiveness, and efficiency in using HIE. And it is also technological in nature as, by using ICT through dedicated software, it may be possible to demonstrate the effectiveness and efficiency of these models.

The initial results from research in electronic databases were not satisfactory for the Brazilian scenario because the proposed theme is very recent in this country. In the United States and Europe, on the other hand, when we refined the search using the terms defined in the methodology, the results showed several publications—including studies, books, and governmental and non-governmental websites—which approached the theme in point.

The criteria for inclusion were: articles published between 2002 and 2016 in Portuguese, English, and Spanish, available in electronic media with integral text. We identified 50 articles, included in the bibliographic reference of this work, referring to the figures, citations and tables.

The criteria for exclusion were works published more than 10 years ago. Research sources that were not primary, that is, with original content, where the concepts and information were originally produced by the author(s) of the chosen source. National and international journals not indexed. The non-recognition by the author(s) who don’t have previously studied the problem. When the information or concepts presented by the authors did not demonstrate meaning to the meaning of the proposed research.

The final sample for this study consisted of 10 articles available in Portable Document Format (PDF) and 01 American organization website, which were adopted as the intellectual foundations on which all the research logic was structured. Thus, being compatible with the approaches that supported the investigation regarding the sustainability in the management of the processes; the quality of care for patients and employees and the effectiveness and efficiency of constant monitoring of costs and operations.

The results of the assessment were grouped along the areas described on Table 1. The idea was to classify the benefits of integrating and exchanging data and information among governmental and non-governmental organizations, regions, and hospital systems, in terms of: 1) Quality of health care (20%); 2) Sustainability in managing work processes and activities among providers, users, and service offers (20%); 3) Quality and Sustainability jointly approached (10%); and 4) Effectiveness and Efficiency in financial and administrative management

Table 1. Categorization of publications by article title, theme and year

through cost monitoring and improved service protocols (60%). These themes are presented in the Literature Review section.

3. Literature Review

3.1. Quality in Healthcare

Information is a powerful tool for institutions and organizations, as it allows to manage diverse parameters that dominate their dynamics ( Spinola & Pessoa, 1998 ).

A study conducted during the Winter Symposium of the American College of Medical Informatics (ACMI-2011) by Payne et al. (2013) has shown that the overlap between health ICT and economics, together with healthcare organizations, might help reform healthcare not only by offering new paths to improve value, but mainly by contributing to reformulate care delivery models. ICT may also be an important driver of fundamental change in healthcare systems, which are facing severe problems with rising costs and national deficit.

The Healthcare Information and Management Systems Society (HIMSS) has conducted a study on operational HIEs, through which different providers such as hospitals, clinics, public health agencies, laboratories, and long-stay institutions may access and send data. Basically, they can do it by two methods. The first one, pull, is based on search. The second one, called push, is directed. Data can be accessed by other providers through HIE, resulting in actual data transfer. Both methods ensure that, thanks to health care records, patients receive the best continued care even when relocated.

Khurshid & Luce (2012) recent assessment found a lack of clarity in the value of EHRs and HIEs. The authors (2012) recommend evaluating and advancing policies to articulate and measure the value of HIEs to users and patients, so that different stakeholders may support system-level transformations that enable exchange of health information.

Even with the significant progress in technologies and substantial investments, the rates of adoption of HIE are low. This may be attributed to resistance to change the healthcare delivery system, to a lack of incentives for efficient exchange of information, to the process of quality improvement, or even to safety in the existing reimbursement and regulatory environment.

Depending on the type of information shared, the messaging and search processes may be combined in different ways to meet a vast array of necessities, including: a) deliver diagnostic test results or a case report; b) input data to an immunization information system; c) review records for investigation of outbreaks; d) perform population health surveillance; and e) alert providers about opportunities to improve patient care, normally called Clinical Decision Support (CDS).

HIE usually refers to an exchange of information between healthcare providers. It also includes the possibility that patients View, Download, and Transmit (VDT) their own records using the Blue Button functionality (supported by Meaningful Use regulations), which has been a new turning point. Information exchange mediated by patients can open an alternative way for HIE to avoid some of the privacy and confidentiality issues in the exchange between providers, as shown in Figure 1.

However, due to the large number of players that occasionally try to communicate (doctors, hospitals, pharmacies, laboratories, and now even patients) it is not possible to establish individual communications with all potential exchange partners. Simple messaging features do not allow more sophisticated exchanges such as pull, neither integrate patient information from various care providers simultaneously. This is when a Health Information Exchange Organization (HIO) may enter the picture to supply these more complex exchange services, as shown in Figure 2.

Walker et al. (2005) estimated the annual benefit of HIEs for the national health system as around US $77 billion, based on reduced duplicate testing and better coordination of health care. Frisse & Holmes (2007) estimated more than a US $8 million annual saving due to use of HIEs in urban emergency situations in Tennessee. According to Hook et al. (2006) , the potential benefits of fully implementing a HIE in the state of New York could amount to $4.54 billion per year. Kho et al. (2008) estimated that three health systems in Indianapolis could save between US $2.3 and US $4.6 million annually by sharing information.

Another goal of these studies was to gather examples of metrics that could prove useful for both already-established and new HIEs. The authors believe that

Source: Blumenthal (2010) .

Figure 1. Structure of the HITECH Act for meaningful use of Electronic Health Records (EHRs).

Source: Blumenthal (2010).

Figure 2. Facilitating direct messaging with Health Information Organization Exchange (HIO).

this evidence is vital for continuous investment on HIE initiatives by communities, healthcare systems, governments, and others.

It is our opinion that a successful eHealth strategy is a critical factor in designing a sustainable model for future healthcare entities. A true, viable, population-focused eHealth model based on robust information technology includes:

· Sharing health care records to facilitate planning and communication between health care and social care providers

· Systematizing patient care to promote safety and effective care

· Planning coordinated care to guarantee that patients receive the right care in all possible healthcare scenarios.

Although every community has its own history, needs, and goals, a careful and continuous approach to eHealth is essential to transform health systems. A practical route to follow includes five components: 1) documented benchmarks and goals; 2) adoption of a proven, connected health record solution for the whole community; 3) investment to engage patients and providers; 4) better use of information at the point of care; and 5) continuous measurement and improvement.

Intersystems (2015) names 4 critical elements in an eHealth program:

1) The right foundation: shared virtual care records. The community-wide model has different stakeholders, regulatory contexts, processes, IT systems, and funding. The proper foundation for population-focused eHealth is not only the EMR, but also the shared virtual care record, that is, the patient’s electronic record, as shown in Figure 3.

2) The right tool for the job: speed at the point of service. Optimal population health management requires safe, high-quality care for everyone.

3) Do the right thing: the IHI describes this as the Triple Aim—excellent quality, optimal costs, and improved population health. Sharing care plans is a necessity for community health.

4) The right focus: actionable analytics. The saying “what gets measured gets managed” applies to healthcare. This is the reason why accurately measured quality and performance indicators have become globally widespread in healthcare systems ( Intersystems, 2015 ).

In eHealth, this new environment must include the community, engage patients, and improve results. Experience has shown that adequately implemented healthcare systems are able to: a) eliminate information silos; b) align a common plan; c) guarantee that participants have a better work tool; d) help continuous learning improve the system.

3.2. Sustainability in Managing Work Processes and Provider, User, and Service Activities

Brown & Ulgiati (1999) developed an important sustainability index called Emergy Sustainability Index (ESI). The emergy yield ratio is defined as the ratio of the emergy of the output of the system (Y) and the emergy of acquired services and resources that are input to the system (F). The Emergy Loading Ratio is defined as the sum of the emergy of local non-renewable sources (N) and acquired resources or services (F) divided by the emergy of the free environmental emergy available from local renewable sources (R) ( Garde et al., 2007 ). This indicator is shown in Figure 4.

Systematic corrective action, sustainably measured in a health information infrastructure, needs to identify and analyze the inputs, outputs and stored resources of the systems they use, thus resulting in a similar sustainable Emergy environmental analysis. Where the Input is the services and resources inserted

Source: Intersystems (2015) .

Figure 3. Shared virtual care record.

Figure 4. Formula to calculate the Emergy Sustainability Index.

into the system. The Output are non-renewable local sources system. And the stored resources are qualified as those used non-renewable. Being able to use different standards of measurement, e.g. grams, dollar, etc. Therefore, the objective here for Emergy is to make it possible to express the cost of a process or product in energy equivalent, which is considered to be a fundamental energy resource. For a given health system, it is necessary to identify and analyze the inputs, outputs and stored resources of system according to Table 2.

Under the perspective of enablers and inhibitors, the most important pillar of a clinical system is agreement on its clinical content, which fosters semantic quality and interoperability between systems. This, in turn, enables intersystem data exchange and migration, thus supporting clinical decision making.

A repository of free clinical content models may be developed and used thanks to a “flexible standardization” of their content, as suggested by Garde et al. (2007) . The selected technology should be able to cope with the constant changes in healthcare and health knowledge without the need to alter a large amount of source code. It should also provide a technical basis for semantic and sustainable interoperability. This is important as it allows systems to exchange data and information without loss; in addition, it avoids vendor lock-in.

As Moore (2006) states: “You need the ontology, the information model and services. [..] If you have one and don’t have the others, it won’t help”. For Garde et al. (2007) , ontology and services are relatively well understood, but the information model is mostly unknown. At this stage the openEHR approach, which is based on a stable, generic information model, becomes greatly relevant. The two-leveling openEHR dissociates technical knowledge (the software’s information model) from clinical knowledge (expressed in archetypes) to obtain semantic interoperability.

Every technology should be designed based on independent components, so that a replacement can be made at a certain point without compromising the sustainability of the whole infrastructure. Implementing an open code will help validate and evolve their specifications. Moreover, if the initial code base is good enough to let other people collaborate, the open source components can serve as building blocks for high-level HIS applications. Systems such as Linux, Apache, OpenBSD, JBoss, Hibernate, and Ant may be part of the present technical infrastructure and greatly contribute to sustainability.

Table 2. Inputs, outputs, storage for health information systems and inhibitors for their sustainability.

Source: Garde et al. (2007) .

The worldwide acceptance of EHRs has been accelerated by the rich input of clinical data by healthcare professionals. Transfer of inpatients to ambulatory or other models of fragmented service delivery using multiple healthcare providers, data transferability, interoperability, and access timing have grown in importance. This means that the growth of investigative EHRs will enlarge the population base to improve healthcare models and research results.

Thus, minimal data research has been growing in the world of data integration, health information exchange, support systems for clinical decision making, and knowledge creation, as huge investments are being made in this area of research.

Considerable obstacles must be overcome, including privacy and security issues, technological, economic, political, and professional practices, as well as social barriers that hinder the spread of EHR. In particular, we mention patient safety and high-quality care, ease and accuracy of access, and consensus so that researchers may use patient information.

In Europe and other countries around the world, for example, promoting the use of health data research has led to a cohesive development of the research platform and to accreditation and certification programs that ensure that the highest quality standards are met. This has contributed to establish legitimate better practices to create a sustainable ecosystem through innovation and excellence.

3.3. Effectiveness and Efficiency in Financial and Administrative Management through Cost Monitoring and Service Protocol Improvement

Information is a crucial strategic input that may influence businesses; it is increasingly becoming a basis for competition ( McGee et al., 2004 ).

In terms of effectiveness and efficiency, sustainability and quality of care, there are substantial visible and invisible costs associated to the implementation of hardware and electronic medical records, such as software license fees and constant updates, and lower productivity and speed levels while teams adapt to new operational standards. In addition to initial labor costs, medical companies will need internal or outsourced IT staff to perform backup, routine maintenance, and patient safety management.

The main tools available to deal with these issues are Total Cost of Ownership (TCO), which reflects the cost of acquiring new technologies, as well as all the costs associated to this acquisition, such as: staff, maintenance, training, users, failures, etc. ( Google Cloud QI Network, 2019 ).

Acquisition Costs relate to the purchase of software and hardware, taking into consideration:

· Purchase of equipment to setup or update the system’s structure

· Depreciation of hardware

· Work hours spent in market research related to choosing platforms and suppliers.

Implementation Costs relate to evaluation of:

· Hiring external consultancy to help configurate the system

· Occasional changes in servers

· Hardware and software implementation.

Maintenance and Support Costs are related to the implementation of new systems, functions, warrants, updates, licenses, staff, and outsourcing.

There is also the possibility to share costs related to the nature of operations, that is, direct or indirect costs.

The main characteristic of Direct Costs is that they are quantifiable. They involve:

· Support (training, manuals, trips, etc.)

· Software acquisition (licenses, updates, etc.)

· Management (systems and networks)

· Communication (infrastructure and fees)

· Development of content and applications.

Indirect Costs, which are hidden and unbudgeted, may often complicate measuring the investment made. These costs are those involved in casual support, the time during which tools or equipment remain inoperative, losses in productivity, repair processes, and setbacks that happen along the way.

It is important to take into consideration product and asset lifetimes, which are of two types:

· Depreciable lifetime: the number of years for product or asset depreciation. Durable goods, for example, usually have around a 5-year depreciable lifetime.

· Economic lifetime: years for the acquisition to give its owner financial return, which may be calculated based on the costs of keeping and operating it. When these costs exceed the product’s return, it is a sign that its economic lifetime has ended.

In addition to TCO, it is important to take into account the calculation of the Total Benefits of Ownership (TBO), which are directly involved in the purchase of a product or asset. Whereas TCO determines the cost of acquisition, TBO aims to evaluate the return of this investment. TCO is useful both to measure the impact of an investment in a certain area, and to serve as a basis for a comparative analysis between different products and assets, offering strategic alternatives. Using TBO precisely and accurately requires calculating the Total Return of Ownership (TRO), which quantifies the net return obtained through the technology. As these calculations use comparative tables, punctuation systems, and other similar tools, they are considered more complex than TCO. Because of that, TBO becomes a metric as crucial as TCO itself, as it uses inputs that are more comprehensive and realistic, such as associated costs, flexibility, and implementation risk.

In business systems, data and information are some of the most important resources for decision making. Organizations need an information network comprising diverse technical, scientific, managerial, methodological, economic, legal, environmental, and political aspects.

The Nationwide Health Information Network (NHIN) implemented in the United States on a state, regional, and local basis, with sustainability models and effectiveness in healthcare delivery and in Return of Investment (ROI), has been a significant stimulus to convert a regional strategy into a transformational long-term strategy. It should be introduced in diverse sectors with stakeholders’ initiatives aiming at external investment for HIEs.

There is also an opportunity for ICT positively influence healthcare expenses, providing a return on the investment on a national level ( Payne et al., 2013 ). The authors (2013) stress that the present dependency on data is noticeable both in health organizations and in healthcare providers, in terms of attention given to ontologies and terminologies, patient safety and its huge economic/financial implications, privacy, bioinformatics, population health, and healthcare equity.

Payne et al. (2013) study has shown how costly health care and health insurance have been for the US in the past 10 years. Of these expenditures, 31% is spent on hospital care, 21% on medical services, 10% on prescription of drugs, and 8% on nursing home care, with the remaining 30% dispersed around capital investment, insurance profits, administrative costs, home care, public health, and other health-related items.

On the other hand, American legislation requires better use of certified EHRs regarding the exchange of clinical data and report of quality metrics. The legislation proposed the application of the concept of Meaningful Use to demonstrate to both healthcare organizations and healthcare providers that using EHRs is not always associated to high levels of quality. Adjusting to the HITECH act is necessary to enhance care quality, safety, and efficiency. However, when properly adjusted, healthcare data help decisions made by providers, insurance officers, American government officers, patients, and families, increasing efficiency in communication and administrative functions.

Kaiser Permanente is one example of a US healthcare organization that implemented EHR to transform care and service delivery by improving quality outcomes, patient satisfaction and engagement, and management of a growing population. At the same time, this system reduced unnecessary services and inefficient processes to help the organization achieve the effectiveness of a top organization ( Payne et al., 2013 ).

Patients more and more use mobile applications for health monitoring and treatments; healthcare providers are increasingly becoming specialized to meet this demand. This renders HIS interoperability critical for the sustainability process. Consequently, potential investment on interoperability and sustainability should have a positive clinical impact.

Using HIE should take into consideration the Nationwide Health Information Network (NHIN), which is a set of standards, services, and policies that can be used by federal agencies and healthcare providers to safely exchange information on health and consumers via the internet on both a national and local basis. This initiative is sponsored by the National Coordinator’s Office (NCO) for Health Information Technology (HIT), which started developing the NHIN in 2004 to supply different entities a shared platform for exchanging health data and information all over the United States ( Rouse & Holman, 2019 ).

Note that the NHIN is a fundamental prerequisite for information exchange on a national level. It is also one of the main objectives of the Health Information Technology for Economic and Clinical Health (HITECH) Act enacted by the American government in February, 2009 with the purpose of structuring how the country should collect, store, and use health-related information. In the healthcare field this program involved the commitment to digitalize the US health information system. This act would assign up to 29 billion dollars along 10 years to support the adoption of Meaningful Use in Electronic Health Records (EHRs), aiming to improve health and healthcare, as well as other types of health information technology ( Modena, 2019 ).

Implementing NHIN depends on state, regional, and local HIEs with sustainable and effective models for sharing electronic information that demonstrate improved efficiency and effectiveness in health care delivery and Return on Investment (ROI) ( Khurshid & Luce, 2012 ). ROI Analytics, in its more orthodox economic version, is a powerful tool that CIOs use to quantitatively measure the potential success of a business or IT project. It is also used to measure the financial benefits of an investment versus the financial return yielded, as well as to guide organizations in evaluating how and where to apply resources. ROI Analytics is associated to improved procedures and patient satisfaction; it is a constant in academic and non-academic discussions about how to maintain quality standards in medical care.

ROI is calculated as the Return/Benefit on an investment minus the Cost of the investment divided by the Cost of the investment, resulting in a percentage or ratio, as shown in Figure 5.

Dupont et al. (2017) performed a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis of the Electronic Health Record for Clinical Research (EHR4CR) platform as a service provider to identify and assess solutions for the political, economic, social, technological, and legal aspects of its comprehensive multi-stakeholder market and business ecosystem.

Dupont et al. (2017) investigation revealed perceived needs and potential opportunities in developing and implementing technological solutions that may improve sustainability in clinical research processes along the following themes:

· Demographic information

Figure 5. Formula for calculating return on investment (ROI).

· Areas for improvement in clinical research;

· Key opportunities

· Expected benefits from the EHR4CR platform and services.

Therefore, internal and external stakeholders act as EHR4CR partners from different Information Communication Technologies (ICT) areas, such as academic centers, vendors of EHR and Electronic Data Capture (EDC) systems, Contract Research Organizations (CROs), clinical research sites, patient lawyers, hospitals, health agencies, pharmaceutical industry, etc.

The market analysis emphasized the importance of developing a business model comprising multi-stakeholders with shared interests, as shown in Figure 6.

The value chain illustrated in Figure 6, developed by the Business Modelling Innovation Task Force (BMI-TF), consists of a market where distinct groups of stakeholders provide each other with network benefits. In a traditional value chain, value moves from left to right. To the left is cost; to the right is revenue. So, this value chain shows how stakeholders are interdependent in creating, delivering, and optimizing value across the EHR4CR network.

The BMI-TF identified a key group of stakeholders and segmented them into 4 categories: Providers, Funders, Users, and Beneficiaries of EHR4CR solutions.

Using the perspective of the EHR4CR for service providers, the BMI-TF developed a Canvas business model with the following building blocks:

· The customer segments of the EHR4CR service providers

· The type of customer relationships and channels to be developed

· Activities

· Resources and partnerships to be implemented and developed

· The cost structure and revenues that will permit to deliver the value propositions to the customer segments, in a sustainable way.

The financial scheme was developed considering an EHR4CR service provider with high estimated expenses and revenues. The expenses were estimated in relation to the scope of activities to be implemented. These aspects were defined

Source: Dupont et al. (2017) .

Figure 6. Optimizing the EHR4CR Multi-Stakeholder Value Chain.

and discussed by the BMI-TF considering the scope of the EHR4CR ecosystem, the multi-stakeholder value chain, the market analysis, the business intelligence (BI) gathered during strategic forums, and the perceived success factors. An implementation roadmap was also designed to leverage the sustainability strategies to optimize delivery of the EHR4CR benefits to stakeholders ( Dupont et al., 2017 ).

The clinical research processes identified the following needs:

· Patient identification for recruitment in clinical trials (70% of respondents)

· Time needed to conduct clinical trials (59%)

· High costs and workload in clinical research (54%)

· Protocol feasibility for taxation/fees (50%)

A market analysis has also emphasized the importance of developing a business model encompassing multiple stakeholders with mutual interests. The value propositions developed for the EHR4CR platform reflect the unmet needs of different stakeholder segments, which are summarized in Table 3. These value propositions were proven relevant while the EHR4CR technical approach and

Table 3. EHR4CR value propositions adapted by stakeholder group.

Source: Dupont et al. (2017) .

business model were being developed and refined, thus confirming the expected benefits, efficiency, effectiveness, and success factors that would be necessary for each stakeholder to accomplish them.

In order to assess the economic sustainability of EHR4CR in Europe, the number of commercial and non-commercial sponsors for a 5-year period was estimated. The estimated per-patient costs (in US$) obtained through access of the Cutting Edge database during the first quarter of 2013 include: a) patient recruitment; b) supplier fees; c) technology; d) site retention de; e) data cleaning; f) statistical analyses; g) reports; and h) patient retention costs.

As global average per-patient costs vary across therapeutic and geographic areas and considering volatile exchange rates over the duration of the project, Dupont et al. (2017) used a conservative average per-patient cost of ?0,000 and an estimated average of 250 - 500 patients per clinical trial (CT) to propose the business model. To estimate the potential fees for service, the model applied increasing percentages to the baseline estimate of per-patient costs, multiplied by the estimated average of patients per CT.

Using the statistics database called EudraCT 2014, the projected values were then compared, confirming the accuracy of the BMI-TF estimations. Phase IV Clinical trials were defined as well-controlled, post-registration clinical trials focusing on special populations, or comparing a new intervention with current treatments. This is shown in Table 4.

As a result, the BMI-TF estimated that 5 to 15 EHR4CR service providers could jointly reach a 5% - 10% market share in Europe over 5 years. The potential 5-year market size (number of potential CTs) per EHR4CR service provider was estimated by applying the EHR4CR market shares to the estimated number of CTs in Phases II, III, IV that could use EHR4CR services, divided by the estimated number of service providers. This is shown in Table 5.

The widespread development of EHRs in Europe and the development of the EHR4CR platform are proving to be an innovative, pioneering opportunity for research using EHR health data on a scalable approach. This may generate new business opportunities with significant growth potential for all involved stakeholders,

Table 4. Estimated ranges of commercially and non-commercially sponsored Phase II, III, IV CTs in Europe.

Source: Dupont et al. (2017) . aEudratCT 2012 (based on estimated baseline). bRanges derived using pharmaceutical industry 5-year projections (2014-2018); Clinical Trials (CT).

Table 5. EHR4CR business model assumptions.

Source: Dupont et al. (2017) . aRanges established by the BMI-TF. bDerived from applying 5% - 10% to the estimated 5-year forecasted ranges (Table 4). Clinical Trials (CT).

as well as add value to those responsible for clinical trials, thus benefitting patients, health systems, and society at large.

The resources to be deployed include IT experts, sales and marketing resources, management and administrative support. Standards should be established with key stakeholders to: a) accelerate the adoption of EHR4CR services; b) identify needs of new business and channels; and c) develop new solutions. By promoting the use of health data research, guiding a cohesive development of research platforms, and designing accreditation and certification programs to ensure that the highest quality requirements and standards are met, this system will contribute to establish true best practices and achieve a sustainable ecosystem driven by innovation and excellence across Europe and the world.

Forrester et al. (2014) , in another study conducted in the USA, estimated the cost-effectiveness of Computerized Provider Order Entry (CPOE) in reducing medication errors and Adverse Drug Events (ADEs) in the ambulatory environment. They created a decision-analytic model to estimate the cost-benefit relationship of CPOE in a multi-disciplinary, mid-sized medical group with 400 providers over a 5-year period (from 2010 to 2014). During this timeframe, these health systems were implementing CPOE to meet the Meaningful Use criteria. The authors (2014) adopted the perspective of a medical group with their costs, changes in efficiency, and actual number of errors in medication and ADEs. One-way, probabilistic sensitivity, and scenario analyses were conducted in the explored setting. The results showed that CPOE cost US$18 million less than paper prescriptions and that it was associated with over 1.5 million fewer medication errors and 14,500 fewer ADEs during the 5-year period. In a setting with five providers, CPOE cost US$ 265,000 less than prescriptions on paper and was associated to 3875 fewer medication errors and 39 fewer ADEs over 5 years. CPOE was also dominant in 80% of the simulations.

Published systematic reviews suggest that CPOE is associated with a 13% to 99% reduction in medication errors and a 30% to 84% reduction in Adverse Drug Events (ADEs) by Ammenwerth et al. (2008) and Shamliyan et al. (2008) . In the ambulatory environment, CPOE, even with limited clinical decision support alerts to guide orders, was associated with a 55% reduction in errors, by Devine et al. (2010a) .

CPOE has gained support despite reports of initial problems related to drop-down boxes according to Devine et al. (2010b) , resistance to adoption by Weiner et al. (1999) , interruption of workflow according to Poon et al. (2004) , increase in workload, and even increase in the number of errors by Ash et al. (2009) . In the past 5 years it has become an integral part of learning in the healthcare system according to Koppel et al. (2005) and Johnson (2011) . Current research is working on alert fatigue using methods from human factor engineering by Olsen et al. (2007) and Riedmann et al. (2011) . A new area of focus is the workflow according to Scott et al. (2011) . New works continue to improve CPOE systems with clinical decision support to further reduce medication errors and increase prescribers’ adherence to guidelines by Baysari et al. (2011) and Austrian et al. (2011) .

One of the main barriers to adopting CPOE and EHRs has been the high initial investment. Incentives to utilize Meaningful Use reduce this cost barrier, promoting higher acceptance with the ultimate goal to improve patient safety. However, few studies have estimated the system’s long-term costs in relation to its safety benefits.

The Everett Clinic works with approximately 18 health plans. Each plan has its own clinical formulary, and together they order 2.7 million prescriptions per year. In 1995, the Everett Clinic acquired a web-based EHR system with point-and-click functions and electronic prescribing integrated into an existing EHR. This system included scheduling, chart notes, laboratory and imaging reports. A basic CPOE software was designed in 2004 and introduced between 2004 and 2006 to generate new and refill prescriptions. Clinical decision support alerts were limited to basic dosing guidance, duplicate therapy checks, and calculation of pediatric dosing. The prescriptions could be electronically transmitted to more than 200 local pharmacies in Figure 7.

Source: Forrester et al. (2014) .

Figure 7. Decision analytic model ADE, adverse drug event; CPOE, computerized provider order entry.

The costs of the system represent those costs actually incurred by the Everett Clinic to implement CPOE in the first 3 years (2004-2006) and the annual, recurring costs between years 1 and 5 (2004-2008).

Using the system in other medical groups may result in more drastic cost savings and higher safety if the computers are installed in examination rooms, or if the system has more sophisticated alerts to support clinical decision making. However, the groups which adopt CPOE but are unable to make a complete transition may not profit from cost savings as great as the ones observed for the Everett Clinic. The estimates and their respective intervals for all cost and error/ADE probabilities inputs are listed in Table 6.

Table 6. Cost and medication error/ADE inputs: Base-case and scenario analyses.

Source: Forrester et al. (2014) . *All costs are in 2010 US dollars. Hardware costs in year 1 reflect wireless installation; hardware costs in years 2 and 3 reflect switch to wired installation of computers in most examination rooms; hardware costs in years 4 and 5 reflect installation in remainder of the examination rooms; the expected life of hardware is 5 years. Software costs include the cost of the operating system, drug database, and virus protection; maintenance costs include costs for maintaining secure Internet connections and costs for parts and labor for failed equipment. Personnel costs reflect software development, software maintenance and updating, testing and training, and ongoing helpdesk support. Personnel costs are included for the following types of personnel, each working part-time on CPOE implementation: one project manager, two programmers, one network security administrator, one database administrator, and one application support person. The helpdesk was staffed by one technical person and two clinical pharmacists who specialized in CPOE implementation. §For first year only. # Meaningful Use incentives truncated at year 4 to align with 5-year life of computer hardware.

The CPOE strategy cost US$ 18 million less than prescriptions on paper and was associated to approximately 1.5 million fewer medication errors and 14,500 fewer ADEs. The probabilistic sensitivity analysis confirms that CPOE was the dominant strategy in 99.6% of the Monte Carlo simulations for medication errors, and in 98.9% of the simulations for ADEs, as presented in Table 7. The one-way sensitivity analyses reveal that the main determinant factors of uncertainty in the models are the salary and number of specialty care providers, the number and cost of charts, and the number of prescriptions ordered per year. The cost-effectiveness acceptability curve illustrates the probability that the CPOE system is profitable at various levels of willingness-to-pay per ADE avoided. The probability that the CPOE system is cost-effective is more than 98% until willingness-to-pay reaches approximately US $16,000, when it drops slightly, remaining steady at more than 97% for the remaining levels.

With an over 95% probability that a CPOE will be profitable at a willingness-to-pay of approximately US $10,000 to avoid an ADE, there is strong evidence that its implementation offers a good cost-benefit relationship to improve safety in prescribing medication, resulting in quality of patient care and sustainability of

Table 7. Types of errors, administrative and clinical, and probabilities

Source: Forrester et al. (2014) .

the whole process. For groups that are unable to implement a complete EHR at once, CPOE—or a basic electronic prescription system—may be an important first step to achieve effective and efficient cost monitoring, taking into account patient safety and cost-saving benefits. Forrester et al. (2014) findings support the initiatives of the Institute of Medicine and the CMS to promote the adoption of CPOE and electronic prescribing, because this system resulted in safer medication prescription and cost savings. It also promoted actual efficiency and effectiveness in the process, and increased quality and safety in patient care as well as sustainability in the work processes.

The application of information technology as done by the Institute of Medicine is one of the main ways to improve health care quality ( Wang et al., 2003 ). Two categories of costs associated to implementing electronic medical records were identified: system costs and induced costs, as seen in Table 8. System costs include costs of software, hardware, training, implementation, and ongoing maintenance and support. Induced costs are those involved in the transition from a paper-based system to an electronic one, such as the temporary decrease in supplier productivity after implementation.

Software costs (US$ 1600 per provider per year) include the costs of system design and development, interfaces to other systems (e.g., registration, scheduling, laboratory), periodic upgrades, and user account costs for support staff. Even though these software costs are based on an internally-developed system, they are consistent with the license fees for commercially available systems (estimated at US $2500 - 3500 per provider for the initial software acquisition), added to annual maintenance and support fees of 12% to 18%. In sensitivity analyses, these software costs can vary between 50% and 200% of the baseline value.

Implementation costs were estimated at US$ 3400 per provider in the first year. They include redesign of the workflow process, training, and historical abstracts of charts on paper. Ongoing maintenance and support costs were estimated at US$ 1500 per provider per year. They included the costs of additional technical support staff and administration of the system/network. Hardware costs were calculated at US$ 6600 per provider for three desktop computers, a printer, and network installation. Hardware was assumed to need replacement every 3 years.

Based on their experience, Wang et al. (2003) modeled the induced costs of temporary loss of productivity using a decreasing approach, assuming an initial productivity loss of 20% in the first month, 10% in the second month, and 5% in the third month. Subsequently, productivity would return to baseline levels. Based on the average annual provider revenues for their model panel of patients,

Table 8. Costs of electronic medical record system used in the model (per provider in 2002 US$).

Source: Wang et al. (2003) . *Data from Partners Healthcare System, Boston, Massachusetts. B. Middleton, MD, MPH, MSc, MedicaLogic, written communication, 1998.

this represented a loss of revenue of US $11,200 in the first year.

The financial benefits included avoided costs and increased revenues, as well as overall efficiency. Wang et al. (2003) obtained the average annual expenditures for a primary care provider before an electronic medical record was implemented and applied to them the percentage of the estimated savings after implementation, as shown in Table 9. The estimated savings for each item varied between the values indicated in the sensitivity analysis. The benefits were classified into three categories: payer-independent benefits, benefits under capitated reimbursement, and benefits under fee-for-service reimbursement.

Standard financial benchmarks were used to attribute the baseline costs for adverse drug events, taking into consideration additional ambulatory visits, prescriptions, and admissions. The experts estimated that alerts for alternative drug suggestions would help save around 15%—ranging from 5% to 25%—of total annual drug costs. This initiative was applied to the baseline annual drug expenditures for the capitated patients in the study. The experts also estimated a reduction of up to 8% (ranging from 0% to 13%) in laboratory charges and savings in radiology orders around 14%—ranging from 5% to 20%—due to use of support to decision making.

As for the benefits from fee-for-service reimbursement, increased revenues and reduced losses improved the quality and sustainability of savings after implementation. Digitalizing the form process improves the capture of internal

Table 9. Annual expenditures per provider (in 2002 US$) before electronic medical record system implementation and expected savings after implementation.

Source: Wang et al. (2003) . *Primary data from the Partners HealthCare Electronic Medical Record System, Boston, Massachusetts. From the Department of Finance, Brigham and Women’s Hospital, Partners HealthCare System. Expert panel consensus.

procedures that were performed but not documented, resulting in a 2% growth in billing capture. The use of an electronic medical record system that is able to supply or request the filling out of required fields may reduce financial losses due to billing errors. Wang et al. (2003) estimated that digitalizing the form would lead to a 70% decrease in errors. They also assumed that initial costs would be paid out at the beginning of the first year and that benefits would accrue at the end of each year, as demonstrated in Table 10.

In the 5-year cost-benefit model (Table 10), the net benefit of implementing a full electronic medical record system was of US$ 86,400 per provider. From this amount, drug expenditure savings represented the major part of the benefits (33% of the total). In the remainder of the categories, almost half of total savings came from decreased radiology utilization (17%), decreased billing errors (15%), and improved billing capture (15%).

The “light” electronic medical records, which use the system only to reduce costs related to paper-based chart pulls and transcriptions, presented a net cost

Table 10. 5-Year return on investment per provider for electronic medical record implementation.

Source: Wang et al. (2003) . *Assumes a 5% discount rate.

of US$ 18,200 per provider, as shown in Table 11. For the “medium” electronic medical records, which have added-benefits from electronic prescribing, the net benefit was US$ 44,600 per provider.

This analysis has demonstrated that the net financial return to a healthcare organization from using an ambulatory electronic medical record system is positive across a wide range of assumptions. The main benefits are: a) reduction in drug expenditures; b) better use of radiology tests; c) improvement in charge capture; and d) decrease in billing errors. The benefits increase as more and more features are used and the time span is extended.

Both capitated and fee-for-service reimbursement produce savings to the healthcare organization, but these savings depend on the reimbursement mix: the greater the proportion of capitated patients, the greater the total return.

This includes cost savings in inadequate care, storage and supply costs, substitution of generic drugs, increase in provider productivity, decrease in staff needed, increase in reimbursement due to more precise evaluation and management coding, and decrease in claims denied because of inadequate documentation of medical requirements. The model considered a temporary, 3-month productivity loss because some providers suffered a longer period of reduced productivity. To measure this effect, a sensitivity analysis was carried out including a prolonged productivity loss of 10% a year. A 5-year net benefit of approximately US$ 57,500 per provider was demonstrated.

According to Hincapie & Warholak (2011) , the focus on using HIE for health care results has shown evidence of cost savings in some departments, such as Emergency Departments (EDs). The aim was to observe the relationship between HIE and health care measures (such as increased costs or quality of service) and to analyze how using HIE in healthcare may affect care settings such as EDs, hospitals, primary care, etc.

The assessment of ninety-four analyses yielded results concentrated on health

Table 11. Effect of electronic medical record feature set variations on net benefits.

Source: Wang et al. (2003) .

care use (71.3%), followed by health care costs (11.7%), quality of care (6.4%), care coordination (6.4%), patient experience (2.1%), and disease surveillance (2.1%). A total of fifty-four analyses (57.5%) found a beneficial effect of HIEs on the result, but 31.9% reported no significant effect of HIEs, and 10.6% found an adverse effect of HIEs on the outcomes of health care. The statistically significant beneficial outcomes and the study’s characteristics are presented in Table 12.

Despite recognizing the particular relevance of HIE in settings where rapid access to information is of critical importance (such as EDs), future studies should focus on other settings besides hospitals and EDs, such as primary care, public health, pediatric inpatient care, and long-term care facilities. Such studies should consider patient populations still unrepresented in the literature, including

Table 12. Frequency of reporting a beneficial relationship between health information exchange (HIE) and various study characteristics.

Source: Rahurkar et al. (2015) .

patients with chronic diseases (e.g., diabetes, asthma, cancer, congestive heart disease, and mental health conditions).

4. Results and Discussion

Efforts toward adoption of data and information integration and exchange among organizations, service providers, communities, regions, and hospital systems have increasingly demanded the use of ITC resources to get purpose and commitment from diverse players. Sharing data and information is still in an incipient stage, as are initiatives that allow different IT systems to communicate. The result is that data remain “stuck” in isolated databanks.

Many countries have realized the need to join efforts aimed to develop the adoption of standards for data integration and exchange. The HIMSS study has demonstrated that the adoption of operational HIE by hospitals, clinics, and laboratories helps ensure that patients receive continued care and the best treatment even when they have to relocate regionally.

After reading several articles we found works that sought to demonstrate actual gains in quality of care, sustainability of process management, and effectiveness and efficiency in cost monitoring thanks to data and information exchange. The works used as reference were conducted by Walker et al. (2005) ; Wang et al. (2003) ; Garde et al. (2007) ; Bates & Bitton (2010) ; Payne et al. (2013) ; Khurshid & Luce (2012) ; HIMSS Europe (2013) ; Forrester et al. (2014) ; Intersystems (2015) ; and Dupont et al. (2017) .

The low level of adoption of an investment in HIE in 2012 can be considered a consequence of resistance to change in the healthcare delivery system, of a lack of incentives to efficient exchange of information, or even quality processes and the need of more reimbursement security in the regulatory environment. This is true even in face of the significant progress in recent years in technologies to support this adoption.

Khurshid et al. (2015) demonstrated that most organizations operating HIEs are nonprofit organizations with a budget of at least US$ 1 million. Hospitals, ambulatories, care clinics, and laboratories are the most frequent players in HIEs. The three data categories exchanged are: test results, medication summaries, and information on patient ambulatory care. In fact, the digital age is transforming the world through technological innovations to bring about gains not only in the quality of patient care but also in process sustainability and effectiveness of cost monitoring.

Considering the use of HIEs to obtain a desired ROI, Khurshid et al. (2015) reported that two thirds of their respondents agreed that community HIEs yielded a positive ROI, whereas one third of the respondents did not have an opinion or simply disagreed. The authors (2015) also demonstrated improvements in care quality and delivery. The challenge faced by policy makers and healthcare organizations, who are investing millions of dollars in HIE to improve efficiency and effectiveness in healthcare results, is the time needed to gather evidence to confirm this belief.

Comparative studies of HIE, despite being extremely challenging in a real situation in the current world, can be adopted through randomized clinical practices, thus enabling improvements in the population’s health outcomes, as well as the use of adopted outcome measures that focus on the quality and coordination of health care and disease surveillance in the context of population health management.

The continued focus on costs and patient satisfaction measures as well as the inclusion of information on the actual use of HIE in terms of the type of information (such as laboratory tests and imaging) and the magnitude of the case of care outcomes health should also be considered.

Note that the most important pillar of data and information exchange inside a clinical system is clinical content. Semantic interoperability between systems is crucial to enable accurate data migration to support clinical decision making. When an inpatient is transferred to ambulatory care or to another fragmented service model, the main purpose of data and information portability, interoperability, and time to access is to collect data to improve the population base for healthcare models and research results.

Dupont et al. (2017) described a business ecosystem that leverages multi-stakeholder EHR4CR inside a value chain. Value propositions were developed to describe the expected benefits from solutions to this system for all its stakeholders. From the system’s perspective, a business model simulation yielded an estimated profitability level of up to 1.8 that could be reached in the first year, with a growth potential in subsequent years depending on the projected market.

On close inspection, many healthcare institutions have already made efforts to digitalize their basic administrative processes, such as patient admission, centralized consultations, discharge, or medical episode recording with relevant coding. This has helped improve some processes and may enable to develop studies on hospital activities. However, nowadays organizations have to deal with situations that demand criteria for real-time management and automation in order to achieve sustainability of the whole process. Administrative tasks have to be eliminated not only from clinical practice but also from patient relationship management toward enhancing quality of care, as well as efficiency in cost management. Telemedicine, Big Data, and Business Intelligence have greatly contributed to reaching these goals.

Forrester et al. (2014) demonstrated that the CPOE system for electronic prescription resulted in a healthcare organization saving US $18 million by reducing paper prescribing. This system was also associated to 1.5 million fewer medication errors and 14,500 fewer ADEs in over 5 years, being dominant in 80% of the simulations. The cost-effectiveness acceptability curve showed the probability of the CPOE system being profitable at various levels of willingness-to-pay per ADE avoided, indicating that the system has a more than 98% probability of being profitable and stable.

Wang et al. (2003) sought to demonstrate the net estimated benefit from using EMR during a 5-year period. The value per provider was US $86,400. Of this amount, 33% represented savings in drug expenditures, whereas of the remaining amount, 17% came from decreases in radiology exam utilization, 15% from reduced billing errors, and 15% from improvements in charge capture. It was made clear that the net financial return on an ambulatory electronic medical recording system is positive for a healthcare organization. This includes effectiveness and efficiency, increased provider productivity, as well as savings in malpractice, storage and supply costs, generic drug substitutions, and so on. A sensitivity analysis with the most optimistic and most pessimistic assumptions showed results ranging from a net cost of US $2300 to a net benefit of US $330,090. As demonstrated by the tornado diagram showing the one-way sensitivity analysis of net 5-year benefits per provider. Each bar depicts the overall effect on net benefits as that input is varied across the indicated range of values, while other input variables are held constant. The vertical line indicates the base case, by Wang et al. (2003) .

Note also that EHR is an innovation that enables logistic and administrative reorganization, as well as control of patients, drug usage, and costs. EHR offers more rigor and accuracy, since data and information are recorded into a system that integrates the care delivery and administrative areas, with the possibility to generate performance indicators for these areas.

For the moment, the priority in most healthcare organizations is to substitute paper documents for electronic records. Even though these efforts may help increase revenues, the impact of IT on cost monitoring and quality improvement in clinical care has been modest and limited. The greatest facilitator is Patient Electronic Medical Records.

Only a few organizations make a real effort to analyze the data in their IT systems to assess effectiveness of the care provided. In other words, most organizations still use ITs only to evaluate potential ways to improve processes and protocols, and only when they get an affirmative answer do these measures become actionable.

A series of internationally-renown organizations, such as the Geisinger Health System and the Mayo Clinic, have shown that health care IT is really effective when all organization members work jointly to achieve its full potential. When this is the case, IT has the potential to increase patient care quality, sustainability in work process management, and effectiveness and efficiency in cost monitoring.

When the IT system becomes integrated to increase communication among professionals and multi-professionals and to coordinate patient care, significant results may be expected, such as: decreased mortality rates and expenditures with post-treatment of acute conditions, and increased profitability in the services provided.

Currently, organizations that want to track and measure improvements in the quality of the care provided must have high-quality data at the right moment. However, the task of collecting information has slowly shifted from clinical professionals to patients, with patient treatment plans being recorded through health monitoring applications. The actual goal is to make this data integration increasingly passive. Some organizations are already using this mechanism to track workflow-related operational issues. These data help departments understand care delivery, identify operational barriers, and, consequently, eliminate these barriers from the workflow.

We believe that one of the most critical tasks is how to structure the systems— not on a technical level, but in terms of the scope of the organizational or cultural guidelines needed to choose the best method to use data in healthcare.

Another advantage worth mentioning is adoption of the data dashboard tool to evaluate what is and is not working in various areas. This may help the institution obtain quality metrics, such as hospital infection rates, receive alerts about changes needed in work processes and how to implement them, or even envision how the collected data may improve productivity.

By applying Analytics, healthcare organizations can understand operational workflows and thus reduce costs, enhance efficiency and effectiveness in management systems, and eliminate unnecessary treatments to increase quality of patient care.

As posited by Brown & Ulgiati (1999) , measuring the sustainability of a health information infrastructure should consider the system’s inputs/outputs and storages, as well as analyze which parts are renewable or not. Which factors can be used to convert these inputs/outputs into the Emergy unit must also be determined. Once the result from these investigations is known, all the Emergy-based, environmental sustainability indices can be applied to health information systems, input/output analyses, and stocked resources. Other similar indicators can also be applied to HIS.

The IT infrastructure is another critical element. Current systems are too rigid; they do not enable personalization, insertion and extraction of information, and updating for new clinical protocols. Additionally, as different systems cannot integrate patient data and information, it is impossible to have a unified health data record that can be accessed by any professional. It is also impossible to exchange anonymous patient data that can be collected and utilized in research on new treatment forms, an issue that Blumenthal (2010) has also pointed out.

When care is delivered with quality and according to patient wishes, resources invested in eHealth will return to the health system itself. Virtual care records aggregate and normalize information from all participating sources—including patients—enabling a relevant care format with an enhanced process. The eHealth model encompasses the community, engages patients, and reduces costs. The model also eliminates information silos and, when aligned around a common plan, it ensures that all players have a better tool to work and learn continuously, thus improving the system.

We highlight the comment made by the medical director of Pró-Cardiaco Hospital, Evandro Tinoco, on a strategy by Anahp:

“(...) to consider the future in terms of triggers for change in care centered on patients, on electronic medical records, family experience with patient care, care results, and Digital Medicine, among others.”

5. Conclusion

The publications analyzed reveal that technological advances in healthcare are continually offering new ways to improve quality in patient care and advance the state-of-the-art in the digital healthcare system on a global basis.

Information technology has contributed significantly to this area. IT demonstrates that data and information exchange and integration, together with the use of healthcare standards, do not depend exclusively on technical aspects, but also on policies to support the exchange of these data and information in compliance with local legislation and policies.

Certainly, HIE can possibly be considered as a transformative electronic data tool with regard to health care management. However, other factors also deserve our attention: the clarity and discernment in the adoption and use of an evaluation methodology for the implementation of projects and investments, which is able to monitor and measure its impacts on society, such as: sustainability of processes; quality in health care and efficiency and effectiveness not only in monitoring costs, but also in improving services and protocols.

It is also worth mentioning factors such as: multidisciplinary teams with experience and permanent training in process management, organization in operational management; and clear long-term incentives and perspectives, as well as resilience and feedback.

The quest for indicators by healthcare organizations is essentially focused on a patient-centered approach, on increased adoption of EHR and EMR, and cost reduction in medical care. These indicators show that, by adopting HIEs, healthcare organizations can create best practices in healthcare sustainability and, as a consequence, new business models.

In effect, efforts toward national governance (on a region and community basis) are crucial to create viable policies, engage people, and make progress toward implementing HIE in healthcare to improve care.

In fact, to discover who is orchestrating an ecosystem response to the outbreak and organizing the collaboration of all stakeholders to save lives while protecting their health systems ( HIMSS USA, 2020 ).

List of Abbreviations and Acronyms

ADE Adverse Drug Event

BI Business Intelligence

BMITF Business Modeling Innovation Task Force

CAUTI Catheter Associated Urinary Tract Infection

CDS Clinical Decision Support

CIO Chief Information Officer

CMS Centers for Medicare & Medical Services

CPOE Computerized Provider Order Entry

CRO Contract Research Organization

CT Clinical Trial

ED Emergency Department

EDC Electronic Data Capture

EHR Electronic Health Record

EMR Electronic Medical Record

ERP Enterprise Resource Planning

ESI Emergy Sustainability Index

EHR4CR Electronic Health Record for Clinical Research

HIE Health Information Exchange

HIMSS Health Information and Management Systems Society

HIO Health Information Exchange Organization

HIPPA Health Insurance Portability and Accountability

HIS Health Information Systems

HIT Health Information Technology

HITECH Health Information Technology for Economic and Clinical Health

ICT Information and Communication Technology

IMCTF Ingenious Middleclass Candid Trusting Forward

MA Medical Assistant

NA Not Applicable

NCO National Coordinator’s Office

NHIN Nationwide Health Information Exchange

OA Office Assistant

ONCHIT Office of National Coordinator for Health Information Technology

PSA Probabilistic Sensitivity Analysis

RN Registered Nurse

ROI Return of Investment

Rx X Ray

TBO Total Benefits of Ownership

TCO Total Cost of Ownership

TRO Total Return Ownership

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1] Ammenwerth, E., Schnell-Inderst, P., Machan, C., & Siebert, U. (2008). The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review. Journal of the American Medical Informatics Association, 15, 585-600.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528040/pdf/585.S106750270800100X.main.pdf
https://doi.org/10.1197/jamia.M2667
[2] Ash, J. S., Sittig, D. F., Dykstra, R., Campbell, E., & Guappone, K. (2009). The Unintended Consequences of Computerized Provider Order Entry: Findings from a Mixed Methods Exploration. International Journal of Medical Informatics, 78, S69-S76.
https://www.sciencedirect.com/science/article/pii/S1386505608001330
https://doi.org/10.1016/j.ijmedinf.2008.07.015
[3] Austrian, J. S., Adelman, J. S., Reissman, S. H., Cohen, H. W., & Billett, H. H. (2011). The Impact of the Heparin-Induced Thrombocytopenia (HIT) Computerized Alert on Provider Behaviors and Patient Outcomes. Journal of the American Medical Informatics Association, 18, 783-788.
https://www.ncbi.nlm.nih.gov/pubmed/21712374
https://doi.org/10.1136/amiajnl-2011-000138
[4] Bates, D. W., & Bitton, A. (2010). The Future of Health Information Technology in the Patient-Centered Medical Home. Health Affairs, 29, 614-621.
https://www.ncbi.nlm.nih.gov/pubmed/20368590
https://doi.org/10.1377/hlthaff.2010.0007
[5] Baysari, M. T., Westbrook, J. I., Richardson, K. L., & Day, R. O. (2011). The Influence of Computerized Decision Support on Prescribing during Ward-Rounds: Are the Decision-Makers Targeted? Journal of the American Medical Informatics Association, 18, 754-759.
https://www.ncbi.nlm.nih.gov/pubmed/21676939
https://doi.org/10.1136/amiajnl-2011-000135
[6] Bell, D. S., Strauss, S. G., Belson, D. et al. (2011). A Toolset for E-Prescribing Implementation in Physician Offices. AHRQ Publication No. 11-0102-EF. Rockville, MD: Agency for Healthcare Research and Quality.
http://healthit.ahrq.gov/sites/default/files/docs/page/ToolsetEPrescribingImplementationFINAL. pdf
[7] Blumenthal, D. (2010). Launching HITECH. The New England Journal of Medicine, 362, 382-385.
https://doi.org/10.1056/NEJMp0912825
[8] Brown, M. T., & Ulgiati, S. (1999). Emergy Evaluation of the Biosphere and Natural Capital. Ambio, 28, 486-493.
[9] Chassin, M. R., & Galvin, R. W. (1998). The Urgent Need to Improve Health Care Quality: Institute of Medicine National Roundtable on Health Care Quality. JAMA, 280, 1000-1005.
https://www.ncbi.nlm.nih.gov/pubmed/9749483
https://doi.org/10.1001/jama.280.11.1000
[10] CMS Centers for Medicare & Medcaid Services (2018a). EHR Incentive Programs: Promoting Interoperability Programs. Baltimore.
https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/index
[11] CMS Centers for Medicare & Medcaid Services (2018b). Physician Fee Schedule. Baltimore.
https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeeSched/index
[12] Coiera, E. (2007). Putting the Technical Back into Socio-Technical Systems Research. International Journal of Medical Informatics, 76, S98-S103.
https://www.sciencedirect.com/science/article/pii/S1386505606001481?via%3Dihub
https://doi.org/10.1016/j.ijmedinf.2006.05.026
[13] Corvalan, C. (2014). Desenvolvimento Sustentável e Saúde: Tendências dos indicadores e desigualdades no Brasil (30 p). Organização Pan-americana da Saúde—Opas/oms Bra- sília: Organização Pan-americana da Saúde.
https://www.paho.org/bra/index.php?option=com_docman&view=download&alias=1500-desen volvimento-sustentavel-e-saude-tendencias-dos-indicadores-e-desigualdades-no-brasil-0&category_slug=saude -e-ambiente-707&Itemid=965
[14] Daniels, J. H. (2014). HIMSS: Transforming Health through Information and Technology. In J. L. Liu (Ed.), Healthcare Quality and Hit—International Standards, China Practices (pp. 39-50). Productivity Press.
https://www.researchgate.net/publication/334425633_HIMSS_Transforming_Health_through_ Information_and_Technology
https://doi.org/10.4324/9780429451539-3
[15] Devine, E. B., Hansen, R. N., Wilson-Norton, J. L., Lawless, N. M., Fisk, A. W., Blough, D. K. et al. (2010a). The Impact of Computerized Provider Order Entry on Medication Errors in a Multispecialty Group Practice. Journal of the American Medical Informatics Association, 17, 78-84.
https://academic.oup.com/jamia/article/17/1/78/2909100
https://doi.org/10.1197/jamia.M3285
[16] Devine, E. B., Hollingworth, W., Hansen, R. N., Lawless, N. M., Wilson-Norton, J. L., Martin, D. P. et al. (2010b). Electronic Prescribing at the Point of Care: A Time-Motion Study in the Primary Care Setting. Health Services Research, 45, 152-171.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2813442/
https://doi.org/10.1111/j.1475-6773.2009.01063.x
[17] Dupont, D., Beresniak, A., Sundgren, M., Schmidt, A., Ainsworth, J., Coorevits, P. et al. (2017). Business Analysis for a Sustainable, Multi-Stakeholder Ecosystem for Leveraging the Electronic Health Records for Clinical Research (EHR4CR) Platform in Europe. International Journal of Medical Informatics, 97, 341-352.
https://www.sciencedirect.com/science/article/pii/S1386505616302520?via%3Dihub
https://doi.org/10.1016/j.ijmedinf.2016.11.003
[18] Forrester, S. H., Hepp, Z., Roth, J. A., Wirtz, H. S., & Devine, E. B. (2014). Cost-Effectiveness of a Computerized Provider Order Entry System in Improving Medication Safety Ambulatory Care. Value in Health, 17, 340-349.
https://www.sciencedirect.com/science/article/pii/S1098301514000175?via%3Dihub
https://doi.org/10.1016/j.jval.2014.01.009
[19] Frisse, M. E., & Holmes, R. L. (2007). Estimated Financial Savings Associated with Health Information Exchange and Ambulatory Care Referral. Journal of Biomedical Informatics, 40, S27-S32.
https://www.sciencedirect.com/science/article/pii/S1532046407000883?via%3Dihub
https://doi.org/10.1016/j.jbi.2007.08.004
[20] Garde, S., Hullin, C. M., Chen, R., Schuler, T., Granz, J., Knaup, P., & Hovenga, E. J. (2007). Towards Sustainability of Health Information Systems: How Can We Define, Measure and Achieve It? Studies in Health Technology and Informatics, 129, 1179-1183.
https://www.ncbi.nlm.nih.gov/pubmed/17911901
[21] Google Cloud QI Network (2019). O que é TCO e como calculá-lo?
https://www.qinetwork.com.br/o-que-e-tco-e-como-calcular/
[22] Hannah, K. J. et al. (Ed.) (2009). Introdução à Informática em Enfermagem (3rd ed., 388 p). Porto Alegre: Artmed Editora.
[23] HIMSS Europe (2013). Experiences and Metrics for Calculating Return on Investment.
https://www.himss.eu/sites/himsseu/files/pdfs/HIMSS_EU_WP_Calculating_ROI_EN_V1.1_ REV1.pdf
[24] HIMSS USA (2020). Accelerating Systems’ Digital Transformation: Why Digital Health Must Be the New Standard in a Post-COVID-19 World.
https://www.himsslearn.org/accelerating-health-systems’-digital-transformation-why-digital-health-must-be-new-standard-post
[25] Hincapie, A., & Warholak, T. (2011). The Impact of Health Information Exchange on Health Outcomes. Applied Clinical Informatics, 2, 499-507.
https://www.ncbi.nlm.nih.gov/pubmed/23616891
https://doi.org/10.4338/ACI-2011-05-R-0027
[26] Hook, J. M., Pan, E., Adler-Milstein, J., Bu, D., & Walker, J. (2006). The Value of Healthcare Information Exchange and Interoperability in New York State. In AMIA Annual Symposium Proceedings (Vol. 2006, p. 953). American Medical Informatics Association.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839305/
[27] Institute of Medicine (US) Roundtable on Evidence-Based Medicine, Olsen, L. A. et al. (Eds.) (2007). The Learning Healthcare System: Workshop Summary (pp. 1-374). Washington DC: National Academy of Sciences.
https://www.ncbi.nlm.nih.gov/pubmed/21452449
[28] Institute of Medicine Committee on Quality of Health Care in America (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: National Academy Press.
[29] Intersystems (2015). Creating Sustainable 21st Century Health Systems: eHealth and Health Information Technology (pp. 1-7). Intersystems Corporation: An Intersystems White Paper, Cambridge, MA: National Academics Press.
https://www.intersystems.com/isc-resources/wp-content/uploads/sites/24/2017/01/Creating-Sustainable-21st-Century-Health-Systems.pdf
[30] Johnson, K. (2011). Computerized Provider-Order Entry: Challenges, Achievements, and Opportunities. Journal of the American Medical Informatics Association, 18, 730-731.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198005/
https://doi.org/10.1136/amiajnl-2011-000579
[31] Jussawalla, M. (1989). Information Resources and Corporate Growth. Prometheus, 7, 395-397.
https://doi.org/10.1080/08109028908629097
[32] Kho, A. N., Lemmon, L., Commiskey, M., Wilson, S. J., & McDonald, C. J. (2008). Use of a Regional Health Information Exchange to Detect Crossover of Patients with MRSA between Urban Hospitals. Journal of the American Medical Informatics Association, 15, 212-216.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2274796/pdf/212.S1067502707003489.main.pdf
https://doi.org/10.1197/jamia.M2577
[33] Khurshid, A. M. L. D., & Luce, S. D. (2012). Health Information Exchange: Metrics to Address Quality of Care and Return on Investment. Perspectives in Health Information Management, 9, 1e.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3392952/
[34] Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R., Kimmel, S. E., & Strom, B. L. (2005). Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors. JAMA, 293, 1197-1203.
https://jamanetwork.com/journals/jama/fullarticle/200498
https://doi.org/10.1001/jama.293.10.1197
[35] McGee, J. V., & Prusak, L. (2004). Gerenciamento estratégico da informação: Aumente a competitividade e a eficiência de sua empresa utilizando a informação como uma ferra- menta estratégica (21st ed.). Rio de Janeiro: Elsivier BV.
[36] Modena, T. (2019). A Lei Hitech: A revolução da saúde mundial.
https://egov.ufsc.br/portal/conteudo/lei-hitech-revolu%C3%A7%C3%A3o-da-sa%C3%BAde- mundial
[37] Moore J. (2006). A New Scheme for Data Sharing—Semantic Interoperability Aims to Ease Data Sharing among Disparate Health Systems. Government Health IT.
[38] Nanji, K. C., Rothschild, J. M., Salzberg, C., Keohane, C. A., Zigmont, K., Devita, J. et al. (2011). Errors Associated with Outpatient Computerized Prescribing Systems. Journal of the American Medical Informatics Association, 18, 767-773.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3197998/
https://doi.org/10.1136/amiajnl-2011-000205
[39] ONU Brazil (2020). Meio Ambiente | ONU Brasil: A ONU e o meio ambiente.
https://nacoesunidas.org/acao/meio-ambiente/
[40] Payne, T. H., Bates, D. W., Berner, E. S., Bernstam, E. V., Covvey, H. D., Frisse, M. E. et al. (2013). Healthcare Information Technology and Economics. Journal of the American Medical Informatics Association, 20, 212-217.
https://academic.oup.com/jamia/article/20/2/212/896043
https://doi.org/10.1136/amiajnl-2012-000821
[41] Poon, E. G., Blumenthal, D., Jaggi, T., Honour, M. M., Bates, D. W., & Kaushal, R. (2004). Overcoming Barriers to Adopting and Implementing Computerized Physician Order Entry Systems in US Hospitals. Health Affairs, 23, 184-190.
https://www.healthaffairs.org/doi/full/10.1377/hlthaff.23.4.184?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed
https://doi.org/10.1377/hlthaff.23.4.184
[42] Rahurkar, S., Vest, J. R., & Menachemi, N. (2015). Despite the Spread of Health Information Exchange, There Is Little Evidence of Its Impact on Cost, Use, and Quality of Care. Health Affairs, 34, 477-483.
https://www.ncbi.nlm.nih.gov/pubmed/25732499
https://doi.org/10.1377/hlthaff.2014.0729
[43] Riedmann, D., Jung, M., Hackl, W. O., & Ammenwerth, E. (2011). How to Improve the Delivery of Medication Alerts within Computerized Physician Order Entry Systems: An International Delphi Study. Journal of the American Medical Informatics Association, 18, 760-766.
https://www.ncbi.nlm.nih.gov/pubmed/21697293
https://doi.org/10.1136/amiajnl-2010-000006
[44] Rouse, M., & Holman, T. Health Information Exchange (HIE). https://searchhealthit.techtarget.com/definition/Health-information-exchange-HIE
[45] Scott, G. P., Shah, P., Wyatt, J. C., Makubate, B., & Cross, F. W. (2011). Making Electronic Prescribing Alerts More Effective: Scenario-Based Experimental Study in Junior Doctors. Journal of the American Medical Informatics Association, 18, 789-798.
https://www.ncbi.nlm.nih.gov/pubmed/21836158
https://doi.org/10.1136/amiajnl-2011-000199
[46] Shamliyan, T. A., Duval, S., Du, J., & Kane, R. L. (2008). Just What the Doctor Ordered. Review of the Evidence of the Impact of Computerized Physician Order Entry System on Medication Errors. Health Services Research, 43, 32-53.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2323150/
https://doi.org/10.1111/j.1475-6773.2007.00751.x
[47] Spinola, M., & Pessoa, M. S. P. (1998). Tecnologia da informação. In J. Celso et al. (Ed.), Gestão de Operações: A Engenharia de Produção a Serviço da Modernização da Empresa (3rd ed., pp. 97-105). São Paulo: Edgard Blucher.
[48] US Bureau of Labor Statistics (2019). Occupational Employment Statistics. Washington DC.
https://www.bls.gov/oes/
[49] Walker, J., Pan, E., Johnston, D., Adler-Milstein, J., Bates, D. W., & Middleton, B. (2005). The Value of Health Care Information Exchange and Interoperability: There Is a Business Case to Be Made for Spending Money on A Fully Standardized Nationwide System. Health Affairs, 24, 5-10.
https://www.ncbi.nlm.nih.gov/pubmed/15659453
https://doi.org/10.1377/hlthaff.W5.10
[50] Wang, S. J., Middleton, B., Prosser, L. A., Bardon, C. G., Spurr, C. D., Carchidi, P. J. et al. (2003). A Cost-Benefit Analysis of Electronic Medical Records in Primary Care. The American Journal of Medicine, 114, 397-403.
https://www.ncbi.nlm.nih.gov/pubmed/12714130
[51] Weiner, M., Gress, T., Thiemann, D. R., Jenckes, M., Reel, S. L., Mandell, S. F., & Bass, E. B. (1999). Contrasting Views of Physicians and Nurses about an Inpatient Computer-Based Provider Order-Entry System. Journal of the American Medical Informatics Association, 6, 234-244.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC61363/
https://doi.org/10.1136/jamia.1999.0060234

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.