Performance Evaluation of the Brazilian Health System in 215 Discourses: Comparative Simplifications and Control Weaknesses ()
1. Introduction
1.1. Control, Contemporaneity, and Managerialism
The role of civil society in controlling contemporary public administration has been reworked through subtle control movements that combine consent and diffusion. This has provided depoliticization and democratic deficit for organized civil society in terms of their ability to arbitrate on the performance of public policies (Paes de Paula, 2005). In this sense, the collective opinion of these stakeholders, an expression of consensus and/or dissent about the institutions conveyed by different media (Rothberg, 2008), has become weakened.
Contemporarily, one of the essential capacities of the subject in civil society is the way in which information is handled. The excess of information that turns into noise (Dias, 2013) and the instantaneousness of information as opposed to its historical process (Castiel & Vasconcellos-Silva, 2006) has been an important factor of confusion. Allied to this, the absence of intersubjectivity in the formulation of individual and collective opinions tends to generate social representations that spread ideas that are less and less exposed to counter-arguments (Dias, 2013).
Thus, the subject presents himself/herself as individualized and hostage to the acute complexification of modern social structures (Beck, 2010), which makes decoding the social scenario somewhat unattractive (Berger, 2007). In this scenario, civil society has been dying from insufficient politicization to exercise accountability when challenging the State in matters of public interest (Arato, 2002). This aspect is especially strategic when it comes to the evaluation of the performance of a public policy by civil society.
Under the aegis of managerialism, the construction of performance evaluation models for public policies has become the most defended instrument in public administrations (Smith, Mossialos, Papanicolas, & Leatherman, 2012) as in the case of Brazil (Carnut & Narvai, 2016). However, the lack of dialogue between this tool with stakeholders of the civil society in general makes them managerial instruments that tend not to capture the measurement objects in daily policy and in addition to making external control difficult.
One aspect of the performance evaluation debate concerns the accuracy of performance and the object under evaluation. What is noticed in the world literature is the profusion of concepts about what performance means, preventing its comparative use. Furthermore, homogenizing what “health systems” are not an easy task because, in reality, these systems present divergences between authors and disciplines in addition to having been defined differently over time (Hoffman et al., 2012).
In addition, when comparing health systems, it is necessary to clearly establish the difference between “health systems” and “health care and service systems” (Lobato & Giovanella, 2012). The former are much more comprehensive and are the result of a complex interaction of a set of factors and actions of different social systems. They include the set of interventions that target specific, social or health problems; cover the entire range of interventions, from preventive services to palliative services. They comprise the major functions of public health (Conill, 2006), but health systems do not have direct responsibilities, or governance, over the set of social conditions that interfere with it.
In fact, when one seeks to comparatively analyze health systems, the focus tends to fall on “service and care systems” (Mendes, 2011). Health systems studies are very close to “policy analysis” and “health service studies”. This is because they use close disciplinary resources, such as epidemiology, economics, social sciences and administration. This interaction is positive and should be encouraged, however, there is a need to distinguish them for measurement purposes (Lévesque & Bergeron, 2003).
1.2. Managerialist Control in Brazil: The Case of the Performance Index of Unified Health System (IDSUS)
When it comes to the Unified Health System (SUS) in Brazil, the development of managerial instruments that seek “more objective” decision-making to put into practice the principles of universality, comprehensiveness and equity has been defended (Tanaka, Tamaki, & Felisberto, 2012), especially in the historical context in which it seeks to establish a “new public administration” focused on results (Garces & Silveira, 2002), as it has been titled in the literature as “the new public management” (Exworthy, 2010).
Thus, the experience of performance evaluation of the SUS, with national coverage, gained prominence with the creation of the Performance Index of the Unified Health System (IDSUS) in 2012. IDSUS analyzes performance based on the crossing of simple and compound indicators applied to federative entities: municipalities (whether or not organized in health regions), states and the Union. It focuses as “performance” the requirements of “efficiency” and “access to health services” at its various levels and works with some elements of social determination of health (Brazil, 2011a).
Nevertheless, the value that can be attributed to this evaluative initiative originating from the Ministry of Health, the crucial fact regarding this evaluation is that the essence of what is taken for “performance” is questionable, especially if the doubt is not restricted to methods and techniques used, but is addressed, epistemologically, to the object (what) these evaluative processes, with the instruments employed, are proposing to measure.
It is worth emphasizing that the basic characteristics of the performance evaluation processes are not explained in the formulations of these evaluation processes, and it appears that their meaning is tacit, requiring no explanation. However, the sense of performance itself is not sufficiently clarified as an expression of the indicators used in both processes. The main consequence of this lack of definition is the possibility of “biases”.
Another aspect that adds to this type of evaluation as a public policy is the polysemy of the word “result”. The trend in “management by results” is to understand the “processes” of service production as “results”, not the real results that are expected from a health system that is about improving the health conditions of a population.
A repeated argument for the relentless search for results (and consequently for performance evaluation) comes from the principle of citizen participation in the (external) control of the State through the evaluation of the performance of public administration (Ceneviva & Farah, 2007). Also called vertical accountability (O’Donnell, 1998), the incorporation of this type of control of accountability of public administration for performance appears as a promise to replace a bureaucratic model.
In this situation, the interface between these subjects and political institutions, which some authors call “citizenship of subjects in networks”, is a growing reality with emphasis on predominantly opinionated participation via Twitter, Facebook etc. This type of participation has increased the public visibility of policies, but with no guarantee of continued mobilizing engagement (Scherer-Warren, 2015). Thus, it remains to be seen how the form and content of this control via the network is constituted in the performance evaluation case of the Brazilian health system.
Therefore, this research was motivated by the dissemination of IDSUS in Brazil in 2012 as an opportunity to understand how the collective opinion on the subject behaved. Therefore, it was taken as a research problem: how the collective opinion about IDSUS was constituted quantitatively and qualitatively to carry out a control of the evaluation policy of the SUS along the lines of public control suggested by managerialism? Due to this question, the objective of this study was to analyze the form and content of the collective opinion of stakeholders on the Performance Index of the Unified Health System.
2. Health System Performance Evaluations: A Literature Review
2.1. Health Sector: Measuring Results and Evaluating Performance to Control
The results achieved by organizations are increasingly object to measurement. Therefore, capturing what has been called “performance” in these organizations has become the goal of those who believe in these mechanisms to assess the fulfillment of their missions. So, these organizations have resorted to performance measurement systems that try to characterize their actions in the contexts in which they operate (Motta, 2007).
For theorists in administration, an organizational performance measurement system, as it aligns organizational strategies (Kaplan & Norton, 1997), seems to be a growing trend in public administration (Horneaux Júnior, Ruiz, & Corrêa, 2005). Therefore, the formulation of indicators and their power to describe what it is essential for performance should be a task to be carried out with great caution.
Lohman, Fortuin and Wouters (2004) report that a performance indicator is a variable that quantitatively demonstrates the effectiveness or efficiency or both, of part or the whole of a process or system, according to a norm or objective. Also, there are times when the indicator is called a “performance metric”. An “indicator” or “metric” allows you to monitor the performance of actions aimed at materializing the strategies, thus creating a “link” between what is planned and what has been done, a relationship that involves strategy, action, and measurement. From the perspective of measurement system, a Performance Measurement System (or “Performance Measurement System”—PMS) can be understood as a “set” of these metrics when applied to the quantification of efficiency and effectiveness of organizational actions (Neely, Gregory, & Platts, 2005).
Performance measurement has some advantages. According to Atkinson, Waterhouse and Wells (1997), with performance measurement organizations we can monitor the implementation of their plans and determine when they are not being successful and how to improve them. Bititci, Turner and Begemann (2000) state that performance measures need to be aligned with the organization’s strategic priorities, under the prism of strategic control. In general, the evolution of Performance Measurement Systems has been outlined in order to follow the way in which organizations that have concentrated their processes in the form of chains or networks (Busanelo, 2011) operate.
However, there are some disadvantages associated with performance measures. A latent concern among scholars of these measures is related to the “definition of coherent indicators for this practice”. It reflects on the dimensions to be considered in the evaluation process (Busanelo, 2011). Another aspect, very common in companies focused on the market, is the abuse in the use of financial indicators as measures that summarize the performance of an organization. Because of this aspect, non-financial measures are increasingly being used to provide performance information of a non-monetary nature, such as market share, customer satisfaction, innovation and development of new products, and employee turnover (Verbeeten & Boons, 2009).
In the field of services, Fitzgerald et al. (1991) emphasize the existence of two basic types of performance measures in organizations: 1) measures that report results (competitiveness, financial performance); and 2) measures that focus on the determinants of results (quality, flexibility, use of resources and innovation). The emphasis on considering financial and non-financial indicators as defining factors of measured performance is evident. Health, understood economically as a service (Meirelles, 2006), would also consider indicators related to the measurement of the determinants of its results.
Neely (1999) presents another list of principles for the development of indicators to be used in Performance Measurement Systems: 1) the metrics or indicators should be directly related to the company’s production strategies; 2) non-financial indicators must be adopted; 3) it should be recognized that indicators vary depending on locations as one measure may not be suitable for all departments or locations; 4) it should be recognized that indicators change as well as circumstances; 5) indicators must be simple and easy to use; 6) indicators must provide quick feedback; and, 7) indicators should be designed to encourage continuous improvement and not just as monitors. Thus, it is possible to apply mathematical optimization techniques to maximize or minimize a function previously defined as Performance Indices (IP), to find an “optimal solution” to the problem.
Indeed, the measurement of some processes just only justified itself if this process can be object of an evaluation. Evaluation is an essential management support tool for its ability to improve the quality of decision-making. An obstacle to a broader use of evaluation in decision-making in health services is that its implementation requires resources and time, which makes it difficult to use it for problems that require immediate solutions. In these situations, which are very frequent when it comes to the health of groups and populations, only the existence of accumulated knowledge, resulting from past or previously planned assessments, can contribute to this decision-making (Tanaka, Tamaki, & Felisberto, 2012).
The word “evaluation”, in its comprehensive sense, means “attribute value to something” (Aguilar & Ander-Egg, 1994). For Arnold (1971) evaluation “is the planned and systematic feedback of information necessary to guide future action”. Evaluation is a way to judge the performance of programs, and it is necessary to define measures to measure the results obtained. Thus, in this field, “outcome measures” are called “evaluation criteria” (Oliveira, Silva, & Bruni, 2012). Without evaluating it is not possible to exercise the administrative function of controlling.
2.2. Health System as an Object of Performance Evaluation in the Exercise of the Control Function
As seen, one aspect of the debate on performance evaluation refers to the precise definition of the object of the evaluation so that it is possible to choose the best indicators. The diversity of conceptualizations highlights the wide variety of ways in which “evaluating performance” is perceived by different authors and models. This also occurs with the perception of “what are health systems?” presents among divergent authors and disciplines (Hoffman et al., 2012).
For Roemer (1991), “health systems” are service structures based on a combination of resources, organization, financing, and management that culminates in the provision of health services to the population. In Mendes (2002), these structures are social responses deliberately organized to respond to the needs, demands and representations [“of health”] of populations, in a society and in a period. For Lobato and Giovanella (2012), it is the set of political, economic, and institutional relationships responsible for conducting the processes related to the health of a population, which set is materialized in organizations, rules and services that aim to achieve the results consistent with a conception of health prevailing in this society.
As for the purpose, according to Starfield (2002), every health system has two main objectives that must be pursued. The first corresponds to the optimization of the population’s health through the application of the most advanced knowledge about the causes of illnesses. The second aim is: to minimize inequality between population subgroups, so that certain groups are not at systematic disadvantages in relation to access to health services.
For the World Health Organization, health systems aim is: “all activities which it primarily intended to: promote, restore or maintain health”. These activities can be grouped into six categories: 1) provision of services; 2) health professionals; 3) health information systems; 4) medical supplies, vaccines and technologies; 5) health financing system; and 6) leadership and governance (WHO, 2007). The growing contribution of the comparative analysis method to the study of health systems coincides with the growth and expansion of these services, since government action in social and health policies has increased considerably throughout the 20th century (Conill, 2006).
For Conill (2006), health systems can be divided into three large groups: 1) National Health Systems (with the perspective of becoming universal health systems, financed from fiscal resources, such as the British system); 2) Public Health Insurance (under state control, but financed by the policyholders themselves, even though they receive some public funding, such as the German system); and, 3) Permissive Business Systems (financed directly by the beneficiaries and under state regulation only, such as in the United States). An ideal health system should basically perform three major functions: 1) regulation; 2) financing (universal or segmented); and 3) the provision of services (integrated or fragmented services, with state monopoly, public contracts, managed competition, or free market).
As for its components, Kleczkowski, Roemer and Van der Werff (1984) state that health systems are constituted by the following components: 1) care model; 2) financing; 3) infrastructure; 4) organization; and management. In a broader proposition, Lobato and Giovanella (2012) identify that health systems depend on “components” that interact through their “functions”. For these authors, among the components of health systems are 1) coverage and benefits catalog; 2) resources (human, financial, service network, technology and knowledge, inputs); and 3) organizations. As for the functions, these would be mainly four: 1) financing; 2) the provision of services; 3) management; and 4) regulation.
For these reasons, when comparing systems, it is necessary to clearly differentiate between “health systems” and “health service systems” (Mendes, 2011). The first ones are much more comprehensive and refer to health in a broader sense, which is the result of a complex interaction of a set of factors and actions from different social systems. Health systems include the set of interventions that target specific social or health problems; they cover the entire range of interventions, from preventive services to palliative services, including diagnostic and curative services. They comprise the major functions of public health (surveillance, protection and health promotion, disease prevention, assessment of the health services system, development of public health skills) (Lévesque & Bergeron, 2003), but health systems do not have direct responsibilities, or governability, on the set of social, economic, cultural, demographic conditions that affect people’s ability to live well and for a long time.
In fact, when people look to a comparatively analysis of health systems, the focus tends to fall on “health service systems” (Conill, 2006). Health systems studies are very close to “policy analysis” and “health services studies”. This is because these studies use close disciplinary resources such as epidemiology, economics, social and human sciences, and administration. This interaction is positive and should be encouraged because it helps a lot to know about health systems. However, there is a need to distinguish them. The “analysis of health policies” prioritize the political and institutional relationships between actors in the health area, while the “studies of health services” prioritize the knowledge of the effects of the actions of institutions providing health services on the living and health conditions of populations (Lobato & Giovanella, 2012).
These aspects correspond to the conditionalities of the studies, analysis and evaluations that take “health systems” as an object. To research in this area, it is required a multidisciplinary and comprehensive look as an elementary prerequisite to understand this field. Also, it requires consideration of factors related to governance, financing and organization of public health service delivery, implementation considerations to reform or strengthen this form of service organization and, more broadly, the economic context, legal, political, and social in which these services are negotiated and operated.
The purpose of health systems research is to improve the understanding the performance of health systems. Health systems research includes all “health services” research, most “health policy” research and some “clinical-epidemiological” research but does not include biomedical research (Hoffman et al., 2012). Therefore, it is suggested that studies on health systems have as a frontier the “analysis of the dynamics of one or more of its components.” In other words, studies must be comprehensive to the point of considering the social, political, and economic aspects that interfere with health systems, but always focusing on, at least, one of its components and its dynamics, or compared to other systems (Lobato & Giovanella, 2012).
Based on this complexity involved in the study of health systems performance that we justify why the evaluation of performance in this field is something so challenging (Carnut & Narvai, 2020). For this reason, controlling the performance of health systems by the population becomes an essential issue (Carnut, 2020). In this sense, the importance of carrying out empirical studies on the form and content of public control carried out by stakeholders is the way to move forward in this field.
3. Methodological Path
The speeches of subjects considered as stakeholders in the performance evaluation process of the SUS (managers, users, health workers, politicians, public health scientists, public health entities and civilians) were analyzed. For that, qualitative and quantitative methodologies were used, from the point of view of approaching the research problem (Turato, 2003). The speeches, taken as secondary data, were captured from the research subjects in virtual environments that made them publicly available (Kotliar, 2015), thus, this study is based on the use of public information available on the internet by online communities. We share the perspective of Germain, Harris, Mackay and Maxwell (2018) when they state that this information is particularly useful for research involving populations whose face-to-face access would be impossible.
Thus, these speeches about the health system evaluation process, written and available on the internet, were systematized using the Collective Subject Discourse (DSC) method (Lefèvre et al., 2002). The DSC is a proposal for organization and tabulation of qualitative data of a verbal nature, obtained from testimonies, which is based on the Theory of Social Representation and its sociological assumptions (Moscovici, 2003).
As for the object (IDSUS), it is worth noting that this instrument/tool is the materialization of performance evaluation within the scope of SUS. According to official documentation, IDSUS is a performance evaluation methodology, developed by a multiprofessional team from the Ministry of Health, whose indicators are basically divided into two groups (access and effectiveness) and thus summarize the performance of SUS through a summary measure (Brazil, 2011a).
Regarding the data collection technique, it was carried out from a “guiding question” that operated methodologically as if the subjects responded to the interviewer, as recommended in the semi-structured interview technique (Flick, 2009). Thus, the question served as a “filter” to identify the speeches considered pertinent to the guiding question, formulated in the following terms: “What do you think about the performance evaluation of SUS, carried out with the use of IDSUS?”. In other words, this question helped to gather stakeholder impressions freely available on the internet about the case.
The collection of the speeches was made in virtual environments related to the theme. Two strategies were used to map these environments: 1) the identification of public and wide-reaching social networks that were considered discussion forums on the topic (facebook, twitter and orkut); and 2) the identification of electronic sites on the world wide web (internet), such as portals, blogs and webblogs that deal with the subject, using an information-seeking program (software), compatible with the purpose of the investigation. The methodological option for that program fell on Google. To recognize these virtual environments, the keyword “IDSUS” was used as a search criterion.
Thereafter, the speeches related to IDSUS were collected, which were in line with the guiding question, that is, they fit as a possible answer to this question. The collection of these speeches had a time limit of 120 days from the disclosure of IDSUS, which took place in March 2012. After capturing the speeches, the material was organized and systematized using the software called “QualiquantiSoft” with the purpose of organizing and systematizing the contents of these speeches. This research does not involve human participants or animals (only public data).
4. Results
4.1. The Quanti-Qualitative Analysis of the Central Ideas
A total of 215 speeches that gave opinions on IDSUS were identified on the internet during the period analyzed. In these speeches, 217 central ideas were identified. Of these, the most frequent were those related to “undue comparisons”, totaling 23.04% of the total (50). Second, the speeches whose central ideas referred to a “good evaluation” of the SUS, totaling 13.36% (29), were observed. The third most common central idea corresponded to “criticisms of the method/index”, totaling 11.52% of the total (25) (Table 1).
Within the central idea that refers to “undue comparisons”, it was possible to verify a diversity of arguments that describe these comparisons in a coarser way to the most refined ones. Certainly, all are invalid from the point of view of the method used to assess performance, each with its specific reason. Furthermore, the arguments of the central idea that refers to “good reviews” are less diverse. The central idea that refers to “criticism of the method/index” presents a greater diversity of arguments as set out in Chart 1.
![]()
Table 1. Proportion of Central Ideas (CI) identified in the speeches about what the social actors think/report after the announcement of IDSUS, in 2012.
*Refers to the order of frequency of the categories (from most frequent to least frequent). Source: elaborated by the authors.
![]()
Chart 1. Three most frequent central ideas (“undue comparisons”, “good performance evaluation of the SUS” and “criticism of the method/index”) and their key expressions in the subjects’ speeches, 2015.
Source: elaborated by the authors.
4.2. The Quanti-Qualitative Analysis of the Anchorages
Regarding anchorages, those related to the justification “on the evaluation model” were identified as the most frequent, totaling 18.42% of the total (21). In the second position of frequency, the anchorages related to the “correct application of investment resources” totaled 17.54% (20) and, in third place were those that justified that it is “the daily service that best explains the reality” and totaled 11.4% (13) (Table 2).
Within the anchorage that refers to the “evaluative method”, it was possible to verify a diversity of arguments that describe the main flaws committed in the elaboration of the performance evaluation method until some considerations regarding the initiative of trying to evaluate the performance of the health systems. As for the anchorage that refers to the “correct application of investment resources” this presents arguments that move from the direct relationship between investment and performance to the use of planning as a determinant for achieving good performances. Finally, the anchorage that refers to the “day-to-day services better explains reality” presents a lesser argumentative variety according to the key expressions of these anchorages shown in Chart 2.
![]()
Table 2. Proportion of Anchorages (Anc.) identified in the speeches about what social actors think/report after the announcement of IDSUS, in 2015.
*Refers to the order of frequency of the categories (from most frequent to least frequent). Source: elaborated by the authors.
![]()
Chart 2. Three most frequent anchorages (“on the evaluative method”, “on the correct application of investment resources” and “day-to-day services better explain reality”) and their key expressions of the subjects” discourses, 2015.
Source: elaborated by the authors.
5. Discussion
Despite the enormous relevance of the initiative in evaluating the performance of the Unified Health System, starting in 2012 with the use of IDSUS, the Brazilian scientific literature does not register the occurrence of any critical study on the topic, except those that reproduce the status quo (Forte & Nobre, 2014; Cielo, Schimidt, & Wenningkamp, 2015; Braz et al., 2013). Hence, therefore, the originality and contribution of this article. The results of this study reveal, however, that despite the aforementioned relevance, the use of the IDSUS instrument did not manage to overcome the difficulties inherent to evaluative operations of this type and scope, all marked by the formation of an ideological construct regarding the evaluation of health systems performance. According to the findings of this study, the theme inhabits public opinion only ideologically, both in the sense of “false ideas” and in the sense of “political project” (Sell, 2006).
5.1. The Central Ideas about the Performance Evaluation of the SUS
The undue comparison was the most frequent idea in the social representation of the subjects as a false idea that public opinion presents on the subject. This denotes the growing demand for specialized knowledge (Nogueira, 2011) by the democratic subject (Touraine, 1998) when inserted in these struggles in the political arena (Filc, 2014). In the case of evaluating the performance of a policy, the idea of measurement (derived from performance) (Souza et al., 2009) and the idea of parameter/criterion (derived from evaluation) seems to naturalize the pre-notion of comparability as intrinsic to the process. Thus, the subjects are not interested in knowing if the objects are subject to comparison (Hoffman et al., 2012) (in the case of SUS due to several factors such as: population size, geographic scales, health regions, federal units and homogeneous groups), nor whether their methods of comparison are valid (Klazinga, 2010) (in the case of using the national average as a cutoff point). This is evident when several forms of (undue) comparisons between health systems (Conill, 2006) are formulated by the speeches of the actors interested in the discussion. In this case, contradicting the main ideologue of the Brazilian managerialism, Bresser-Pereira (2009), it can be said that civil society has a deficit of content that hegemonizes the public debate and testifies against the idea of the effectiveness of democratic control over public policy.
Based on this ideological discourse, the fragile expression of the idea of criticism elaborated by civil society, fosters the feedback of ideology, now, in the sense of political project. In the findings, the performance of some health service systems was well evaluated by IDSUS. Thus, the idea of “good evaluation” spreads out through social media and starts to inhabit the social representation of interested subjects functioning, according to Ramos (2008), as a mechanism of ideological control in the face of the “crisis of reason”. In this mechanism, it can be observed that the ideology of the managerial state is established and gains supporters. As Di Pietro (2002) warns, the management ideology of the new public management changes the way of conceiving the State and public administration regarding the vision of services and their functions. In this sense, this study demonstrated that, instead of the collective opinion worrying about the service systems that needed to improve, in fact what happened was a dichotomy of this opinion. Thus, the debate on performance was restricted between the “systems” that are “good” (and therefore deserve to be rewarded) and the “bad” (which deserve to suffer scarcity), hiding the discussion about the way in which evaluation was made. This thinking translates the notion of distributive justice into “merit” (Bobbio, 1987) on which this political ideology is based, revealing the existence of an architecture of manipulation that induces this type of discourse in the subjects (Dias, 2013). Believing that the health service systems that were well evaluated, in fact are in their concreteness, in addition to being an expression of the fragility of the criticism of organized civil society (Silva, 2006) about the data released, highlights the reproduction of managerial ideology through the way in which the evaluation of a public policy is made (Souza, 2007). In this way, this evaluation crystallizes as the “correct way” of evaluating itself. For the subjects, the good results do not cause strangeness, but relief; after all, everyone has a latent feeling that managers “are doing their duty”, or that services are “on the right track”.
Even under all the hegemonic force of the social representation that ratifies IDSUS, a less frequent representation was that related to criticisms of the method and the index as a whole. Aimed at the ideological forces of manipulating the discourse and maintaining the idea of the managerial political project (Carnut & Narvai, 2016), the set of criticisms can be understood as a counter-hegemony that tries to requalify the debate and identify the superficialities and distortions hegemonically produced.
Pay attention to the fact that a first criticism was made to the fact that the developed methodology of the index did not propose to measure the barriers to access/use. Although the index operates with the concept of “effective access”, Travassos and Martins (2004) point to the complexity of the binomial “access/use” and the need to define which access domain one seeks to capture, something that is not sufficiently clear in the index methodology, opening room for criticism. Still, according to these arguments, it is also unclear whether the object of the assessment would be the Brazilian health system or just the public and private hired segment. As provided by several studies in the health systems area (Wagstaff, 2009; Thompson, Osborn, Squires, & Jun, 2012; Basu, Andrews, Kishore, Panjabi, & Stuckler, 2012), although they lack a well-defined frontier, it is well known that in studies on health systems it is essential to involve all types of legal nature of service provision. When such data are subtracted, a deep understanding of the dynamics of its financing, which could have a considerable explanatory power on performance, remains unfeasible. Therefore, it seems to make sense that IDSUS is not composed of data on the system’s financing. The absence of investment data from different spheres of management, in the formulas used to weight the results, could incite a criticism of the chronic underfunding of public health in Brazil by the federal government, a characteristic reported by successive studies (Marques & Mendes, 2007; Mendes & Carnut, 2018) which overloads the need for local investment and transfers the responsibility to this manager over the results of other federal entities.
Traditionally, there is a fetish in associating performance only with financial aspects, which has been criticized even in the most traditional organizational proposals on performance evaluation models (Verbeeten & Boons, 2009). Even so, these proposals do not dispense with the “economic criterion”, this being only displaced from centrality, to constitute yet another variable that is part of the causal chain of performance. Therefore, economic data should be used to reinforce/denounce the financing logic, according to the type of system it is in (in the case of Brazil, a National Health System) (Conill, 2006). IDSUS, on the contrary, presents its mathematical projection that implies financing and whose calculation of the index is elaborated by the logic of “benchmarking”.
Another criticism was in the set of data/indicators used. It is an inherent characteristic of performance evaluation models, the work with data related to the past, contributing negatively to the idea of “anticipation” so necessary to decision making. Thus, IDSUS, as a model that intends to evaluate performance, also suffers from this characteristic. In addition, these indicators, presented in a fragmented way, corroborate the feeling that the data emerge from what is available without a reflection on what data would actually be needed (Champagne & Contrandiopoulos, 2005), or that the data is displaced from context in which they operate (Frenk, 2010). For this reason, as identified in the speeches of the subjects, the evaluation ends up not helping much in the construction and updating of information for SUS, contributing to generate a note, summary number of the evaluation process, devoid of logical meaning. This is perhaps the most essential criticism of IDSUS, despite the performance indexes of health systems and which gave rise to the whole debate on the performance of the Brazilian system, at the national level and when referring to local systems.
In historical terms, the 2000 World Health Report was responsible for “triggering” the debate on the development of an index capable of capturing the performance of health systems. Right after this announcement, the theme “health systems performance” became more of a chapter than what is currently being studied as Global Health (Frenk, 2010). Comparing health systems around the world, so diverse, inserted in different socio-historical contexts, in economic situations so disparate and immersed under the phenomenon of globalization is a herculean task to be submitted to a synthesis number. In this sense, the fragility of measuring the index developed inevitably compromises its comparative power (admitting this possibility, only for analytical purposes) (Szwarcwald & Viacava, 2008) creating serious injustices especially with those countries that have health systems based on the attempt of universalization (Pereira & Stein, 2010).
5.2. The Anchorages about the Performance Evaluation of the SUS
In order to support these three most frequent central ideas in the public opinion of the interested subjects, the most frequent anchoring, in contradiction to the central ideas, focused on the justifications for supporting the criticism of the evaluation method.
For the subjects, a first issue is multi-dimensionality. It is valid to say that health is multidimensional because, ontologically it requires an expanded look and needs to capture all the factors involved in the essence of its production (Meirelles, 2006). Therefore, each dimension requires specific indicators and must be evaluated by matrix support, as well as the evolution of each of the variables over time. The way in which the SUS performance assessment entered the agenda, without financial data and above all at a time when the approval of the Constitutional Amendment 29 (moment lived at the time)1, can be interpreted as if the problems with SUS were merely managerial.
The logic of the territorial design adopted to apply IDSUS was subjected to criticism. After the promulgation of decree 7.508/2011 that made the organization of health systems official by health care networks in regional regions (Brazil, 2011b), the health territories approach from the legal-administrative division which has its organizational basis in the municipality has become, at least partially, incoherent, since it lacks health significance. Unless, exceptionally, when referred to those municipalities whose territories present the integrality of care for a specific care network.
Some speeches emphasized the importance of IDSUS as a way of encouraging the development of a supposed “evaluative culture” in the management of health services, a statement corroborated in the literature on the topic (Tanaka, 2006). However, it is worth considering the meaning, including ethical, of making inadequate, if not innocuous, assessments that generate an unnecessary symbolic “debt” to civil society (Demo, 2002). The evaluative method, as put forward by IDSUS, has so many deficiencies that they don’t even result in the possibility of asserting, with any degree of consistency, whether the system is “good” or “bad”. With the aggravating factor, as can be deduced from the DSC analyzed in this article, that the results of the performance evaluation induce a negative competition between municipal managers. One does not want, with this argument, to deny the availability of an index as a way of gaining accountability, as long as its conception is coherent both in form and in content. For the creation of a solidary and methodologically plural evaluative culture, it would be necessary to incorporate the views of other spheres of management and other social actors involved in the production.
Ranking also appears as a problem. For interested subjects, ranking is only meaningful when using synthetic indicators, which would not apply to health systems due to its inherent complexity, as mentioned in the central idea of the DSC regarding the index. Another argument concerns the way in which the indicators are aggregated. When equal numbers are aggregated in different ways they result in different final numbers that change the positions in the ranking (Valente, 2002). Thus, the subjects pondered that IDSUS is an evaluation method that attends more to the media and its “ranking hunger” and, also, to a political ideology of “importing tools”. As already identified by Marcelino (1988), it is typical of the Latin American countries to try to solve a public problem by mechanically extracting managerial solutions from the private sector. In this sense IDSUS, as a managerial tool, lends itself to injustices with the dissemination of results in the form of summary numbers for the media and public opinion. As the results indicate, from a political point of view, public opinion manifests itself in a plural-elitist way (Pio & Porto, 1998) where the plurality of expressions and multiple affiliations punched the debate and a group of “experts” seems to take the power of argumentation for technical superiority.
For the subjects of this study, the political misuse of the results gives rise to a superficial debate, since the appropriate analysis of public policies (and their results) is a difficult job when the media are limited to reproducing numbers, without even explain its meanings to the public (Miguel, 2002). This occurs especially because the attention of public opinion focuses on the comparison of results (in most cases, those “actors” who want faster access to “information”) rather than promoting further reflection on the complexity of seeking to translate a set of public policies using only a single synthesis number (Marques & Faria, 2013). It is practically impossible to do this without creating distortions and injustices.
Meritocracy as an ideology is evident, as stated in the classic study of Brazilian reality described by Barbosa (1996), regarding the criticism of the link of financial transfers according to performance. In the subjects’ discourse, it is possible to detect that ministerial norms are generally linked to the transfer of resources. However, the coupling of IDSUS as a new evaluation project in the midst of the reform of the State does not break the old performance evaluation projects of the traditional Brazilian administrative culture (Barbosa, 1996). On the contrary, evaluation remains as a way of punishing alleged disabilities rather than improving them (Andrews & Bariani, 2010). The transfer of resources, according to this rationality is, therefore, nothing more than a way to reward the most developed. In a systemic logic, these transfers can accentuate regional disparities, further aggravating situations of wicked inequalities. Thus, an instrument that should be useful in assessing the implementation of public policies aimed at reducing asymmetries starts to operate, the instrument itself, as a vector that amplifies them.
In this inhospitable scenario, the subjects also anchor their criticism in the fundamental question: what are the objectives underlying the performance evaluation? According to the subjects, first of all it is important to know what the performance is, what is being evaluated and its way of evaluating. One way of evaluating collaboratively would be one that would allow identifying the weakest points to make interventions. This format (known as “formative evaluation”) is what should be pursued in States (Ala-Harja & Helgason, 2000) that have national health systems. In the case of IDSUS (understood as part of a public evaluation policy), it appears that this tool is designed more to offer answers to the management group of the State (Faria, 2005) than to capture a concrete reality. Thus, this way of evaluating leads to a way of contracting results in public management, doing little to help managers to identify the problems that affect SUS.
For a part of the interested subjects, what justifies good or bad performance is the management of resources within the scope of public administration. Three main arguments have been identified.
First, in some speeches, it was seen those investments in general (or in specific programs) are responsible for good performance evaluation. In this case, the subjects make a direct association between the application of the municipality’s resources and the transfers from the federal/state governments with the good performance. For Murray and Frenk (2000) there are several factors that explain the performance of systems, in which the economic question is only one of these factors. A typical example is that service provision does not occur even under the combination of adequate investment and imbalance in human/material resources. Another relevant aspect is that, when thinking about investment, most of it is drained into “services” and not “system” and its other functions, which makes this argument inconsistent.
Some speeches point out that the minimum required by law (15%) is responsible for a good performance. It seems valid to say that when investing municipal resources well above the minimum required by law (around 30%) associated with the commitment of health teams, the “service system” tends to have a better performance. The few Brazilian studies (Facchini et al., 2006) that attempt to ascertain this association have little elucidative conclusions about how financing interacts with other variables to explain performance. Several aspects, from socio-political to structural, tend to modify the financing-performance relationship, which, in theory, reinforces the arguments about the methodological difficulty of measuring systems performance (Hoffman et al., 2012).
Planning and evaluation as managerial instruments were also mentioned by the subjects. For them, the act of planning is primarily responsible for good performances. When it comes to the service system, Voekler, Rachik and French (2001) point out that it is possible to measure the use of strategic planning, especially with the use of the balanced scoredcard as an evaluation model, but only when dealing with health organizations. “A priori” it makes sense to say that local planning helps to organize strategic priorities. However, there is no theoretical basis for stating that it would be the same as for evaluating the performance of health systems.
Finally, according to the subjects, the daily life of the services best explains the reality, and in this sense, it is very difficult for any types of indexes or models proposed to capture the complexity inherent in the daily provision of services in the SUS. Among the arguments, it is worth mentioning three.
The first lies in the fact that there is an expectation that the general data, in a certain way, reflects the reality of the services and the professionals see themselves in it. It is, therefore, the classic tension between the general and the particular and vice versa. The mentioned speech pointed out that it is possible to directly associate the particularity found in five services with the worsening performance of a municipal system as a whole. When it comes to phenomena that involve human interaction, the tension between structure and agency will always be put at stake, as it is a fundamentally socio-anthropological dilemma (Simmel, 2006; Garcia, 1993). Health systems, understood as the social processes that they are, would not be outside this area, and this expectation does not contribute much to the effort to evaluate performance. It should not be ruled out, of course, that this discourse is just one more rhetorical figure, reinforcing the ideology that advocates the failure of SUS.
The focus on empirical reality also emerged as an argument, whose core emphasizes the need to understand what levels of analysis on the subject are needed, as well as the articulation of all of them. It would not be possible to state, as a consequence, that the numbers generated by IDSUS reflect the perception that users have of SUS, nor would the index also reflect the effects of supplemental health or the degree of satisfaction of users. Thus, for a more appropriate evaluation, the results of the IDSUS would have to be compared with the data found in each empirical reality.
Finally, in some subjects’ speeches, the belief that a part of the system, service or program could be responsible for the total success of their performance is manifested. However, it is not possible to state, based on data, that a “health system” as a whole can change its conditions as a result of a single dimension indicator that presents a “satisfactory” result. In addition, the indicators from which IDSUS is composed are not intended to capture service strategies. Furthermore, as social systems they are, their positivist understanding (the whole as equal to the sum of the parts) contributes little to a critical view on the subject.
6. Final Remarks
In this article, it is recorded how the opinion of stakeholders was expressed about the IDSUS announcement and the reactions to the first SUS performance evaluation. In order to understand the emergence of this evaluation instrument within the scope of a universal health system, such as the Brazilian one, it was necessary to understand the articulation of IDSUS with initiatives whose objective is to evaluate the performance of health systems. To this end, it was sought to reference the analysis in the world scientific literature on such assessments, noting the fragility of the theory on which the initiatives that, in Brazil and other countries, have this purpose are based. Performance evaluation of health systems is an ideological construct, more than anything else.
Despite this weakness, empirical material was collected whose secondary data were organized, systematized and analyzed with a view to identifying the main arguments presented by subjects interested in the theme of evaluating the performance of SUS. The data search procedures were exhaustive as to the purpose, although it cannot be guaranteed that this methodological perspective is not vulnerable. However, the material obtained made it possible to compose speeches by the collective subject whose central ideas and respective anchorages made it possible to build a wide panel to understand the arguments for and against IDSUS as a possible instrument to evaluate the performance of SUS.
In this scenario, civil society loses, since it cannot count on an effective instrument in the democratic attempt to control the actions of the State on SUS. In the analysis undertaken in this study, it was possible to identify that there is a major pole of civil society that, in general, trivializes and, therefore, disqualifies the evaluation, by simplifying extremely complex processes, making it shallow and fragile. This pole generates a negative ideology about the performance evaluation of SUS that does not match what the system actually is. At other times, this pole accepts the performance evaluation of SUS as a legitimate way of evaluating, thus reproducing the political ideology of the managerial state.
In contrast, a hegemonic, minority pole seeks to re-qualify the debate on SUS performance evaluation, seeking to recover the complexity of comprehensive evaluation processes, with a view not to the production of hierarchies and rankings, but to the production of knowledge that makes it possible to understand situations from results that are not detached from the processes that generate them and that are useful for the permanent reorientation of the system itself and the work that maintains it. For this pole, the IDSUS, as formulated and used, does not correspond to a useful instrument to advance in evaluative terms in a perspective that, surpassing the managerialism in the ways of conducting the public thing, makes it possible to create performance evaluation systems consistent with their own SUS principles and guidelines, notably those that recognize value in community participation, health workers and democratic forms of participatory management.
Acknowledgements
The author(s) received financial support for research from the Coordination for the Improvement of Higher Education Personnel (CAPES), Brazil, which we thank.
NOTES
1The Constitutional Amendment 29 (EC-29) is an amendment to the Brazilian federal constitution that deals with the percentage that each sphere of public administration should invest in health. This amendment went unregulated for 20 years, allowing federal level managers, states and municipalities to choose how much they could invest in the public health system. At the time of the IDSUS announcement, the amendment had been regulated relativizing the obligation of the federal sphere and making states and municipalities responsible, giving the impression that the problem is reduced by the inability of these entities to “manage resources” and not that “there is a lack of resources”.