Indicator Selection for Quality Measurement in Maternal Neonatal and Child Health Services: Application of Random Forest Classifier ()
1. Introduction
Sustainable Development Goals (SDGs) call for a global commitment to ensuring healthy lives and promoting well-being for all ages [1] . This means that all people and communities, everywhere in the world, should have access to high-quality health services. As the concern for quality health care increases, the need for quality measurement also increases [2] . In health care, quality measurement is the process of using data to evaluate the performance of health plans and health care providers against recognized quality standards [3] [4] . In Maternal, Neonatal and Child Health (MNCH) care, quality measurement is especially important if there is a need to improve the health outcome of pregnant women and children [5] [6] . In MNCH, a quality measurement not only provides insights on whether the services provided are of quality enough to make a difference in the health and survival of pregnant women and children, but also what works and what does not to drive improvement in MNCH care.
To date, all around the world, several approaches have been developed and used for quality measurement in MNCH. These approaches include but are not limited to Quality Evaluation (QE) and monitoring frameworks like that developed by Hulton et al. [7] , Renfrew et al. [8] and World Health Organization [9] , Quality measurement models like everlasting Avedis Donabedian healthcare quality model [10] , Demographic Health Survey (DHS) as used in the studies by Dettrick et al. [11] , Souza et al. [12] and Munos et al. [13] and other tailor-made approaches like direct observation, questioners, document reviews and exit interviews [3] [14] [15] . Each quality measurement approach comprises of a special set of quality indicators that are used to describe quality status, track progress and performance or act as a guide to quality measurement according to the purpose and functionality of an approach. Currently, quality indicators used are derived from three health domains, namely structure (referring to the settings where care is delivered), process (referring to whether or not what is known to be good medical care has been provided) and outcome (referring to the impact of care on health) as proposed by Donabedian [10] . The quality indicators derived from the structure, process and outcome have been commonly used for quality measurement and literature reveals rich sets of quality indicators for maternal and child health care [16] . However, it was observed that the available quality indicators lack demonstrated scientific soundness and do not refer to particular phases within the continuum of care [17] . It is further revealed that extra efforts are needed to identify indicators for comprehensive maternal and child health services quality monitoring and measurement [18] .
Countries around the world have been relying on the World Health Organization (WHO) to recommend and provide guidelines and guidance on indicators to be used for quality measurement in MNCH. However, it has been observed that to cater to the specific quality measurement needs additional indicators and sometimes existing indicators are amended to suit quality measurement requirements. It is also noted that apart from the guidelines and quality indicators recommended by WHO, other indicators have been identified and developed by individuals, government agencies and other stakeholders in MNCH. For example, indicators used in measuring the quality of maternal and newborn care in developing countries were identified from the 2012 Indonesian Demographic and Health Survey dataset [11] . Quantitative indicators for labour and delivery care quality measurement were identified from the studies describing indicators for labour and delivery care quality [16] . And in 2014, Kenya took the initiative to validate and identify maternal health indicators that can practically be applied in health facilities and population-based health surveys to accurately evaluate national and global maternal health care in the country [19] .
Nevertheless, despite WHO being the only organization to recommend quality indicators for MNCH quality measurement, the organization does not maintain a finite set of quality indicators that can fit all quality measurement contexts. In the year 2012, the WHO recommended eleven indicators for maternal and child health monitoring and evaluation [20] . Later in 2014, WHO came up with another set of indicators for global measurement and reporting on the quality of care provided for maternal, newborns and children at health facilities that should be used to evaluate Millennium Development Goals (MDGs) [21] . From this observation, it is evident that there is no special and dedicated quality indicator selection process or a set of quality indicators that is universally accepted and readily available for quality measurement in MNCH. Given the importance of quality indicators in quality measurement, this study intends to use the machine learning-based technique to develop a random forest classifier that will enhance the indicator selection process for comprehensive quality indicator selection. Furthermore, the study will use the developed random forest feature selection classifier to identify the most relevant set of quality indicators for MNCH quality measurement.
In machine learning context, feature selection is a process of selecting the features in a dataset that contribute much to the prediction variable or output of interest [22] . This is a common process in machine learning model development where it is used to select important features from the dataset with thousands and hundreds of features [23] . In model development, feature selection informs about the relevant features that have to be included for better model performance, helps in reducing model overfitting by removing redundant data, improves accuracy by removing irrelevant features, and reduces training time by reducing features in the dataset. In this study, feature selection focuses on informing about the relevant indicators that have a great contribution to MNCH quality. A tree-based strategy used by random forest naturally ranks the trees by how well they improve the purity of the node, by doing so a Gini purity (a process of decreasing impurity in all trees) is accomplished. When decreasing impurity, the nodes with the greatest decrease in impurity appear at the beginning of the tree while nodes with the least decrease in impurity appear at the end of the tree. Thus, by pruning the nodes below a particular tree, a new subset of features is created, which is a set of important features. This process is accurate, generalizable and interpretable; it is used in this study to identify important indicators for quality measurement in MNCH.
2. Material and Methods
2.1. Indicator Selection Overview
Indicator (feature) selection using Random Forest (RF) is under the category of embedded methods. Embedded methods combine the qualities of filter and wrapper methods. The algorithms that implement these methods have the built-in feature selection. This method was selected because it is generalizable, easily interpretable and they have high accuracy. To select a feature random forest use four to twelve hundred decision trees, each of the built over a random extraction of the observations from the dataset and a random extraction of the feature. Not every tree sees all the features or all the observations; this guarantees that the trees are de-correlated and therefore less prone to over-fitting. Each tree is also a sequence of yes-no questions based on a single or combination of features. At each node, the tree divides the dataset into two buckets, each of them hosting observations that are more similar among themselves and different from the ones in the other bucket. Therefore, the importance of each feature is derived from how pure each of the buckets is.
2.2. Algorithm
Indicator selection using select from model starts with determining the threshold value to give a boundary between the feature to be selected and the features that will be eliminated, then all features will be sorted by Gini importance score which is the total decrease in node impurity from the smallest to the largest. Furthermore, features with Gini importance score that are below the stated threshold value will be eliminated. Selected feature will be used in the random forest algorithm. The flowchart in Figure 1 shows the indicators inclusion and exclusion process.
2.3. Experiment Set-Up
2.3.1. Dataset Description
The study used a dataset from the District Health Information System 2 (DHIS2). DHIS2 is a centralized data warehouse that collects and stores routine health data from all health facilities in Tanzania [24] . The data consist of five years MNCH routine records from seven districts of Kilimanjaro region—Tanzania. The records contain all quality measurement indicators established by Ministry of Health, Social Welfare, Gender, Elderly and Children as per World Health Organization (WHO) standards, see Figure 2.
2.3.2. Dataset Preprocessing
Data is often incomplete, inconsistent or sometimes missing certain features and is likely to contain many errors if not thoroughly checked. Data preprocessing is
![]()
Figure 1. Quality indicator selection flowchart.
a common preliminary machine learning process that transforms data into a more flexible and effective format for further machine learning processing. The dataset was preprocessed to remove columns and rows that were not required in the experimental stage. The Columns with more than 50% missing values were removed to ensure data reliability. Columns that could reveal data originality were removed as well to ensure the privacy of personal health data.
2.3.3. Identification of Important Indicators
Important indicators were identified by random forest classifier. The classifier algorithm runs by imputing the training set M with certain fixed number of features let’s say N which are constant, and a certain fixed number of trees let’s say T. Randomly the Tx tree is selected from the dataset, then the selected TX tree is used to build decision trees. Decision trees are made several time let’s say S times. At each node, a smallest subset N1 of feature N is made and separates the best feature for each N. The result is the selected N2 feature which has the highest score. The algorithm in Figure 3(a) and a tree diagram in Figure 3(b) illustrate the indicators identification process.
3. Results
The study intended to establish the set of most important indicators for quality measurement in MNCH. Random Forest (RF) classifier was used to accomplish the task. From the experiment, twenty-nine indicators shown in Figure 4 were identified as important indicators for quality measurement in MNCH. The details on the selected quality indicators are shown in Table A1 and Table 1, which provide the details on the most important quality indicators for MNCH quality measurement.
From the experiment, the indicators importance value ranges from 0.00 - 0.25. The indicator with the highest Gini importance value is considered to be more
![]()
Figure 3. (a) Indicator identification algorithm; (b) Indicator identification tree.
![]()
Figure 4. Important indicators with random forest.
![]()
Table 1. Selected most important indicators for quality measurement.
important indicators for quality. To identify the most important indicators, selector object was created to enable the random forest classifier to identify indicators that are more important. A mean threshold of Gini index value was selected as a cutoff point, indicators with more than mean Gini index value was considered to be more important for quality measurement in MNCH. From the previous twenty-nine important indicators, selector classifier yielded eight indicators namely maternal mortality ratio, still-birth rate, delivery at health facility, deliveries assisted by skilled attendants, proportional of breach delivery, normal delivery rate, born before arrival rate and antenatal care visit coverage, see Figure 5
4. Discussion
In developing countries, quality measurement or monitoring and evaluation of MNCH care is often done periodically; commonly once in fifth years or upon request or need from the government or external agencies involved in MNCH. Similarly, the process of identifying and selecting indicators for quality measurement is done periodically based on experts’ perceptions and judgments. This procedure has been commonly applied to obtain rich sets of indicators that are currently used to measure quality and evaluate the impacts of health projects and interventions in MNCH care. In this study, quality indicator selection approach has been established and a new set of indicators for quality measurement has been identified. Twenty-nine indicators have been identified as important for quality measurement and further selection has identified eight indicators as the most important indicators for quality measurement. The selected set of indicators is considered as the best for MNCH quality measurement because of the high Gini importance index from the random forest classifier. The random forest classifier chooses an indicator depending on its contribution to the decision or result of interest. The selected indicators have more contribution to the quality of MNCH care.
Previous studies have also highlighted the contribution of selected indicators in improving the quality of MNCH care. [25] [26] noted that, the increase accessibility and utilization of maternal and child health services alone is not enough
to reduce maternal and children deaths. They acclaimed that, quality maternal and child health services can potentially enhance the reduction of maternal and childhood deaths. This indicates that Antenatal Care (ANC) visit is important indicator in a set for quality measurement. ANC comprise the services delivered to women during pregnancy. [27] noticed that quality ANC has the potential to prevent stillbirth hence reduce neonatal death. Additionally, it has been revealed that the quality health services a woman receives during ANC visits and labour and delivery phase affects her health, the health of her child and the likelihood that she will seek for such health care in the future [28] .
Delivery complications such as obstructed labour are said to be well managed if a woman gives birth at a health facility. Therefore, facility delivery (delivering at the health facility) has an important role in reducing maternal and neonatal mortality [29] . Skilled birth attendants are capable of recognizing and handling various obstetric complications. Births attended by skilled birth attendants have the potential to save both the mother and the newborn in case of any maternal complications. Observation on how selected indicators above contributes to reducing maternal and child births in developing countries. It is worth noting that the selected set of indicators is essential to be included in a set for quality measurement in MNCH.
5. Conclusion and Future Work
Various scholars have revealed the need to identify and validate indicators for comprehensive maternal and child health care quality measurement and monitoring. Furthermore, it is revealed that most of the indicators currently used lack demonstrated scientific soundness and reference to a particular phase in MNCH continuum care. This study aimed at filling this gap by identifying quality measurement indicators. Using a random forest classifier, twenty-nine indicators were identified to be suitable for quality measurement. Further selection using a random forest selector yielded eight indicators, namely maternal mortality ratio, still-birth rate, delivery at the health facility, deliveries assisted by skilled attendants, proportional of breech delivery, normal delivery rate, born before arrival rate and antenatal care visit coverage. The latter is considered to be the most important indicator for quality measurement. This study, therefore, considers a set of eight indicators as the best indicators and recommends them to be used for quality measurement in MNCH. It should be noted that the recommended set of indicators is yet to be used in quality measurement, though individually or in a combination of two to three indicators with other indicators has been used to measure quality. Literature has also shown the contribution of each indicator in improving MNCH quality. Future work will be using the identified indicators to develop the quality measurement model for MNCH services.
Acknowledgements
The authors so gratefully thank all maternal and child health stakeholders who participated in this study and the quality assurance unit staff from the Ministry of Health, Gender, Elderly, Children and Social Welfare, Tanzania.
Funding
African Development Bank (AfDB), through Nelson Mandela African Institution of Science and Technology, Arusha.
Appendix
![]()
Table A1. Selected important indicators for quality measurement in MNCH.