Feedback Control of Medication Delivery Device Using Machine Learning-Based Control Co-Design

Jacob Anthony¹, Jacob Anthony¹, Ashley Dixon¹, Chung Hyun Goh^1*, Matthew Lucci²

¹Department of Mechanical Engineering, University of Texas at Tyler, Tyler, TX, USA.
²The Runatek Corporation, Dallas, TX, USA.

DOI: 10.4236/jsea.2022.157013                

Abstract

Although the opioid crisis is a problem worldwide, recent emerging technology has the potential of curtailing the epidemic. By administering micro-doses of medication as needed, a feedback-driven medicine pump could lessen the highs and lows associated with the formation of an addiction. The focus of this study was to develop a feedback control loop for this pump that optimizes drug concentration in the bloodstream based on set criteria. In the process of optimization of the system, the mathematical model representing the system was analyzed to find an open loop transfer function. Using this function, a PID tuner was applied to set feedback control. Both machine learning (ML) and deep learning (DL) techniques are explored to act as a classifier that aids the pump in administering doses. The setpoint concentration of medication in the patient’s bloodstream was calculated to be 7.55 mg/ml this setpoint was the basis for steady state concentration of the transfer function. When a PID tuner was added to the feedback system, the plot was optimized to satisfy the design criteria of a rise time less than 25-minutes and no more than a 5% overshoot of the setpoint concentration. Naïve Bayesian (NB), Tree and support-vector machines (SVM) classifiers achieved the best classification accuracy of 100%. A DL network was successfully developed to predict patient class. This work is the theoretical basis for developing a feedback-driven medicine pump and an algorithm that can classify patients based on their body’s metabolism that will aid the doctor in formatting the medicine pump so that the patient is receiving the proper amount of medication.

Keywords

Machine Learning, PID, Medication Delivery Device

1. Introduction

Since the spread of the coronavirus in 2020, the number of deaths due to opioid overdose has risen dramatically. In order to combat the increase of this crisis, new technology is in development which will be able to automatically measure the concentration of medication in the user’s bloodstream, determine how much medication the patient needs based on a specific concentration, and administer the medication in micro-doses. This “specific concentration” will be high enough to keep the patient from feeling pain, but low enough to not cause the high associated with addiction. In order to keep the concentration between these levels, the device will utilize a ML algorithm through its feedback control system.

A product similar to this medication delivery device, in that it is made to automatically analyze and administer the amount of medication needed, is a smart insulin pump [1]. With the addition of a sensor, the pump is able to be automated. This combined system takes continuous glucose monitor (CGM) readings and automatically adjusts the patient’s insulin levels as needed. Additionally, it can take precautions to help prevent hyperglycemia and includes adjustment settings for sleep and exercise. A substantial issue with this pump for application in the new medication delivery device, is that its cartridge holds only 3 mL of medication, much less than the 40 mL criteria of the medication delivery device in this study [2].

Another medical device that uses feedback control is an Implantable Cardioverter Defibrillator (ICD) [3]. This device operates by monitoring and regulating continuously the heartbeat of a patient. If the heart rate of the subject reaches a set lower threshold limit, then the ICD will send an electric pulse to the heart to contract it and force a heartbeat, acting similarly to a defibrillator. The real time monitoring of the heart helps a device such as this to be effective in saving a subject’s life, as the shock will be delivered as soon as no heartbeat is detected. Despite the benefits, there are some risks to consider when using an ICD device, including internal bleeding, blood vessel damage, and tearing of heart muscle. Because these are due to the device being implanted in the body, they may be resolved by converting the ICD to an external defibrillator, while still maintaining a discreet form, similar to the medication delivery device in this study. Another benefit to an external device is ease of removal or replacement because there is no need for surgery.

In addition to reviewing similar devices, previous research was beneficial to the progress of the ML research discussed later. This included article discussing the influence of the training and testing process and of data splitting on ML, the prediction of blood glucose levels, a look into DL applications, and more [4] [5] [6] [7] [8].

An important aspect of ML is the training/testing ratio that is incorporated to evaluate data. In an article discussing the effect of training/testing ratios on ML in biomedical engineering, Uçar et al. explained that with a low number of samples in a dataset, or low trust in said dataset, ratios of 90/10 or 80/20 should be selected. With a high number of data, and to test “under the most difficult conditions,” a ratio of 50/50 was said to be best [8]. However, only one of four articles analyzed came to a similar conclusion of 90/10 for prediction capability when compared to 70/30 and 80/20 [7]. The other three decided on the 70/30 ratio as the best option [4] [5] [6].

Gadze et al. investigated a few deep neural networks, including a convolutional neural network (CNN) and long-short term memory (LSTM). This research focused specifically on the networks’ training of a ML model to “detect and mitigate [a distributed denial of service (DDoS)] attack on [Software-Defined Networking (SDN)] controllers” [4]. When compared to some linear-based ML models, the LSTM model showed similar accuracy and proved to be a viable classification model for DDoS detection. One of these linear models, the k-nearest neighbor (KNN) model, showed to have particularly high accuracy (99.4%) in this investigation, proving it to be a promising option when deciding on a classification model [4].

Each of the sources discussed provides helpful information as to how certain variables may affect the ML algorithm, the accuracy of a variety of models, and more, but three of them are not biomedical applications, making the information provided somewhat limited. Essentially, this research may be used as a general guide in this biomedical application, with more emphasis on the investigations which focus on biomedical applications [5] [8].

In general, for a device like this to work, it must have a well-refined feedback control system and ML algorithm, allowing it to properly receive, analyze, and output correct data. Through a feedback control system, the device should be able to receive a measurement of medication concentration in the bloodstream, determine the amount of medication that needs to be administered, and finally, dispense the medication. Although this seems like it may work alone, the system must have a target concentration, and it must be able to accommodate disturbances in the user’s physical activity. This is where the ML algorithm becomes necessary. The algorithm is capable of classifying patients based on their body’s metabolism, this helps the doctor know what setpoint concentration and PID settings need to be used. This ensures that the patient is not getting too much or too little of the medication. The process begins by finding the open loop transfer function, altering it as needed to create a more accurate function, and adding a PID tuner. Between each of these steps, the system’s accuracy, and other values, are analyzed. Overall, the main goal is to implement ML through a feedback control system that will allow the medication delivery device to perform at its full potential and help to ease the current opioid crisis.

2. Design and Methods

The goal of this project is to create a system that provides a constant level of medication to the bloodstream. The method used here is a feedback control system, where the concentration of medicine within the bloodstream will be the system’s main variable. To achieve a constant level of medication in the bloodstream, a feedback control system that considers the amount of medication that can be administered at one time, resistive forces due to biological factors and disturbances, and a maximum allowed concentration will be considered. Figure 1 below is a flowchart showing how the ML algorithm was developed.

Shown in the flowchart above, testing is done by administering the medication to the patient and measuring the concentration of the medicine in the bloodstream. This collection of data shows how a patient responds to the medication. This data is then used to train and validate a machine learning algorithm that is designed to place the patient in a category based on their metabolic response to the medication and how the medication is metabolized for a particular patient. This machine learning algorithm can then be implemented into the control system and then utilized in the medicine pump to accurately dispense medication at a rate that the patient does not feel much pain and does not get a high from the medication and develop an addiction to the medicine.

2.1. Deriving the Open Loop Transfer Function

The most important component of this project is to create an open loop transfer function that correctly models the concentration of medication. Figure 2 below

Figure 1. Schematic diagram showing the process of machine learning algorithm development.

Figure 2. Flow Chart representing feedback control system in medication delivery device.

shows an open loop transfer function that represents a model of the concentration of medication in the bloodstream.

In this feedback loop, the input is the quantity of medication to be administered into the bloodstream. The output of the system is the concentration of medicine in the bloodstream. The transfer function of the loop will be set up with a mathematical model of the form presented in Equation (1) below.

$m\ddot{x}+b\stackrel{\dot{}}{x}+kx=f\left(t\right)$ (1)

After deriving the open loop transfer function, the objective is to optimize it using a PID tuner to satisfy the design criteria based on the preliminary results from previous experimentation. The given criteria are used to optimize the effectiveness of the device itself. The device should be able to reach 90% of the optimal medicine concentration in under 25 minutes, should not exceed the optimal medicine concentration by more than 5%, and once the optimal concentration is reached, it shall not fluctuate by more than 2% of its setpoint concentration value [9].

2.2. Machine Learning Implementation

The focus area of our project is ML of individual scenarios. Included in this focus is trying to understand how disturbances and other factors may accelerate or decelerate the rate at which the body metabolizes the medicine. A list of disturbances includes physical activity, such as running, standing still, and laying down. For each of these scenarios, bodily activity will differ, and the drug usage will increase or decrease, depending on the altered rate of metabolization. To do this, we used various sets of input data and attempted to predict what will occur. By repeating computations for many sets of input data, we can gain a better understanding of how the rate of metabolization affects medicine concentration. In this situation, the variable input data includes a change in the medicine’s rate of decay in the blood, amount of medicine allowed in the bloodstream, the natural resistance to a certain amount of the drug, and factors regarding consumption of food and or drink.

To setup the ML, input data from 100 patients is divided accordingly: 70 percent of the data is used to train the algorithms and 30 percent of the data is used to validate the algorithms. The excel sheet containing the data needs to be converted into a comma-delimited CSV file. This is the file that needs to be loaded in MATLAB. The ML process will be accomplished using the Classification Learner app found in MATLAB. Each patient is assigned a letter (A, B, C, D, or E) depending on how the medication concentration changes over the course of 10 hours after the patient is injected with 20 mg of medication. The letters are used as the classifiers and the concentration values at the 10-time intervals are used as predictors to conduct the ML. Four different algorithms are considered: a fine tree algorithm, a fine k-nearest neighbors (KNN) algorithm, a gaussian naive Bayesian algorithm and a cubic support vector machine (SVM) algorithm. The algorithms are trained using a cross-fold validation that is set to 5 folds by default. The algorithms are evaluated based on any losses between the training prediction accuracy and the validation prediction accuracy. The algorithm with the smallest loss between the training and validation accuracy is the best suited for this application. These results will then be compared to the results found from a larger data set featuring 1000 patients, this larger data set is synthesized from the original small data set.

DL is also implemented in Simulink to predict the concentration of medication in the bloodstream this time using the output data from the device model as the input data for the network. A multilayer perceptron (MLP) network is trained using the same dataset used in the ML setup with the same data split of 70/30. The CSV file that was used needed to be edited, such as deleting the patient’s name column and moving the category column to the rightmost column so that the data entries are the first 10 columns in the table. The structure of this network is illustrated in Figure 3.

The network that is developed is based on the feature network demonstrated in [10] and is designed with 7 layers: A feature input layer set at 10 (the number of predictor columns) with options selected to utilize z-score normalization. A fully connected layer set at 50. A batch normalization layer. A rectifying linear unit layer. Another fully connected layer this time set at 1 (the number of classifier columns). A softmax layer. And finally, a classification layer. The training options utilized in this network use the Adam optimizer with mini batch size set to 10, shuffling occurs at every epoch, training progress will be plotted and verbose is set to false. This network is then implemented into a separate MATLAB/Simulink model using a predict block. Make sure that the MATLAB software being used has the Deep Learning Toolbox installed and the version needs to be 2020b or later, because the deep neural network blocks are only available from this version of MATLAB onwards.

The data collected is the difference between the concentration set point and active blood concentration, this difference is then scaled up by a factor of 10,000 to make the difference detectable by the network. This data is then collected as a timeseries and then formatted as a feature of size 11 using a MATLAB program

Figure 3. Structure of neural network.

file (MLP networks only takes data that is formatted as a feature). The network that is designed removes an entry from the feature, making the feature’s size 10, which is the number of samples that are taken over the 10-hour time period. The formatted feature is then manipulated by a matrix transpose operation in the new MATLAB/Simulink model, the transposed feature can then be utilized by the predict block. After the Simulink model finishes running, the result is a validation prediction accuracy based on the input data.

3. Results and Discussion

To approach the design of our device, the first task is finding the open loop transfer function that represents the concentration of medicine in the blood. As a prerequisite to completing this phase of the project, many values, such as the volume of the medication reservoir, the minimum dosage of the drug, and the reference and maximum concentration of the drug were computed. Table 1 shows each computed value and how it was computed.

Analyzing these values, we can notice the necessary set point of the feedback system to equal the concentration setpoint 7.55 mg/ml. The minimum dosage of the drug is the amount of the drug administered with one step of the linear actuator. A known value of the project is that the minimum step of the actuator is 0.05 mm. The maximum concentration of the drug is the maximum allowed concentration of the medicine in a patient’s body without being dangerous. This is equivalent to the overshoot of the feedback loop, which was given as 5% in the design criteria. Thus, the maximum allowable concentration of the drug in the bloodstream is 7.95 mg/ml.

After finding all initial values and requirements of the project, the open loop transfer function can be computed. From Equation (1) presented in the design approach, m, b, and k represent known values and x(t) represents the output of the function. The output of the function is the concentration of medicine in the blood, so we can replace x(t) with c(t). The input of the function, f(t), is the amount of medicine introduced to the system. The input can be represented as

Table 1. List of computed values.

u(t). By replacing Equation (1) with these updated variables. We can form the mathematical model of the system in Equation (2).

$m\ddot{c}+b\stackrel{\dot{}}{c}+kc=u\left(t\right)$ (2)

To find the transfer function of the system, we must take the Laplace Transform of the function and solve the equation to a ratio of output over input. In this equation, we have four known variables. The mass of blood is 8 kg, the damping coefficient of blood is 0.0006 N∙s/m, which can also be represented as 0.0006 s∙ml, and the spring constant can be represented as the volume of blood, because the blood stores energy of the medication. The volume of blood is equivalent to the mass divided by the density of blood, which is 1060 kg/m³. Therefore, we find the volume of blood to be 7.55 l, which converts to 7550 ml. The input, u(t) is the medicine introduced to the blood with one step of the actuator. This minimum dosage has been calculated as 22.5 mg. The collective mass of the system can be calculated using Equation (3).

$M=\frac{u\left(t\right)-b\stackrel{\dot{}}{c}-kc}{\ddot{c}}$ (3)

Using the above equation, we find the mass of the system to be 328 s²∙ml. The updated equation is shown in Equation (4).

$328\ddot{c}+0.0006\stackrel{\dot{}}{c}+7550c=22.5$ (4)

Taking the Laplace Transform of the function and putting it in terms of the ratio of output over input, we find the transfer function in Equation (5).

$\frac{C\left(s\right)}{U\left(s\right)}=\frac{171s^2+319s+1}{328s^2+0.0006s+7550}$ (5)

This transfer function is an unoptimized and untuned control system, meaning there are no gain factors that bolster the quality of the feedback system. As mentioned previously, this transfer function is a mathematical model representing the concentration of medicine in the bloodstream of the device user. As this transfer function is untuned, it oscillates many times before reaching a steady state at the optimal concentration.

A simplified Simulink model of the open loop system is presented in Figure 4.

In this model, the concentration of medicine in the blood transfer function is the plant of the system. The medicine input is a step function that represents the optimal amount of the drug that should be introduced into the bloodstream to maintain safety of the patient while also providing the highest level of comfort. This value was computed using trial and error in the Simulink model and was found to be 92 mg of medication. So, over the course of 1500 seconds (about 25 minutes), 92 mg of medication must be introduced to the bloodstream to reach the optimal setpoint. In addition to the medicine input set, an input step function titled “Concentration Setpoint” was inserted in the model to give a reference for the transfer function plot. This step was also given a final value equal to 7.55 mg/ml. This model was from a time of t = 0 seconds to t = 1500 seconds, and the resulting plot is shown below in Figure 5.

Figure 4. Open loop system model in MATLAB/Simulink.

Figure 5. Open loop Simulink plot of the model.

In the graph, there are two different plotted variables. The red plot is the concentration setpoint, and the blue plot is the open loop transfer function with no feedback. The transfer function plot repeating oscillates, due to the small damping coefficient of the blood. As the plot is slowly stabilized, we can notice that the signal oscillates more sporadically as the simulation progresses. This open loop control system is not optimal because of the frequent highs and lows of medicine concentration, as well as the extremity of these highs and lows. To optimize these functions, a PID tuner in MATLAB/Simulink was used to find the Proportional, Integral, and Derivative gains of the system that assist to stabilize the system in a quicker and more efficient manner.

To optimize the feedback control system, we will use a PID tuner block and the PID automatic tuning application in Simulink. The first step of this process is to model the feedback control system in Simulink. In Figure 6, the closed loop model with a feedback loop and PID tuner is shown. As opposed to the open loop system, the medicine step input in the closed loop system is equal to the concentration setpoint of medicine. This is due to the optimization of the system using PID tuning.

Figure 6. Closed loop function model in MATLAB/Simulink.

To tune the system, three gain parameters, Proportional, Integral, and Derivative, must be found. In Simulink, an automatic PID tuning application can be used to optimize the transfer function and reach the desired parameters. Per the design criteria, the rise time must be less than 25 minutes and the overshoot must not exceed 5% of the concentration setpoint of 7.55 mg/ml Figure 7 below shows the PID tuning application with the mathematical model of the system inserted for tuning.

To tune the function, two slider bars at the top can be adjusted to change the response time and the aggressiveness with which the tuner will act to steady the system. By adjusting these variables in conjunction with each other to satisfy all criteria of the design, we can find the optimal gain controllers. These gains can be exported into the Simulink model. When exported, the gain values automatically tune the system with reference to the step input of the system. The data results of the system are shown in Table 2, and the Simulink plot of the optimized feedback control system is shown in Figure 8. As seen, the optimized feedback loop has a rise time of 24 minutes and an overshoot of 0%.

For the ML aspect of the project, training methods were used to predict blood concentration values for five different metabolism types. A data pool of 100 entries was provided, of which 70% were used to build the algorithm. Figures 9-12 present the confusion matrices for the fine tree, gaussian naïve Bayesian, cubic SVM and fine KNN learner respectively, which indicates the number of correct classifications per class, in blue, and incorrect classifications per class, in red. The validation and test accuracies for the four classifiers under the small dataset is shown in Figure 13. The chart shows that with the use of the smaller dataset, the validation and test accuracies are not consistent with either a decrease or increase in accuracy.

The same ML method was repeated, this time using the larger 1000 entry dataset to train the learners. The confusion matrices for the four learners, in the same order as before, are displayed in Figures 14-17. The validation and test accuracies for the learners under the larger dataset is shown in Figure 18, The Tree, NB and SVM classifiers had the highest validation accuracy of 100% and retained this accuracy during the testing phase. This shows that a larger dataset

Table 2. Optimized feedback control system data results.

(a)(b)

Figure 7. PID tuning application: (a) Step Plot and (b) Variables.

Figure 8. Simulink plot showing the PID optimized feedback loop result.

Figure 9. Confusion matrix for tree learner using small dataset.

Figure 10. Confusion matrix for Bayesian learner using small dataset.

Figure 11. Confusion matrix for SVM learner using small dataset.

Figure 12. Confusion matrix for KNN learner using small dataset.

Figure 13. Validation and test accuracies for classifiers using small dataset.

Figure 14. Confusion matrix for tree learner using large dataset.

Figure 15. Confusion matrix for Bayesian learner using large dataset.

Figure 16. Confusion matrix for SVM learner using large dataset.

Figure 17. Confusion matrix for KNN learner using large dataset.

Figure 18. Validation and test accuracies for classifiers using large dataset.

will aid in developing a better classification learner. Going forward either the Tree, NB or SVM learners should be chosen for implementation in the control system of the medicine pump.

The difference between the concentration set point and the blood concentration is stored in a MAT-file named “signal”, as shown in Figure 19.

The data collected in the MAT-file mentioned above is filtered to make it a feature of size 11 where the first 10 datapoints reflect the 10 samples taken over the course of the 10-hour timeframe. The filter data set is then used as the input for the DL Simulink model, shown in Figure 20, where the input undergoes a

Figure 19. Closed loop function model modified for data collection.

Figure 20. DL model in MATLAB/Simulink.

Figure 21. Documentation for predicting block parameters.

matrix transpose and is then utilized by the predict block. The output is stored in a MAT-file titled “results.” The documentation for the predict block is displayed in Figure 21 to show the MAT-file containing the network is used and the mini batch size is set to 10. From the output, the model calculated a 61.4% probability that the concentration data is typical of category A patients. It is also reported that there is a 37.1% probability that the patient could fall under category D. This demonstrates how ML can be implemented alongside control design to personalize medication distribution.

4. Conclusions

In this paper, we have examined various methods for classifying the metabolization behavior of patients in order to appropriate control parameters for dosing pain medication. These parameters indicate a starting point for a control system to provide a steady concentration of medication to address patient pain following a major surgery. We have determined that the Tree, SVM, and NB machine learning classification methods can be used to accurately classify patients into groups of appropriate control parameters based on temporal profiles of their metabolization of opioid pain medications. Although these three of the four methods gave similarly accurate results, further research should be done to indicate which of the classification methods would be best for this specific application. This may be done through more testing with a wider variety of patients and patient activity or by varying other parts of testing to find which methods is the most consistently accurate.

An open loop transfer function was designed to model the blood in the patient. A PID tuner was then utilized to form a closed loop transfer function using our initial transfer function. Machine learning (ML) and deep learning (DL) techniques were explored to theorize how the medicine pump can predict how much medication to dispense based on a patient’s immediate metabolic state. It was determined that the closed loop transfer function is a far more optimized representation of the mathematical model than the open loop transfer function. Higher validation and test accuracies were achieved using large datasets during the design of machine learning algorithms. The proposed DL algorithm was able to successfully predict patient class based on given medication concentration measurements over a 10-hour period. Through use of a PID controller, alongside with ML methods, open loop transfer functions can be totally redesigned and optimized for reality.

We have validated the PID control parameters using a closed-loop feedback control system that produced an appropriate medication concentration rise time of 24.3 minutes and a settling time of 46.0 minutes, in accordance with the typical time of peak concentration in human patients. Therefore, it is shown that the method of maintaining a steady concentration of pain medication can be achieved through a use of a PID feedback control system. Furthermore, it is shown that the PID control parameters can be accurately determined by classifying patient metabolization profiles with machine learning methods. Now that it has been deemed possible to use this type of system for our applications, additional research may be performed to optimize this closed loop transfer function even further, therefore optimizing the entire system. This would first involve collecting more data for testing.

Working in concert, these approaches can ensure that a patient does not overdose on medication or achieve a dangerously high concentration, as is typical with contemporary dosing methods. This work lays the theoretical groundwork for a feedback-driven control system and is integral to the design of a feedback-driven medicine pump.

Additional research to be conducted would involve utilizing reinforcement learning techniques to develop an algorithm that governs the medicine pump’s dispersal of medication based on the patient’s metabolic response to the medicine. In order for this to be realized, a reward function needs to be designed to relate the medication concentration in the patient’s bloodstream to the concentrations that are associated with a certain class. The reward that the algorithm receives is dependent on the difference between the two concentration arrays, the smaller the difference, the larger the reward.

Acknowledgements

This work was partially supported by The Runatek Corporation, Dallas, TX, USA.

Nomenclature

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Tandem Diabetes Care Inc. (2021) t: slim X2 Insulin Pump with Dexcom G6CGM. https://www.tandemdiabetes.com/landing-pages/t-slim-x2-landing-page-g6
[2]	Tandem Diabetes Care Inc. (2019) t: slim X2 Insulin Pump User Guide. https://www.tandemdiabetes.com/docs/default-source/product-documents/t-slim-x2-insulin-pump/user_guide_pump_tslimx2_wg5.pdf?sfvrsn=eb30ad7_26
[3]	Johns Hopkins University (2021) Implantable Cardioverter Defibrillator (ICD) Insertion. https://www.hopkinsmedicine.org/health/treatment-tests-and-therapies/implantable-cardioverter-defibrillator-icd-insertion
[4]	Gadze, J.D., Bamfo-Asante, A.A., Agyemang, J.O., Nunoo-Mensah, H., and Opare, K.A.B. (2021) An Investigation into the Application of Deep Learning in the Detection and Mitigation of DDOS Attack on SDN Controllers. Technologies, 9, Article No. 14. https://doi.org/10.3390/technologies9010014
[5]	Indrawan, R., Saadah, S., and Yunanto, P.E. (2021) Blood Glucose Prediction Using Convolutional Long Short-Term Memory Algorithms. Jurnal Ilmu Komputer dan Informatika, 7, 90-95. https://doi.org/10.23917/khif.v7i2.14629
[6]	Nguyen, Q.H., Ly, H.B., Lanh, H.S.., Al-Ansari, N., Le, H.V., Quan, T.V., Prakash, I. and Pham, B.T. (2021) Influence of Data Splitting on Performance of Machine Learning Models in Prediction of Shear Strength of Soil. Mathematical Problems in Engineering, 2021, Article ID: 4832864. https://doi.org/10.1155/2021/4832864
[7]	Tao, H., Al-Sulttani, A.O., Salih Ameen, A.M., Ali, Z.H., Al-Ansari, N., Salih, S.Q. and Mostafa, R.R. (2020) Training and Testing Data Division Influence on Hybrid Machine Learning Model Process: Application of River Flow Forecasting. Complexity, 2020, Article ID: 8844367. https://doi.org/10.1155/2020/8844367
[8]	Uçar, M., Nour, M., Sindi, H. and Polat, K. (2020) The Effect of Training and Testing Process on Machine Learning in Biomedical Datasets. Mathematical Problems in Engineering, 2020, Article ID: 2836236. https://doi.org/10.1155/2020/2836236
[9]	Goh, C.H. (2021) System Dynamics & Control Project Guidelines [Class Handout]. The University of Texas, Tyler, MENG 4312.
[10]	The MathWorks, Inc. (2022) trainNetwork—Train Deep Learning Neural Network. https://www.mathworks.com/help/deeplearning/ref/trainnetwork.html