Multiperspective Assessment of Enterprise Data Storage Systems: The Use of Expert Judgment Quantification and Constant Sum Pairwise Comparison in Finding Criteria Weights

Abstract

Digital transformation has taken center stage in every IT organization. Data is being created at various sources: edge, core, and cloud at an unprecedented rate. For enterprise IT infrastructure, this means more places to store data and more ways to store them. Storage solutions can be broadly categorized as Direct Attached Storage (DAS), Storage Area Network (SAN), Network Attached Storage (NAS), Hyperconverged Infrastructure (HCI), and Public Cloud Storage, each with its advantages and potential drawbacks. Besides computing and networking, storage is one of the core physical components of an IT infrastructure. Application performance and availability depend strongly on their underlying storage. As such, the selection of storage systems is one of the critical decisions for IT executives. Assessment of Enterprise Data Storage Systems (EDSS) for selecting the one that provides a comprehensive solution requires not only the consideration of technical performance and economic feasibility but also other perspectives such as strategic, operational, and regulatory. An assessment model with multiple perspectives and related criteria will serve as a valuable reference in the decision-making process. This study uses expert judgment to validate an assessment model covering Strategic, Technological, Operational, Regulatory, and Economic (STORE) perspectives and their related criteria. Expert judgment is also used to calculate the criteria weights using the constant sum pairwise comparison method. The results can be used for the evaluation of various storage alternatives under consideration. It is anticipated that the STORE assessment model and criteria weights will be valid for IT organizations in their long-term strategic decision-making.

Share and Cite:

Shrestha, L. and Sheikh, N. (2021) Multiperspective Assessment of Enterprise Data Storage Systems: The Use of Expert Judgment Quantification and Constant Sum Pairwise Comparison in Finding Criteria Weights. Open Journal of Business and Management, 9, 955-980. doi: 10.4236/ojbm.2021.92051.

1. Introduction

The source of unprecedented data growth in recent years is many and varied. International Data Corporation (IDC) estimated that the Global Datasphere, a measure of all new data collected, created, and replicated in a year across the globe, will grow from 33 Zettabytes (1021 bytes) in 2018 to 175 by 2025 (Reinsel et al., 2018). A zettabyte is a trillion Gigabytes.

In May 2020, IDC published an update. The estimated growth of data for the year 2020 alone was 59 Zettabytes with a forecast of continued growth through 2024 with a five-year compound annual growth rate (CAGR) of 26%. The COVID-19 pandemic contributes to this figure by causing an abrupt increase in the number of work-from-home employees and rapid digitization (IDC, 2020).

A vast majority of this data is transmitted, processed, and stored at enterprise data centers. The remarkably high rate of new data creation means there need to be more storage systems in the enterprise data centers. These days, enterprise IT infrastructure consists of tens of thousands of physical and virtual servers and their associated hardware like servers, network equipment, and storage systems spread across geographies. The amount of data stored in each data center is in multiple Petabytes (1015 bytes).

Storage systems consist of dedicated servers, storage media, and related software to obtain a high-performance, high availability, and efficiently managed system. The main types of storage media used in data centers are tape drives, magnetic hard drives, and solid-state drives. Enterprise-grade hardware is meant to run continuously, twenty-four hours a day, seven days a week.

Storage systems have developed over decades, improving performance, cutting cost, and, most importantly, enabling modern computing needs by supporting the new types of workloads. The recent industry trend in IT infrastructure management emphasizes automation of the daily repetitive tasks through various commercially available software or homegrown scripts. Another trend in enterprise IT is to leverage public cloud storage for offloading some or most management responsibilities to a third party.

Enterprise data storage systems go through hardware refresh every three to five years to take advantage of newly available features and reduce risks due to aging hardware. These are multi-million-dollar decisions involving implementation and data migration plans that span months to years.

Technology assessment is the evaluation and estimation of the nature, quality, or ability of the technology. It started as a form of public policy research to examine various short and long-term consequences of technology use. Such analysis requires consideration of multiple perspectives and criteria.

Multiple Criteria Decision Making (MCDM) is one of the most widely used decision methodologies in various fields that aim to satisfy the multitude of conflicting objectives in the best possible way. We derive measurements by directly comparing objects. Thomas L. Saaty established that direct comparisons are necessary to establish measurements for intangible properties with no scales of measurement (Saaty, 2008). Methods based on pairwise comparison form a significant part of multiple criteria decision making.

Decision-makers need models that are updated, capturing all significant perspectives and criteria. IT executives in the decision-making positions often supplement their knowledge with advice from experts in the field. This study discusses the use of expert judgment in the assessment of enterprise data storage systems. Expert judgment can be defined as an expert opinion given in the context of a specific decision problem. Expert judgment is a recognized, mature research methodology suitable for assessing emerging technology where benchmarks have not been established or no objective data is available. Expert judgment quantification utilizes rating instruments in the form of a questionnaire to convert informed estimates from the experts to numeric values. This study uses a constant sum pairwise comparison to record one criterion or perspective’s importance versus another. The information that the experts provide becomes data. Conclusions are drawn by combining expert judgments as an aggregation of quantitative estimates.

In this study, we map the process of expert judgment to validate the STORE assessment model for assessing enterprise data storage systems (Shrestha & Sheikh, n.d.). We also used quantification of expert judgment in finding the criteria weights. This study speaks primarily to the decision-makers who fill one of these roles:

· Senior Executives responsible for leadership and technology purchase decisions

· Storage Managers responsible for providing the storage services

· Storage Architects responsible for the design of storage solutions

· Storage Engineers engaged in the implementation of storage solutions

· IT Operations Staff responsible for the daily operations

2. Literature Review

We performed a literature review to understand and develop a scholarly base for the three related research areas: Enterprise IT, Data Storage, and Technology Assessment as shown in Figure 1 (Shrestha & Sheikh, n.d.).

The review enabled identifying gaps in prior research, specifically, the lack of a comprehensive decision model, covering Strategic, Technological, Operational, Regulatory, and Economic (STORE) perspectives. Each of the STORE perspectives was subjectively categorized and considered a preferentially orthogonal dimension. Preferentially orthogonal perspectives are the independent dimensions. The combined dimensions are then deemed to be comprehensive for the assessment of EDSS.

Strategic perspective considers high and low level, short- and long-term goals of the organization. Technological perspective considers various criteria that relate to the capability and efficiency of the storage system. The operational perspective considers how the product will affect the day-to-day operations of the IT function. The regulatory perspective considers the legal aspects of technology implementation. The economic perspective considers the financial aspects of the solution. We derived twenty criteria by grouping the related concepts and categorized them under the five STORE perspectives.

This study expands our literature review to expert judgment quantification and constant sum pairwise comparison in finding criteria weights for multiple criteria decision making. We explore the recent use of these methods in the information technology domain.

Multiple criteria decision making refers to all methods that help designate a preferred alternative and rank alternatives based on subjective preferences, where there is more than one criterion (Ho, 2008). Some authors (Zimmermann, 1991; Chen & Hwang, 1992) categorized MCDM into two categories: 1) Multi-Attribute Decision Making (MADM) problems, where the number of alternatives is predetermined, and 2) Multi-Objective Decision Making (MODM), where they are not.

MCDM has been used in the study of cloud service selection (Rehman et al., 2011), IT infrastructure selection for smart grid (Rezagholizadehl et al., 2013), big data storage selection (Kachaoui & Belangour, 2019), IT disaster recovery site selection (Yang et al., 2015) and justification of IT investments (Borenstein & Betencourt, 2005).

Figure 1. Literature review intersection.

Expert judgment quantification adds substantial value in analyzing complex problems when there are no universally accepted scientific laws or extensive data available. Keeney & Von Winterfeldt (1989) stressed the value of quantifying expert judgments to complement the expert’s qualitative thinking and reasoning. They also highlighted the need for explicit judgments to avoid misinterpretations and misuse.

Expert judgment has been studied in various fields, including cybersecurity (Holm et al., 2014), web development projects (Torrecilla-Salinas et al., 2019), regression models of software effort estimation (Tsunoda et al., 2012), the potential of blockchain in supply chain management (Kopyto et al., 2020) and addressing uncertainty in high technology system design (Chytka et al., 2006).

The constant sum pairwise comparison method is used in the scientific study of preferences, attitudes, and requirements engineering. It reflects the importance or priority attached by a respondent to one entity compared to another. The number of independent pairwise comparisons for n number of entities is n(n − 1)/2. In multiperspective hierarchical decision making, criteria are the lower-level entities to perspectives in the hierarchy. Constant sum pairwise comparison has a relative orientation providing more decision context than binary choices. It is unavoidably more complex than a binary or discrete choice task, resulting in inattention or higher drop-out rates (Skedgel & Regier, 2015).

Pairwise comparison has been used in various fields, including IT infrastructure refresh planning for enterprises (Daim et al., 2011), engineering design (Dym et al., 2002), document ranking algorithms in information retrieval systems (Ozbey & Dincsoy, 2020), intelligent transportation recommendation systems (Borodinov et al., 2020), and image quality assessment (Zhang et al., 2017).

3. Development of Assessment Model

Figure 2 shows the STORE assessment model with five perspectives and twenty criteria in a hierarchy (Shrestha & Sheikh, n.d.). Note that certain aspects of a criterion are covered by others in the model. For example, technical aspects of data security under regulatory perspective are considered in technology features under technological perspective.

Short definitions of the perspectives and criteria:

Strategic Perspective: Strategic perspective considers high and low level, short- and long-term goals of the organization.

Technological Perspective: Technological perspective considers various criteria that relate to the storage system’s capability and efficiency.

Operational Perspective: Operational perspective considers how the product will affect the day-to-day operations of the IT function.

Regulatory Perspective: Regulatory perspective considers the legal compliance aspects of technology implementation. Note that some regulations like GDPR cover more than one criterion in the STORE regulatory perspective.

Economic Perspective: Economic perspective considers the financial aspects of the solution.

Figure 2. STORE assessment model.

Business Strategy: Business strategy is a set of guiding principles that aim to achieve business objectives like service enablement, revenue growth, and cost-saving.

Technology Strategy: Technology strategy explains how we should utilize technology as part of an organization’s business strategy.

Organizational Readiness: Organizational readiness refers to the availability of a skilled workforce and defined processes for technology implementation and operations.

Technology Feature: Technology features of storage systems include policy-based provisioning, orchestration, storage snapshot, and replication.

System Performance: System performance of storage systems includes throughput in IOPS and latency.

System Reliability: The system reliability of a storage system is a measure of performing consistently well. It contributes directly to the high availability of applications supported by the system.

Capacity Management: Capacity management refers to the ease with which capacity is expanded when needed. Capacity management is aided by data reduction techniques like compression and deduplication.

Technological Complexity: Technological complexity refers to a difficulty understanding the storage system and its interaction with other IT components.

Storage Implementation: Storage implementation for an application involves setting up the physical hardware and cables, configuring the device, testing, and migrating data.

Storage Administration: Storage administration tasks include provisioning storage, creating storage units, and performing cleanups.

System Monitoring: System monitoring is essential for the fine-tuning of storage systems and identifying performance bottlenecks.

System Reporting: System reporting is essential for keeping track of how well the system fulfills the needs. It also helps in making decisions related to capacity expansion and others.

Vendor Support: Vendor support is characterized by the ease of creating service requests, communicating with support engineers, and a clear escalation path.

Data Privacy: Data privacy relates to the protection of consumer data. Examples of the regulations that might be applicable are European Union General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and others.

Data Security: Data security refers to the prevention of unauthorized access. Laws like the Cybersecurity Information Sharing Act (CISA) equip organizations to secure the data from the latest cyber threats.

Data Retention: Data retention means the safe keeping of data for future access. Examples of regulations that might be applicable are the Sarbanes Oxley Act (SOX) and Health Insurance Portability and Accountability Act (HIPPA).

Data Transfer: Data transfer refers to the mobility of data. An example of data transfer regulation is cross-border data transfers under GDPR.

Capital Expense: Capital expenses (CAPEX) consist of purchasing equipment or services towards fixed assets that the company will use beyond the current year.

Operating Expense: Operating expenses (OPEX) refers to the ongoing expenses for the operation and maintenance, including the cost of power, space, and cooling in the data center.

Total Cost of Ownership: Total cost of ownership (TCO) is a holistic view of the enterprise’s expenses over time.

4. Expert Judgment Process

Figure 3 shows the process of expert judgment quantification used in this research study. It involved five steps: Define Decision Problem, Recruit Experts, Design Research Instruments, Collect Expert Judgment, and Analyze Expert Judgment.

4.1. Define Decision Problem

Data storage systems are essential components of enterprise IT operations. Application performances depend heavily on the underlying data storage systems. Storage system implementation and data migration can take years of coordinated effort and tens of millions of dollars in investment. A successful storage implementation can bring new IT capabilities, and a bad case can jeopardize entire enterprise IT stability. Performance issues in the storage systems or a total failure can cause costly service outages. Therefore, the selection of EDSS is a critical decision for IT executives.

Figure 3. Expert judgment process.

In enterprise IT infrastructure, hardware refresh takes place every three to five years. The supplier’s continual development in computer hardware and software technologies provides new features, but it also introduces more variables for the decision-making process. The availability of more options further complicates the process.

IT executives need to assess various storage systems and select the best alternative to fulfill the business needs. Evaluation of an EDSS needs careful consideration of all related perspectives and criteria. A hierarchical model with criteria weights allows decision-makers to apply numeric methods. A multi-criteria decision model to assess the storage systems must be developed for a comprehensive approach to this problem. Experts are needed to validate the assessment model and assign weights subjectively.

4.2. Recruit Experts

The potential participants, experts in EDSS, were contacted initially through email, telephone calls, and in-person interviews to introduce them to the research. Their potential value was determined by expertise on the research topic and ability to answer related questions. All selected experts have demonstrated experience through vocation, education, or both with a minimum of ten years of related experience.

We selected the experts based on their years of related experience working with leading information technology organizations. We formed six panels from the twenty-six participants—one for calculating the perspective weights and one each for criteria weights in the five STORE perspectives. Based on work experience, an expert could be included in more than one panel. Participation was voluntary and, we did not provide any financial incentives for participation.

4.2.1. Expert Qualification

Table 1 shows the expert’s experience in years—total IT experience, and experience in each of the STORE perspectives related to EDSS. An asterisk (*) indicates the experience of fewer than ten years.

Table 1. Work experience of experts in years.

4.2.2. Formation of Expert Panels

From the expert’s pool (Table 1), we formed six panels—one for calculating perspective weights and one each for criteria weights under the five STORE perspectives. Experts with at least ten years of experience in all five perspectives are included in panel 1. Panels 2 through 6 include experts with at least ten years of experience in the related perspectives.

An expert can be included in more than one panel.

Panel 1: Experts in all five STORE perspectives (14 experts).

Expert 2, 4, 6, 7, 8, 11, 12, 14, 15, 20, 21, 23, 24, 26.

Panel 2: Experts in Strategic Perspectives (18 experts).

Expert 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 19, 20, 21, 23, 24, 26.

Panel 3: Experts in Technological Perspectives (24 experts).

Expert 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23, 24, 25, 26.

Panel 4: Experts in Operational Perspectives (22 experts).

Expert 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 23, 24, 26.

Panel 5: Experts in Regulatory Perspectives (19 experts).

Expert 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 23, 24, 26.

Panel 6: Experts in Economic Perspectives (16 experts).

Expert 2, 4, 6, 7, 8, 10, 11, 12, 14, 15, 17, 20, 21, 23, 24, 26.

4.3. Design Research Instruments

Pairwise comparison is a process in which experts rate a set of criteria, perspectives, or alternatives only two at a time. While this method is time-consuming to elicit all possible combinations and only provides relative data relations, recent research shows that people make better relative judgments than direct estimates (Benini et al., 2017). This study adopts the pairwise comparison method with seven stepped levels to assess solar photovoltaic technologies (Sheikh, 2013).

We selected an elicitation situation of individual experts instead of interactive group or Delphi to avoid potential bias from group dynamics. We chose a web-based form as the mode of communication to capture encoded expert judgments. There were two questionnaires for the experts—the first questionnaire (Appendix A) validated the assessment model, and the second established criteria weights (Appendix B).

4.4. Collect Expert Judgment

We met with each expert through a video chat to explain the research study and question format through screen share. The sessions took about 20 minutes each. We then sent the web link to the experts with the following greeting message:

“This research study aims to develop a comprehensive assessment model to evaluate Enterprise Data Storage Systems (EDSS). EDSS represents the servers, storage media, or appliance used for storing digital data. Some examples of EDSS are Storage Area Network (SAN), Network Attached Storage (NAS), Direct Attached Storage (DAS), Hyperconverged Infrastructure (HCI), and Public Cloud Storage.

You are being asked to participate in this research study because of your EDSS expertise. Your experience working with the leading information technology organizations makes you uniquely valuable to this research.

The questionnaire (I) relates to the validation of the STORE assessment model. Questionnaire (II) has a total of 42 pairwise comparison questions with multiple-choice answers. Questions are not specific to any job, organization, or vendor but EDSS in general. These questionnaires should not take more than 30 minutes.

Informed Consent: Participation in this research activity is voluntary. The participants may withdraw at any time without penalty or loss of benefits. The questionnaires are anonymous. Please do not enter any personally identifiable information.”

4.5. Analyze Expert Judgment

After collecting the expert judgment, we used a quantification scale described below with a constant sum of 100.

Attribute A is four times as important as Attribute B; A = 80 and B = 20

Attribute A is three times as important as Attribute B; A = 75 and B = 25

Attribute A is two times as important as Attribute B; A = 67 and B = 33

Attribute A is equally important as Attribute B; A = 50 and B = 50

Attribute A is one-half times important as Attribute B; A = 33 and B = 67

Attribute A is one-third times important as Attribute B; A = 25 and B = 75

Attribute A is one-fourth times important as Attribute B; A = 20 and B = 80

A score of zero is entered for both A and B when experts do not qualify for the panel. In the next steps, we aggregated the scores to obtain combined values.

4.5.1. Panel 1: Perspective Weights

Table 2 shows the quantified expert judgment on questions 1 through 10 (Appendix B) using the method explained in Section 4.5.

Sum of perspective scores from all pairwise comparisons:

Strategic Perspective = 845 + 789 + 662 + 713 = 3,009

Technological Perspective = 555 + 843 + 696 + 763 = 2,857

Operational Perspective = 611 + 557 + 717 + 777 = 2,662

Regulatory Perspective = 738 + 704 + 683 + 860 = 2,985

Economic Perspective = 687 + 637 + 623 + 540 = 2,487

Total Score = 3,009 + 2,857 + 2,662 + 2,985 + 2,487 = 14,000

4.5.2. Panel 2: Strategic Perspective

Table 3 shows the quantified expert judgment on questions 11 through 13 (Appendix B) using the method explained in Section 4.5.

Sum of criterion scores from all pairwise comparisons:

Business Strategy = 956 + 875 = 1,831

Technology Strategy = 844 + 1,056 = 1,900

Organizational Readiness = 875 + 744 = 1,696

Total Score = 1,831 + 1,900 + 1,696 = 5,400

4.5.3. Panel 3: Technological Perspective

Table 4 shows the quantified expert judgment on questions 14 through 23 (Appendix B) using the method explained in Section 4.5.

Sum of criterion scores from all pairwise comparisons:

Technology Features = 1,177 + 1,048 + 1,267 + 1,252 = 4,744

System Performance = 1,223 + 1,158 + 1,471 + 1,461 = 5,313

System Reliability = 1,352 + 1,242 + 1,491 + 1,571 = 5,656

Capacity Management = 1,133 + 929 + 909 + 1,437 = 4,408

Technological Complexity = 1,148 + 939 + 829 + 963 = 3,879

Total Score = 4,744 + 5,313 + 5,656 + 4,408 + 3,879 = 24,000

4.5.4. Panel 4: Operational Perspective

Table 5 shows the quantified expert judgment on questions 24 through 33 (Appendix B) using the method explained in Section 4.5.

Sum of criterion scores from all pairwise comparisons:

Storage Implementation = 1,020 + 1,011 + 1,091 + 988 = 4,110

Storage Administration = 1,180 + 1,269 + 1,353 + 1,057 = 4,859

System Monitoring = 1,189 + 931 + 1,214 + 1,091 = 4,425

System Reporting = 1,109 + 847 + 986 + 940 = 3,882

Vendor Support = 1,212 + 1,143 + 1,109 + 1,260 = 4,724

Total Score = 4,110 + 4,859 + 4,425 + 3,882 + 4,724 = 22,000

4.5.5. Panel 5: Regulatory Perspective

Table 6 shows the quantified expert judgment on questions 34 through 39 (Appendix B) using the method explained in Section 4.5.

Sum of criterion scores from all pairwise comparisons:

Data Privacy = 937 + 1,080 + 1,068 = 3,085

Data Security = 963 + 1,246 + 1,154 = 3,363

Data Transfer = 820 + 654 + 917 = 2,391

Data Retention = 832 + 746 + 983 = 2,561

Total Score = 3,085 + 3,363 + 2,391 + 2,561 = 11,400

4.5.6. Panel 6: Economic Perspective

Table 7 shows the quantified expert judgment on questions 40 through 42 (Appendix B) using the method explained in Section 4.5.

Sum of criterion scores from all pairwise comparisons:

Capital Expense = 1,011 + 838 = 1,849

Operating Expense = 589 + 860 = 1,449

Total Cost of Ownership = 762 + 740 = 1,502

Total Score = 4,800

Table 3. Quantified expert judgment for criteria related to Strategic Perspective.

Table 6. Quantified expert judgment for criteria related to Regulatory Perspective.

Table 7. Quantified expert judgment for criteria related to Economic Perspective.

5. Results

We used the following set of equations to obtain the final criteria weights represented by Cji

C j i = p j c j i | p j (1)

Variables Used:

p represents a perspective.j is the index of perspectives ranging from 1 to J.

c represents a criterion. i is the index of criteria ranging from 1 toI.

The method of constant sum pairwise comparison ensures the following three conditions:

Condition I: Sum of all perspective weights equal to one. Note: pj is the weight of the jth perspective.

j = 1 J p j = 1 (2)

Condition II: Sum of all initial criteria weights within each perspective equal to one. Note: c j i | p j is the initial weight of criterion i under perspectivej. Ij is the maximum number of criteria in perspective j.

i = 1 I j c j i | p j = 1 (3)

Condition III: Sum of all final criteria weights equal to one. Note: Cji represents the final weight of criterion i under perspective j.

j = 1 J i = 1 I j C j i = 1 (4)

5.1. Calculation of Perspective Weights

The first column in Table 8 gives the sum of scores for each perspective from all four pairwise comparisons.

Table 8. Calculation of perspective weights.

Total score possible = Number of pairwise comparisons (10) * Number of panelist (14) * 100 = 1,400.

Perspective weight pj = The sum of scores from all pairwise comparisons/Total score possible.

5.2. Calculation of Criteria Weights

Table 9 shows the final result with weights for all twenty criteria, the lowest-level in the STORE hierarchical model.

Table 9. Final results.

6. Discussion

Twenty-five out of twenty-six experts responded with “yes” to the first questionnaire for model validation. One of the experts replied “no,” suggesting a new criterion, Data Resiliency under Regulatory Perspective. Data resiliency refers to the speed and quality of recovery in case data is compromised. From a regulatory perspective, this is similar in concept to data retention. If data retention is ensured through periodic backups, resilience is confirmed. Also, we did not find literature with an established data resilience regulations applicable across industries. We conclude that the STORE assessment model for EDSS is complete and validated by the experts.

The perspective weights in the order of higher importance are Strategic Perspective (21.5%), Regulatory Perspective (21.3%), Technological Perspective (20.4%), Operational Perspective (19.0%), and Economic Perspective (17.8%).

The criteria weights in the order of higher importance are Technology Strategy (7.565%), Business Strategy (7.290%), Capital Expense (6.857%), Organizational Readiness (6.645%), Data Security (6.284%), Data Privacy (5.764%), Total Cost of Ownership (5.570%), Operating Expense (5.373%), System Reliability (4.808%), Data Retention (4.785%), System Performance (4.516%), Data Transfer (4.467%), Storage Administration (4.196%), Vendor Support (4.080%), Technology Features (4.032%), System Monitoring (3.822%), Capacity Management (3.747%), Storage Implementation (3.550%), System Reporting (3.353%), and Technological Complexity (3.297%).

7. Conclusion

A group of twenty-six experts working for large US enterprise IT organizations, each with at least ten years of experience, validated the STORE assessment model. Six expert panels, each with at least ten experts, made constant sum pairwise comparisons to derive weights for the twenty criteria.

Experts put the most weight on Technology Strategy, followed by Business Strategy and Capital Expense. The lowest weight of Technology Complexity indicates that the nature and role of complexity in IT infrastructure management are underestimated or poorly understood.

The results give a reliable understanding of various criteria required to evaluate an enterprise data storage system. However, we caution against over-interpreting the expert judgments in this study and relying too heavily on minor numerical differences.

Contribution to the Body of Knowledge

This original empirical research study validated the STORE assessment model for enterprise data storage systems using expert judgment and calculated criteria weights using constant-sum pairwise comparison.

Assumptions and Limitations

All twenty-six experts worked for large US enterprise IT organizations—this research study considered only the consumer worldview.

A 0 to 100 continuous scale can improve the results’ accuracy compared to the seven stepped levels used in this study. We chose the seven steps at the unanimous request of the experts.

Future Works

Other worldviews such as a vendor, consulting company, or regulating agencies can be considered in future research. Researchers can expand each criterion to multiple factors as the field evolves, definitions are clear, and more commonly accepted benchmarks are established. Results from this study can be used in studies related to the selection of enterprise data storage systems.

Acknowledgements

We thank the twenty-six experts who, despite their busy schedules, took time to listen to our research and participated with great enthusiasm.

Appendix A

Q1. Does the STORE assessment model capture all significant perspectives and criteria in evaluating an EDSS?

 Yes

 No

Q2. If your answer to question 1 is No, please write the new perspectives or criteria to complete the assessment model. (_____________________________________________________)

Appendix B

There is a total of 42 pairwise comparisons with multiple-choice answers. All questions are strictly in the context of EDSS. FigureB1 shows the answer options and screen capture for question 1. The seven answer options are the same for all questions.

Q1. Please rate the importance of Strategic Perspective with respect to Technological Perspective.

 Strategic Perspective is FOUR times as important as Technological Perspective.

 Strategic Perspective is THREE times as important as Technological Perspective.

 Strategic Perspective is TWO times as important as Technological Perspective.

 Strategic Perspective is EQUALLY important as Technological Perspective.

 Strategic Perspective is ONE-HALF times as important as Technological Perspective.

 Strategic Perspective is ONE-THIRD times as important as Technological Perspective.

 Strategic Perspective is ONE-FOURTH times as important as Technological Perspective.

Figure B1. Screen capture of Question 1.

Q2. Strategic Perspective with respect to Operational Perspective

Q3. Strategic Perspective with respect to Regulatory Perspective

Q4. Strategic Perspective with respect to Economic Perspective

Q5. Technological Perspective with respect to Operational Perspective

Q6. Technological Perspective with respect to Regulatory Perspective

Q7. Technological Perspective with respect to Economic Perspective

Q8. Operational Perspective with respect to Regulatory Perspective

Q9. Operational Perspective with respect to Economic Perspective

Q10. Regulatory Perspective with respect to Economic Perspective

Q11. Business Strategy with respect to Technology Strategy

Q12. Business Strategy with respect to Operational Readiness

Q13. Technology Strategy with respect to Operational Readiness

Q14. Technology Features with respect to System Performance

Q15. Technology Features with respect to System Reliability

Q16. Technology Features with respect to Capacity Management

Q17. Technology Features with respect to Technological Complexity

Q18. System Performance with respect to System Reliability

Q19. System Performance with respect to Capacity Management

Q20. System Performance with respect to Technological Complexity

Q21. System Reliability with respect to Capacity Management

Q22. System Reliability with respect to Technological Complexity

Q23. Capacity Management with respect to Technological Complexity

Q24. Storage Implementation with respect to Storage Administration

Q25. Storage Implementation with respect to System Monitoring

Q26. Storage Implementation with respect to System Reporting

Q27. Storage Implementation with respect to Vendor Support

Q28. Storage Administration with respect to System Monitoring

Q29. Storage Administration with respect to System Reporting

Q30. Storage Administration with respect to Vendor Support

Q31. System Monitoring with respect to System Reporting

Q32. System Monitoring with respect to Vendor Support

Q33. System Reporting with respect to Vendor Support

Q34. Data Privacy with respect to Data Security

Q35. Data Privacy with respect to Data Transfer

Q36. Data Privacy with respect to Data Retention

Q37. Data Security with respect to Data Transfer

Q38. Data Security with respect to Data Retention

Q39. Data Transfer with respect to Data Retention

Q40. Capital Expense with respect to Operating Expense

Q41. Capital Expense with respect to Total Cost of Ownership

Q42. Operating Expense with respect to Total Cost of Ownership

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1] Benini, A., Chataigner, P., Noumri, N., Parham, N., Sweeney, J., & Tax, L. (2017). Expert Judgment: The Use of Expert Judgment in Humanitarian Analysis: Theory, Methods and Applications. Geneva: Assessment Capacities Project.
https://www.acaps.org/sites/acaps/files/resources/files/acaps_expert_judgment_-_full_study_august_2017.pdf
[2] Borenstein, D., & Betencourt, P. R. B. (2005). A Multi-Criteria Model for the Justification of IT Investments. Information Systems and Operational Research, 43, 1-21.
https://doi.org/10.1080/03155986.2005.11732711
[3] Borodinov, A., Agafonov, A., & Myasnikov, V. (2020). A Method of Preference and Utility Elicitation by Pairwise Comparisons and Its Application to Intelligent Transportation Recommendation Systems. 10th International Conference on Information Science and Technology, Bath, London, and Plymouth, 9-15 September 2020, 77-85.
https://doi.org/10.1109/ICIST49303.2020.9201952
[4] Chen, S.-J., & Hwang, C.-L. (1992). Fuzzy Multiple Attribute Decision Making Methods (pp. 289-486). Berlin, Heidelberg: Springer.
https://doi.org/10.1007/978-3-642-46768-4
[5] Chytka, T. M., Conway, B. A., & Unal, R. (2006). An Expert Judgment Approach for Addressing Uncertainty in High Technology System Design. Portland International Conference on Management of Engineering and Technology, 1, 444–449.
https://doi.org/10.1109/PICMET.2006.296590
[6] Daim, T. U., Letts, M., Krampits, M., Khamis, R., Dash, P., Monalisa, M., & Justice, J. (2011). IT Infrastructure Refresh Planning for Enterprises: A Business Process Perspective. Business Process Management Journal, 17, 510-525.
https://doi.org/10.1108/14637151111136397
[7] Dym, C. L., Wood, W. H., & Scott, M. J. (2002). Rank Ordering Engineering Designs: Pairwise Comparison Charts and Borda Counts. Research in Engineering Design, 13, 236-242.
https://doi.org/10.1007/s00163-002-0019-8
[8] Ho, W. (2008). Integrated Analytic Hierarchy Process and Its Applications—A Literature Review. European Journal of Operational Research, 186, 211-228.
https://doi.org/10.1016/j.ejor.2007.01.004
[9] Holm, H., Sommestad, T., Ekstedt, M., & Honeth, N. (2014). Indicators of Expert Judgement and Their Significance: An Empirical Investigation in the Area of Cyber Security. Expert Systems, 31, 299-318.
https://doi.org/10.1111/exsy.12039
[10] International Data Corporation (IDC) (2020). Worldwide Global DataSphere IoT Device and Data Forecast, 2020-2024.
https://www.idc.com/getdoc.jsp?containerId=US46718220
[11] Kachaoui, J., & Belangour, A. (2019). A Multi-Criteria Group Decision Making Method for Big Data Storage Selection. In M. Atig, & A. Schwarzmann (Eds.), NETYS 2019: Networked Systems (pp. 381-386). Cham: Springer.
https://doi.org/10.1007/978-3-030-31277-0_25
[12] Keeney, R. L., & Von Winterfeldt, D. (1989). On the Uses of Expert Judgment on Complex Technical Problems. IEEE Transactions on Engineering Management, 36, 83-86.
https://doi.org/10.1109/17.18821
[13] Kopyto, M., Lechler, S., von der Gracht, H. A., & Hartmann, E. (2020). Potentials of Blockchain Technology in Supply Chain Management: Long-Term Judgments of an International Expert Panel. Technological Forecasting and Social Change, 161, Article ID: 120330.
https://doi.org/10.1016/j.techfore.2020.120330
[14] Ozbey, C., & Dincsoy, O. (2020). An Efficient Pairwise Comparison Scheme for Document Ranking. 2020 28th Signal Processing and Communications Applications Conference, Gaziantep, 5-7 October 2020, 1-4.
https://doi.org/10.1109/SIU49456.2020.9302078
[15] Rehman, Z. U., Hussain, F. K., & Hussain, O. K. (2011). Towards Multi-Criteria Cloud Service Selection. Proceedings of 2011 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, Seoul, 30 June-2 July 2011, 44-48.
https://doi.org/10.1109/IMIS.2011.99
[16] Reinsel, D., Gantz, J., & Rydning, J. (2018). The Digitization of the World from Edge to Core. Framingham, MA: International Data Corporation.
[17] Rezagholizadehl, M., Mehrannii, P., Barzegar, A., Fereidunian, A., Moshiri, B., & Lesani, H. (2013). A Probabilistic Partial Order Theory Approach to IT Infrastructure Selection for Smart Grid. International Conference on Control, Automation and Systems, Gwangju, 20-23 October 2013, 488-493.
https://doi.org/10.1109/ICCAS.2013.6703983
[18] Saaty, T. L. (2008). Relative Measurement and Its Generalization in Decision Making Why Pairwise Comparisons Are Central in Mathematics for the Measurement of Intangible Factors The Analytic Hierarchy/Network Process (To the Memory of my Beloved Friend Professor Sixto Rios Garcia). Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas, 102, 251-318.
https://doi.org/10.1007/BF03191825
[19] Sheikh, N. J. (2013). Assessment of Solar Photovoltaic Technologies Using Multiple Perspectives and Hierarchical Decision Modeling. Ph.D. Theses, Portland, OR: Portland State University.
[20] Shrestha, L., & Sheikh, N. J. (n.d.). Multiperspective Assessment of Enterprise Data Storage Systems: Literature Review. Portland International Conference on Management of Engineering and Technology (PICMET).
[21] Skedgel, C., & Regier, D. A. (2015). Constant-Sum Paired Comparisons for Eliciting Stated Preferences: A Tutorial. Patient, 8, 155-163.
https://doi.org/10.1007/s40271-014-0077-9
[22] Torrecilla-Salinas, C. J., De Troyer, O., Escalona, M. J., & Mejías, M. (2019). A Delphi-Based Expert Judgment Method Applied to the Validation of a Mature Agile Framework for Web Development Projects. Information Technology and Management, 20, 9-40.
https://doi.org/10.1007/s10799-018-0290-7
[23] Tsunoda, M., Monden, A., Keung, J., & Matsumoto, K. (2012). Incorporating Expert Judgment into Regression Models of Software Effort Estimation. 2012 19th Asia-Pacific Software Engineering Conference, Hong Kong, 4-7 December 2012, 374-379.
https://doi.org/10.1109/APSEC.2012.58
[24] Yang, C. L., Yuan, B. J. C., & Huang, C. Y. (2015). Key Determinant Derivations for Information Technology Disaster Recovery Site Selection by the Multi-Criterion Decision Making Method. Sustainability, 7, 6149-6188.
https://doi.org/10.3390/su7056149
[25] Zhang, Z., Zhau, J., Liu, N., Gu, X., & Zhang, Y. (2017). An Improved Pairwise Comparison Scaling Method for Subjective Image Quality Assessment. 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, Cagliari, 7-9 June 2017, 1-6.
https://doi.org/10.1109/BMSB.2017.7986235
[26] Zimmermann, H. J. (1991). Fuzzy Set Theory—And Its Applications. Dordrecht: Springer.

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.