1. Introduction
Tourism mobile applications currently can be argued to be one of the most proficient applications that can facilitate the movement of travelers, and dramatically due to their availability on accessing information online [1] . Mobile tourism applications allow the tourists to access and get information everywhere and at any time, where they can use the apps on the move. In particular, every tourism application provides different features and services that user can access easily [2] . There are some features that users can get from these applications in older to satisfy tourist information needs as follows: accommodation, weather, tourist attractions, shopping, news and so on. Among popular mobile applications for tourism are TripAdvisor, Triposo, Airbnb, Smart travelling, KAYAK and Skyscanner [3] .
The assessment of the availability of tourism applications often overlooks the potential impact of both mobile and consumer applications [3] . This makes the usability evaluation of tourism applications more demanding. The literature shows the most challenges of mobile application is its interface and some applications are difficult to use based on content and design [4] . Previous studies attempted to evaluate tourism mobile application but they did not propose metrics such as time taken for each task, error capturing and accuracy of each task [5] . In addition, previous research did not focus on tourism application that provides effective functions for tourists and users can access information easily based on their knowledge.
Besides that, improvement of the usability measurements [6] , is really needed to ensure tourism applications requirements are accessible. However, this research extended from previous student work where she focused on the identification of dimension and criteria for mobile tourism applications, but not on metrics. In identifying the above problems, there is a need for a new usability evaluation metrics that provides an appropriate and suitable measurement for usability evaluation metrics of tourism mobile application. Thus, this study aims in providing an approach for conducting usability evaluation for the tourism mobile applications through expert review. This article organized as follows: 1) Background of the study; 2) Related work; 3) Methodology applied in this study; 4) Result of the proposed metric and conclusion.
The contribution of this paper is to propose new measurement metrics for tourism mobile applications and to evaluate the application. The proposed metrics can measure the applications with their functionalities and features of applications. Therefore, this study simplified to measure the tourism mobile application with the knowledge for the users.
2. Literature Review
2.1. Usability Evaluation
Usability evaluation is an important element to analyze usability issue in any system or applications. Usability issues are identified through evaluation conducted with users. Literature shows many usability models has been referred for conducting usability evaluation. Among the common usability evaluation model are Nelsen, QUIM, mGQM, Harrison and ISO. These are some of the usability evaluation models that have been in use for usability evaluation. However, when application is developed for specific targeted user, requirements of the user need to be incorporated into the application [7] . Application will fail to satisfy the user if the requirement is absent and make it more difficult.
2.2. Usability Evaluation Metrics
Usability metrics measurement is generally used by the examiners to identify what they are going to measure. Between the current usability measures is ISO 9241 part 11, guidance on usability [8] . However, this measure has been used usually but is too common, tough to relate to explicit domains and does not assistant with any quality characteristic to the measure [9] .
The GQM method in emerging measurement metric could be experimental in. Moreover, this approach has also been requested as one of the most goal-focused and most popular methods used among several measurements approaches [8] . Although GQM was previously used to express and evaluate goals for specific projects and environments, its aim has been extended to a larger perspective including quality improvement, measure progress, and plan for project. Therefore, it is assumed that the GQM approach could also possibly be extended to measure the usability guidelines by providing metrics for usability guidelines. In combination, in this study also explains the definition phase of the GQM paradigm, proving the outputs of the first three steps of Basili’s GQM process, the hierarchy of goals, questions, and meaningful metrics [10] .
The previous study only focused on dimension and criteria measurement. But in this new research, it was extended the metrics measurements.
2.3. Tourism Mobile Applications
The tourism mobile applications currently can be said as the most proficient applications that can facilitate the explorers' movement of travelers. The users of this mobile tourism application can easily accomplish the way to places that they are not familiar with. The information gives relating to places of interest is useful and crucial for tourists [2] . The tourism mobile application deals not only the descriptive text of the information provided, but shows pictures of hotspots. In addition to that, tourists will be able to check for the facilities available surrounding certain places like entertainment and restaurants, hotels and other nearby hotspots. As disparate to printed maps that were mainly used by travelers beforehand, the mobile application may be more advantageous. It will help tourists to plan their tour ahead of time, and have an open feeling of quietness amid their occasions, as they will be increasingly arranged [11] .
Therefore, Smart tourism changes tourist information search activities. The tourism information nowadays became very flexible and several at the stages of collecting and getting data. Therefore, tourists can have easily access to all information and activity concerning via internet using different mobile applications [2] . Besides, tourists can also enjoy an experience of the tourist destinations and attraction places by using and applying mobile application. In this way, tourists they can get easily and know about various information related to tour destinations and places that are interesting (Table 1).
The above table shows the most popular and useful mobile tourism applications currently, each application has different functionalities that help users to obtain their trips concerning flights, hotels, restaurants and possible activities on destination through Google Play Store or App Store [12] . Therefore, this table shows Trip advisor is amongst one for tourism mobile application in terms of the functionalities of the application.
3. Methodology
Research Design
Choosing the appropriate method to carry out specific research topics requires a good understanding and careful method of each method because it is not assigned to it [13] . Similarly, in selecting the research method, certain metrics should be taken into account, such as the title of the study, the time period given the nature of the research, budget, available resources and research environment. As mentioned earlier, the main purpose of this study is to propose usability metrics for evaluation of tourism mobile applications, to validate the availability of experts through the use of standards metrics and assessments to ensure the effectiveness of the application. In this case, the metrics and indicators defined as the availability of the metrics in the study are the combinations of objective and subjective measurements, quantitative methods will be used in this study.
The methodology proposed in this paper aims to identify the usability evaluation metrics of tourism mobile application. The methodology proposed has been developed based on the metrics that were identified from the literature review. The methodology that used in this research is quantitative method through a semi-structured questionnaire. So, the research will follow four phases as Figure 1 below:
4. Finding
4.1. Generating Usability Metrics
The main aim of the study is to propose usability evaluation metrics for tourism mobile application. As part of generating metric for the proposed metrics,
![]()
Table 1. Most popular applications of tourism mobile applications.
![]()
Figure 1. Research methodology phase and activity.
requirements of tourism were analyzed to be matched appropriately with the metrics developed earlier through a literature review. This will ensure that, metrics developed are not very general but focused on the tourism applications users need as well. To derive proper metric, QUIM and mGQM were used as basis in this part. This is to ensure identified metrics are adopted from proper and established studies in accordance with the identified requirements of the tourism towards usage of mobile applications.
The next step in the development phase is identifying the metrics for the generated usability dimensions and criteria from previous section. Two types of data for metric have been identified which are the objective and subjective data. Objective data refers to the task performance analysis and measures performance of usage while subjective data refers mainly on the user’s feedback towards usage of the applications [14] . To create the metrics for the criteria that have been generated, models such as QUIM and mGQM were analyzed and modified metrics from these researches were employed into the proposed metrics according to the criteria defined and suited for the study.
Reasons for accommodating these two models are due to the factor which allows freedom in selecting metrics according to the study and applied into the criteria that leave many opportunities to thoroughly analyses suitability of the metrics [15] , as discussed in earlier. Besides that, research having guideline selecting metrics for broader usability goals [15] [16] , and categorizes relationship among metric and criteria appropriately [16] .
The selection of the metrics for the measurement also considered the requirements of tourism that have been analyzed earlier and keeping in mind that the data to be collected is for a mobile platform which differs from desktop platform. This is matched with the criteria to be measured and finally developed the measurable metrics to gather both objective and subjective data. Short listing of selected metrics is done by analyzing the literature from literature review. User context is the most important aspect considered since the user is the tourism applications which will ensure applications are measured accordingly and meet the purpose of the study based in a mobile platform.
Besides that, metrics in literature are mostly focused on desktop application, however, few studies discuss metrics on mobile application compatibility and thus, metrics were chosen based on that perspective as well. Basically, metrics are derived in conscious to be suited with the selected dimensions and criteria that have been applied by studies in literature review.
4.2. Result of Metrics Verification through Expert Evaluation
The expert review was conducted to provide the verification of the proposed measurement of usability evaluation metrics of tourism mobile application. Expert review is one of the significant ways in detecting and removing defects [17] , thus, the study adapted this technique for verification of the proposed usability measurement of usability evaluation metrics of tourism mobile application. Through verification, all the components developed would be able to be confirmed as well as organized and presented appropriately [18] . The main aspect need to be verified in the proposed usability evaluation metrics of tourism mobile application is the usage of the appropriate subjective and objective metrics. Therefore, potential experts in the domain of usability and tourism especially researcher and academician, were identified. The expert was chosen according to the suggestion by [19] [20] .
Five (5) experts were contacted through email and willing to verify the proposed metrics which all of them are academician [21] . Contacted experts led for verification is conducted since it can significantly lead to give accurate result.
Among the five (5) experts, four (4) experts agreed on face to face meeting which were arranged and held with the experts, while one (1) expert agreeing in doing the review through online due to work hectic and unable to set appropriate meeting hour because the expert had the distance between cities that had been a hindrance. The expert’s background was listed in Table 2 below.
Besides completing the verification form provided, comments and suggestion were also given by the experts for metrics improvements. Table 3 below
![]()
Table 3. Verification of usability metrics by the experts.
P: Relevancy of the appropriateness of the metrics. *: irrelevant or unsuitable placements of the mentioned metrics.
summarizes on the verified proposed metrics components by the experts which determined the relevancy of data collection and analysis of the tourism mobile applications usability evaluation. Moreover, the components in the proposed metrics verified in align towards concept of Human Computer Interaction (HCI) and amendments done for any the components needed to be modified or retained in the finalized metrics.
Metrics were marked with symbols were considered to be dropped from the list or moved to more suitable metrics as per suggested by the expert. While metrics that were accepted for relevancy are retained unless comments expecting the metrics moved to suitable metrics. Metrics are being selected from the proposed usability evaluation metrics of tourism mobile applications based on the percentage score obtained by individual metrics as well as comments from the experts. If an item is scored at least 60% average percentage, possibility of the significant results to be produced are high and such item can be considered (Clarke & Warwick, 2001). However, results obtained from the experts shows some metrics scored less than 60% means the metrics are unsuitable and irrelevant for the proposed study. However, mostly the metrics are scored more than 60% means those metrics are suitable and appropriately developed and can produce a significant result. The comments are analyzed carefully on the appropriateness for acceptance and metrics have been modified accordingly. The experts have given comments and suggestions which helps in modifying the proposed usability evaluation metrics of tourism mobile applications. These comments from the experts have been significant contribution to the proposed metrics. Table 4 shows the comments and suggestion received from the experts.
The comments received are then taken into consideration for the modification of the proposed metrics which is discussed further in this section. The comments and suggestion from the experts were examined carefully and the proposed metrics were modified whereas some metrics were removed and redundant metrics were omitted from the proposed metrics (Table 5).
As the expert suggested, the above table shown the metrics that were removed or replaced from the measurement. Experts have also given suggestion for the measurement improvement through other criteria that are appropriate that can measure the dimensions derived. Table 6 below shows the added criteria as per suggested from the experts.
5. Conclusion
The entire study has provided usability evaluation of tourism mobile applications. The literature review method to use the QUIM, mGQM, and GQM models for proper and reliable availability measurements. There are thirty-six metrics that used to achieve respective goals of this study. As a result, the proposed metrics results have been well received by all usability experts who participated in this study based on the results obtained. In addition, the verification results show the ability to assess the proposed metrics reviewed by usability experts.
![]()
Table 4. Expert comments/suggestion.
Through literature review conducted it is also found that there exist mobile applications usability evaluation measurement that for specific tourism applications, therefore, this study attempted in overcoming the shortcoming of the mobile application user are facing especially known as tourism applications. Therefore, the study identified the metrics verification results of tourism mobile application availability assessment based on mobile tourism.
For the proposed metrics, a total of thirty-five metrics is then verified by five (5) experts from the academician. Based on the verification also, the metrics were agreed by the experts to be accurate and applicable for tourism mobile applications usability evaluation. Overall, the experts are satisfied with the metrics results determined by the small adjustments made. Experts’ evaluation results show that metrics can be identified the functionality of usability mobile application, specifically, tourism mobile applications. From the results obtained through the evaluation of usability, it indicates that the collected data is reliable and allows the identification of results and availability problems.