Usability testing on website wadaya based on ISO 9241-11

Wadaya (in Bahasa: Warisan Budaya) is a crowdsourcing information system which is intended for collecting and spreading cultural heritage information in Indonesia. As a crowdsourcing system, Wadaya relies on community participation to provide information, hence it must be easy to use. This research aims to evaluate the usability of Website Wadaya based on ISO 9241-11. The usability attributes consist of effectiveness, efficiency and user satisfaction. The effectiveness attribute is measured by the accuracy and completeness of the users to achieve specified goals. The efficiency attribute is measured by resources spent by user in order to ensure accurate and complete achievement of the goals especially based on expert completes on the scenario. The user satisfaction attribute is measured based on the comfortability and acceptability of use by using Software Usability Test (SUS) questionnaire. This research is divided into three steps, namely: usability evaluation on existing system, development of wireframe as alternative design, and usability evaluation on alternative design. Data collection was done by distributing questionnaire and conducting observation based on the scenario task. The number of participants are 10 users who consist of expert users and first time users. The result of this research shows that there are some usability issues on Wadaya especially when the users attempt to complete contribute cultural object task. The alternative design is developed based on the result of usability testing and Shneiderman’s Eight Golden Rules. The result of usability testing on alternative design shows that the usability of Wadaya has been improved.


Introduction
Wadaya is an information system aimed at documenting cultural heritage that relies on community participation or often referred to as crowdsourcing. In Wadaya, people who want to inform a cultural heritage must log in first, then input cultural heritage information consisting of images and location maps. The cultural heritage then published and can be seen by the entire community. Validation is also done by crowdsourcing by reporting inappropriate content.
The social networking concept is applied to Wadaya to make people more interested in studying cultural heritage. People can give ratings, comments, share them to social networks and report if there is inappropriate content. In order to make users feel closer to the system, users can also add other versions of existing cultural objects. For example, there is already a cultural object of Tari Pendet in the system. Other users who have other images of pendet dance can add it to the system, so that the content in Wadaya is increasingly diverse.
The challenge of the system that is used by the wide community in crowdsourcing and social media concepts is that it must be usable. Usability is the quality level of a system so that it is easy to use so that users can achieve their goals in using the system. Systems that have good usability ensure users are happy and will return to using the system. Usability has five attributes such as learnability, efficiency, memorability, errors and satisfactions [1]. This attribute must have by system. Whereas according to ISO 9241-11, usability consists of attributes of effectiveness, efficiency and user satisfaction. According to [2], learnability, memorability and errors are included in effectiveness, efficiency and satisfaction.
This study focuses on usability testing on Wadaya website. Usability testing used the ISO 9241-11 standard which consists of effectiveness, efficiency and user satisfaction. Effectiveness shows the success level of users in completing their tasks. Efficiency shows the level of usage of the user's time in completing the task. While user satisfaction shows user satisfaction in using the website. Effectiveness and efficiency were measured based on the task scenario given to the user. The success rate and time of completion of the task were recorded. While user satisfaction was measured by the SUS questionnaire. The usability test results were used as a reference to improve the website. Recommendations were given in the wireframe form based on user input in usability testing and based on the Shneiderman's concept of the Eight Golden Rules.

Wadaya
Wadaya is web and mobile based application to collect and spread cultural heritage information in Indonesia, as can be seen in Figure 1. It has two main feature such as Collect Data and Information View. The users can search, view, report and give feedback to cultural object on Wadaya. On the homepage, Wadaya provides search form, the newest object, and information about data. Users can search information of cultural object by entering keyword on search form. Then, users can view detail of object. On detail object, users can give feedback comment, rate, or report the Wadaya object if the object is inappropriate.

Figure 1. Homepage of Website Wadaya
Wadaya use crowdsourcing concept to collect data from users. Users are expected to be voluntary contribute cultural object information to Wadaya. Users can input cultural objects through the Dashboard Menu (Figure 2), then proceed with filling in the input form ( Figure 3). Information of cultural heritage object can be in multi format such as text, image, or photo. Information of cultural object that is contributed by user will directly appear on the system. To avoid inappropriate information by users, another user can report cultural object to administrator. The administrator will review user report and hide the object if so. Users can login using social media account (Facebook or Google) or login manually with username and password.

Usability
The ISO 9241-11 standard defines usability as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" [3]. Based on this definition, ISO defines usability has 3 attributes include effectiveness, efficiency and satisfaction. Effectiveness refers to the accuracy and completeness of the users to achieve specified goals. Efficiency based on the resources expended when users achieve goals, and satisfaction is the comfortability and acceptability of use.
Another version of usability attributes defined by [1] consist of efficiency, satisfaction, learnability, and errors. Learnability is the ability of system to be easy to learn so user can immediately use the system without reading manual book first. Memorability refers to the system must be easy to learn so when the user return to system after some period of not having used it, the user should not learn everything from scratch. The system should have a low error rate, so that is expected to reduce user errors.
In ISO standard, learnability, memorability and errors are not included as the attributes of usability. It argued that these attributes included implicitly in effectiveness, efficiency and satisfaction, like error rates can be argued to have a direct effect on efficiency [2]. In this research, we used the attribute standard from ISO 9241-11 to evaluate usability of Wadaya. Usability metrics from each attribute defined as: 1. Effectiveness can be measured based on completion rate of the users when achieved specified goals of the task. Completion rate is percentage of the number of task completed successfully with total of task undertaken. According to a study carried out by [4], the average Task Completion Rate is 78% (based on an analysis of 1,100 tasks). 2. Efficiency refers to the resources expended when users achieve goals. Efficiency can be measured by Time-Based Efficiency, Overall Relative Efficiency and Expert Relative Efficiency. Overall Relative Efficiency is the overall relative efficiency uses the ratio of the time taken by the users who successfully completed the task in relation to the total time taken by all users. 3. Satisfaction refers to the comfortability and acceptability of use. Satisfaction can be measured by SUS (System Usability Scale), CSUQ (Computer System Usability Questionnaire,) QUIS (Questionnaire for User Interaction Satisfaction) and SUMI (Software Usability Measurement Inventory). SUS is a reliable tool for measuring the usability that consist of 10 item questionnaires with five response options for respondents. Some benefits of using a SUS are very easy scale to administer to participants, can be used just small sample and valid [5]. SUS consists of (1) I think that I would like to use this system frequently, (2) I found the system unnecessarily complex, (3) I thought the system was easy to use, (4) I think that I would need the support of a technical person to be able to use this system, (5) I found the various functions in this system were well integrated, (6) I thought there was too much inconsistency in this system, (7) I would imagine that most people would learn to use this system very quickly, (8) I found the system very cumbersome to use, (9) I felt very confident using the system, (10) I needed to learn a lot of things before I could get going with this system. According to [6], "a method of usability evaluation is a procedure composed by a series of welldefined activities for data recollection related to end user's interaction with a software product and/or how a specific feature of this software product contributes in achieving a certain degree of usability". Usability evaluation classified in two categories consists of usability inspections and usability test. Usability inspection just done by evaluators (usually experts), not involving participations of end users. In Usability tests, the evaluations involving real users. In this research we evaluate usability evaluations by usability test that involving end users to evaluate website Wadaya.

Methodology
This research methodology is divided into three phases, as can be seen in Figure 4. First, we evaluate the usability of existing system, then we develop wireframe as alternative design and at last we evaluate alternative design to get usability results.

The Usability Evaluation of Existing System
This phase aims to measure usability of Wadaya. The attributes used are effectiveness, efficiency and satisfaction. Effectiveness and efficiency are measured based on task scenario. The task scenario consists of seven activities that users must achieved. The task scenario consists of : 1. Please login to Wadaya using one of your social media accounts. 2. What do you do if you want to change your profile picture? 3. You are interested to know one of cultural object in Central Kalimantan. 4. If you think the cultural object is so interesting and you want give feedback (as comment or rating), what will you do? 5. What will you do if you think that this cultural object is not valid? 6. If you know that one of the cultural objects in your province is not in Wadaya yet, what you will do? 7. You are interested to know the number of cultural heritages in each province, what will you do?

The Development of Wireframe as Alternative Design
In the second phase, wireframe was developed as an improvement from the Wadaya website design. Wireframe development based on the evaluation results in the first phase. In addition, recommendations for improvement are also based on Shneiderman's Eight Golden Rules. Wireframe was developed with Mockplus software. This phase produces wireframe as an alternative design for the Wadaya Website.

The Usability Evaluation on Alternative Design
In the third phase, usability of Wireframe which was carried out in the second stage was evaluated. The attributes tested in this phase are effectiveness, efficiency and satisfaction. Evaluation for this phase uses the same tasks and SUS questionnaire. Respondents used remained as many as 10, but 5 respondents were replaced with new respondents who first used the system.

Phase 1: The Usability Evaluation on Alternative Design
Respondents were asked to complete the tasks according to the scenario. The researcher records the tasks that were successfully done and those that failed. The successful task was symbolized by number 1, while the failed task was symbolized by the number 0. In addition, the time needed to complete the task is also noted. The results of the first stage of testing can be seen in the Table 1. n t n*t n t n*t n t n* t n t n* t n t n* t n t n* t n t n* t  R : Respondent n : Completeness of task t : time taken

Effectiveness
Based on Table 1, the effectiveness of each task can be seen in Table 2. Based on the evaluation result, the average effectiveness of the website is 74%. This average value is smaller than the effectiveness standard of 78% so that it can be concluded that in phase 1, the website was still not effective. The task that has a low level of effectiveness was Task 4 and Task 8. In task 4, respondents find it difficult to find a button to give feedback. Most of the respondents think that the rating label on the top of the website was a button. In task 8, user failed to find the Wadaya contribution button. On the website, the contribution button is placed in the profile section. Most of the respondents thought that it was in the contributors menu.

Efficiency
According to Table 3, efficiency of Website Wadaya is 61%. Most of the respondent spent their time in task number 6, when the respondents contributed cultural object. The respondents found difficulty to find contribute cultural object menu, then the respondent also confuse to fill the form due to poor of information about the form.

User Satisfaction
In phase 1 evaluation of user satisfaction using SUS, the user satisfaction level obtained from 10 respondents was 71.5%. This was actually bigger than the SUS standard of 68%. The value of 71.5% means that the system is satisfactory in terms of Usability but must be improvised [7]. The detailed results of SUS evaluation in phase 1 can be seen in Table 3. Based on the analysis of the questionnaire results, according to question number 6, many respondents assumed that the Wadaya system was still inconsistent. This can also be seen in the failure when looking for giving feedback button. Respondents thought that the feedback information label should also be able to provide action when clicked like on a general website. In addition, a small value was also found in question 3,