User experience evaluation of academic progress information systems using retrospective think aloud and user experience questionnaire

SIsKA-NG is the result of advanced development of the thesis management information system at the Computer Science Study Program, Universitas Pendidikan Ganesha. Usability evaluation needs to validate development results based on user requirements. This research aimed to identify problems and user satisfaction when using SIsKA-NG with student access. This research used Retrospective Think Aloud in specifying difficulties or user suggestions and User Experience Questionnaires to obtain user satisfaction scores. An evaluation was carried out on 20 respondents who were selected using simple random sampling from 139 active SIsKA-NG users. Retrospective Think Aloud produced 4 recommendations for improvement of SIsKA-NG. User satisfaction analysis showed that all aspects of the user experience were in the excellent category. The average score for each aspect was as follows: attractiveness aspect was 2,23; perspicuity aspect was 2,30; efficiency aspect was 2,49; dependability aspect was 2,55; stimulation aspect was 2,30; and novelty aspect was 2,35. Future research will focus on the evaluation of SIsKA-NG in all other study programs at Postgraduate program in Universitas Pendidikan Ganesha.


Introduction
The Academic Progress Information System (SIsKA), which has been used since 2016, is proven to assist the Computer Science Study Program managers at the Universitas Pendidikan Ganesha in managing student research data [1] [2]. During its use, two problems were identified, those are inadequate functionality [1] and inappropriate interface [3]. These problems have been resolved and implemented into SIsKA-NG [3]. Previous research concluded that in the future, it is crucial to carry out further user experience evaluation to implement recommended improvement and to evaluate the codes periodically [4].To assess the user experience, further evaluation needs to be conducted to determine its usability [5]. This research focused on user experience evaluation using usability evaluation [4]- [6].
Usability evaluation focuses on evaluating how well users can learn and use the system while maintaining user satisfaction [7]. Evaluation involving users is advantageous due to the direct feedback from users as they use the system [7]. Usability evaluation is categorized into inspection, testing, and inquiry [8]. Inquiry and testing methods involve users in the evaluation process [8]. Therefore, this research requires both methods.
Retrospective Think Aloud is an evaluation technique of a testing method that involves the user's continuous feedback in using the system [9]. This technique has five advantages, including affordable, robust, flexible, convincing, and easy to use. However, the main advantage of this technique is that it allows the evaluator to find what the users think about the system during the evaluation process [9], [10]. The output of this technique is a verbal protocol of the problems perceived by users [10]. RTA is carried out when the respondent has finished interacting with the system [11].
In addition of seeking user suggestions for system improvement, it is necessary to evaluate user satisfaction using the inquiry method. The questionnaire technique is one of the inquiry methods that measure the level of user satisfaction [8]. User Experience Questionnaire (UEQ) is a questionnaire that provides an overview of user satisfaction's level based on user experience [12]. The User Experience Questionnaire (UEQ) has six aspects of the user experience with 26 statement items [13]. These aspects are attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty [14]. UEQ has been tested in providing an overview of user satisfaction in several test cases [14]. It usually takes 3-5 minutes to read and fill in the UEQ [12]. Another advantage of UEQ is that it can be accessed for free [15] and is available in the Indonesian version [12]. UEQ data analysis was performed using the UEQ Data Analysis Tool by comparing the value of each aspect with the product data set [16].
Based on the explanation above, this research conducted a user experience evaluation of SIsKA-NG using usability evaluation. Usability evaluation was carried out using RestrospectiveThink Aloud and User Experience Questionnaire. This research produced a list of recommendations for improvements, user satisfaction scores, and the implementation of SIsKA-NG improvements.

Research Method
The research method is shown in Figure 1. The initial stage of the research started with problem identification, namely the evaluation of the SIsKA-NG that has been developed [4]. A literature study was conducted to determine the methodology used in this research. Based on the results of a literature study, this research used Retrospective Think Aloud (RTA) and User Experience Questionnaire (UEQ). RTA evaluates the results of the assigned task and UEQ will evaluate the value of user satisfaction based on the results of the questionnaire.
In evaluating user experience, using 20-30 respondents provided good and stable results [3] [16]. This research used 20 respondents who were randomly selected from 139 active users using simple random sampling [17]. These 20 respondents were involved in the evaluation of RTA and UEQ conducted in this research. The steps taken in collecting data using RTA are as follows.
a. Before starting the playback of video data, the respondents were asked to clearly state their thoughts when completing the tasked scenario. b. Then the video was played again and the researcher recorded any difficulties or suggestions submitted by the users In the RTA, data analysis was carried out by identifying the difficulties or suggestions and then summarizing them into formal language [18]. In this research, 21 tasks were used to evaluate SIsKA-NG. The example of the task scenario can be seen in Table 1. Please upload a research proposal, with the following data: Please access the exam schedule menu and see the details of your exam schedule. 3 Please upload the revised data with the following data:  User satisfaction data were collected using a User Experience Questionnaire (UEQ). In UEQ, there were six aspects of user experience with 26 statement items that must be filled in by respondents. The aspects of the user experience were attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty [12]. The statement on each UEQ item consists of a pair of terms with opposite meanings (eg good -bad). Respondents could express their impression to the statement by choosing a circle that was closest to it [15]. The questionnaire in this research was added with additional questions to make it easier for respondents to fill out the questionnaire, as shown in Figure 2 [19].
The results of the questionnaire were entered into the UEQ Data Analysis Tool to analyze user satisfaction [15]. In the UEQ Data Analysis Tool, a benchmark test was carried out by comparing the value of each aspect with the existing data set [15]. The benchmark testing illustrated the relative quality of SIsKA-NG compared to other products. The benchmark ranges in each category are shown in Table  2 [16].  The results of the analysis from the evaluation of the RTA and UEQ were compiled into a final conclusion, namely a list of recommendations for improving SIsKA-NG. The final stage of this research was to implement the recommendations for SIsKA-NG's improvement.

Result and Discussion
The results of further evaluation of SIsKA-NG using the RTA technique show that there are still some difficulties and suggestions for improvement from the user. Recommendations for further improvement of SIsKA-NG were focused on the perspicuity of menu names and menu grouping. The save button was needed to be improved by adding a button at the top of the page. The data storage process was needed to be adjusted, so that it display the main menu page after data storing. Regarding the aesthetics of a process, it was necessary to add a loading animation to each data management process. Difficulties and suggestions when using SIsKA-NG are shown in Table 3.  Table 3. Recapitulation of Difficulties and Suggestionson SIsKA-NG.

R01
There is no difficulty. There are no suggestions.

R02
The save button takes additional time to access because it is at the bottom of the page.
Add a save button at the top of the page.

R03
There is no difficulty. The save button should be at the top of the page.

R04
There is no difficulty. Add loading process animation while waiting for saving data.

R05
There is no difficulty. There are no suggestions.

R06
There is no difficulty. There are no suggestions.

R07
The names of the submission and proposal menus are somewhat similar.
Set the menu name so it's easy to distinguish.

R08
There is no difficulty. There are no suggestions.

R09
There is no difficulty. Add loading animation while data is being processed.

R10
There is no difficulty. There are no suggestions.

R11
The save button at the bottom of the page is not efficient.
Add a save button at the top of the page.

R12
There is no difficulty. There are no suggestions.

R13
There is no difficulty. There are no suggestions.

R14
There is no difficulty. There are no suggestions.

R15
There is no difficulty. There are no suggestions.

R16
There is no difficulty. There are no suggestions.

R17
There is no difficulty. There are no suggestions.

R18
There is no difficulty. There are no suggestions.

R19
There is no difficulty. There are no suggestions.

R20
There is no difficulty. After the data is saved, system must reload to the main menu.
The recapitulation value of each statement item that has been filled in by the respondent was processed using the UEQ Data Analysis Tool. The data processing stage began with transforming the initial data, then continued to look for the average value of each aspect of the user experience. The average score for each aspect is shown in Table 4.  Table 4 shows the average score of the six aspects of the user experience. The confidence interval was an estimate of population parameters based on the value of the sample data interval. The lowest value of the interval estimate was obtained from the average value minus the confidence value. Then, the highest value for the interval estimate was obtained by adding the average value with the confidence value [16]. UEQ results were also compared against benchmarks. In the UEQ Data Analysis Tool, the average value of each aspect linked to the existing data set on the benchmark. The results of the comparison are shown in Table 5.  Based on the data in Table 5, it is found that the category is excellent in all aspects of the user experience. The excellent category indicates that SIsKA-NG is in the best 10% products compared to the UEQ benchmark [16]. Previous SIsKA evaluation with six aspects of user experience was assessed with the UEQ by score 1.95, 1.63, 2.13, 2.00, 2.04, and 1.59 [4]. The UEQ result of SIsKA-NG as shown in Table 5, was considered as a significant improvement compared to the previous UEQ result of SIsKA [3], [4]. Especially in the aspect of perspicuity, there has been an improvement from the results of benchmark comparisons from good to excellent. Even though the user satisfaction score was already at an excellent level, the results of the RTA evaluation showed that there were several recommendations for improvements. Based on Table 3, the implementation process was then carried out. Some of the improvements that have been implemented can be seen in Figure 3. Based on that figure, there is an improvement in the layout of the save button which can be accessed from any position, a change in the name of the proposal submission menu, and also the implementation of the loading animation.

Conclusion
The results of the SIsKA-NG usability evaluation showed that a list of existing problems was obtained based on users' feedback by using Retrospective Think Aloud and users' experience score obtained from the User Experience Questionnaire. The results of the analysis showed that there were 4 recommendations for improving SIsKA-NG, namely perspicuity of menu, positioning of the save button, data storage process, and aesthetics of a process. The results of the user satisfaction analysis showed that all aspects of the user experience were in the excellent category. The excellent category indicated that SIsKA-NG is in the best 10% products compared on the dataset in UEQ Data Analysis Tool. The average score for each aspect was as follows: attractiveness aspect was 2.23; perspicuity aspect was 2.30; efficiency aspect was 2.49; dependability aspect was 2.55; stimulation aspect was 2.30; and novelty aspect was 2.35. Further research can be carried out by evaluating the use of SIsKA-NG in other study programs at Postgraduate faculty of Universitas Pendidikan Ganesha.