Evaluation of a soft skills serious game educational methodology in an industrial branches technical training program

The technical training of the industrial branches sometimes shows important shortcomings of new graduates in training and practice in analytical skills, high-level critical thinking, communication skills, teamwork and understanding of engineering in business practice. Gamification, understood as the application of design elements and game principles in the context of achieving learning outcomes, acquires a remarkable dimension and relevance. In this context, this job is focused on designing, developing, and measuring the results of a training module including gamification. This module is taught in an experimental training course aimed at a group with a heterogeneous educational level whose all its members play the role of team leader in manufacturing industries. The course is promoted by the Aragon (Spain) Government, and the Aragon Automotive Cluster Association, Caar. This work carries out a methodology and results evaluation project for soft skills problem solving and conflict resolution module, through gamification. For this evaluation, a discussion is carried out on the evaluation methods of gamified training. The work allows to obtain significant results on the methodology used in relation to the heterogeneity of the group. The study draws conclusions about the design of the methodology, the validity and efficacy of the educational method, the evaluation model and the relationship between training, experience and educational results.


Introduction
During the 20th century, the structure and methodology of engineering education has remained basically stable. Changes have been limited to the advancement of the technology-based industry of what is now known as the third industrial revolution [1].
Technical training of the industrial branches sometimes shows important shortcomings of new graduates in training and practice in analytical skills, high-level critical thinking, communication skills, teamwork and understanding of engineering in business practice [2], commonly called soft skills.
Other authors add that education in industrial areas has not been able to keep pace with advances in manufacturing technology, nor with the demands of the labour market [3]. There is a real mismatch between the industry demand and the labor market offer. Industry is asking for more and more flexibility with constantly updated competencies and knowledge [4]. Companies have been adopting competency management models to improve the results of their talent onboarding processes. In the case of engineering talent, the strategic development of industrial IOP Publishing doi: 10.1088/1757-899X/1193/1/012128 2 companies is closely linked to the availability of that talent. Consequently, recruitment systems have stopped being based exclusively on academic performance and additionally on tests of technical ability.
The basis of competency management is the competency matrix. It is a matrix that relates sets of competencies with professional profiles in the organization's structure, indicating, for each cell of the matrix, a value that measures the level of demand that the organization assigns to each role for each competency. There are also some standard competencies models, which paradigm can be the McClelland and McBer Competency model [5].
Caar, the Automotive Cluster of Aragon, a region of norther Spain, was founded in 2008. It integrates Aragonese companies that manufacture automotive components, with universities, technology centers, research centers and business and economic development entities in the autonomous community of Aragon. Caar promotes and develops private, public, and mixed initiative programs in innovation, business development, and training. In 2015 the cluster drew up a strategic training plan for the period 2016-2020. In this plan, a two-matrix model of technical and personal competencies was defined, which is attached in the Figure 1. In this model, the participants determined that two profiles are strategic to maintain the operational competitiveness of the plants: • The middle manager is profiled as a higher university graduate in technical careers, particularly engineering, responsible for production areas of the company, or support departments, such as quality, maintenance, or continuous improvement. It uses to correspond with European Qualification Framework (EQF), level 6-7 [6]. • The supervisor is profiled as a graduate in vocational training, (EQF 4-5 level), responsible for one or more cells or lines of production processes. He is also responsible for support tasks teams in the manufacturing processes like quality, maintenance, or continuous improvement.
Personal competencies matrix also includes one skill of high interest for this sector called "management models". This is defined as Lean Management, Six Sigma, 8D, or Problem Solving. Problem solving is a basic feature in continuous improvement and lean management models. Lean management has its formal origin in the work of the Toyoda family and their collaborators in their companies. Taiichi Ohno in the Toyota company developed the Toyota Production System [7]. Ohno adopted at Toyota the methodology initially developed by Walter Shewart of Bell Labs in the 1930s and further developed by W. Edwards Deming. It was also developed and improved by professors from the Japan Union of Scientist and Engineers with their work at the Toyota plants and other Japanese IOP Publishing doi:10.1088/1757-899X/1193/1/012128 3 manufacturers after the Second World War. [8] This methodology receives today the name of its developer: Deming Cycle. It is based on the Plan-Do-Check-Act cycle. And it is implemented in companies using the A3-PDCA methodology, which combines several lean tools in a A3 paper sheet to deploy de Deming cycle.
In both aforementioned profiles, problem solving skill is very important, since they are assigned high level (light green) or very high level (intense green) for each of them, as we can see in Figure 1.
The strategic plan also determined that it was necessary to develop projects to improve the practical training of graduates in technical areas, in collaboration with educational organizations.
To achieve this goal, Caar developed an experimental training course with the Aragon Regional Government with administrative competencies in vocational training, to complete the training of students of higher technological degrees of vocational training (EQF 4), aimed at both recent graduates and active workers with experience in the supervisor/team leader profile [9]. The course is called for practical purposes "Specialization Course in the automotive industry. CEIA ". In the area of personal skills, the course includes in its training objectives and learning outcomes, among others, to propose solutions to problems in the workplace in his role as team leader and to identify problem situations and face them by offering solutions and anticipating consequences.
The course pursues the objective of implementing innovative training methodologies that help solve the problem presented in this work, referring to the results of traditional methodologies.
To achieve these objectives, the realization of a training module that combines traditional skills training methods with a gamification methodology is proposed.
Gamification, understood as the application of design elements and game principles in the context of achieving learning outcomes, acquires a remarkable dimension and relevance. The development of educational methodologies based on gamification is necessary to ensure meaningful training results. Such methodologies should help to reduce skills gaps. Gamification is present in the training of university students in a growing and more relevant way. This is especially significant in engineering and architecture studies. [10] 2. Methodology

Description of the Synchro PDCA game
The game is to reinforce the training on the A3-PDCA methodology received in the three hours theoretical-practical session of the lean management unit in the course. For the educational experiment, we started from the base of the simulation game Synchro from Festo Didactic [11], a company specialized in process automation. Government of Aragon has an installation of Festo game in its facilities at the Innovation Center for Vocational Training of Aragon, CIFPA. CIFPA is a center of training excellence in vocational education and training area, where this course is taught. It is a simulation game of a production plant oriented to training in a set of tools and lean methodologies such as SMED, Kanban, VSM and Kaizen principles. Three rounds of the game were proposed, based on an inefficient production design. The competitive element is introduced by facing several teams in the game and establishing the winning team as the one with the greatest number of correct parts.

Game evaluation
2.2.1. Evaluation model. The general objective of this work is to study differences in the quality and results of education in the group based on educational profile and experience of trainees.
Gresse von Wangenheim et al [12] define the Quality of an educational game as which "has clearly defined educational goals, motivates students to study and promotes the learning of curricular content through activities that are fun, enjoyable and challenging". The objective of the experiment is focused on evaluating differences in the perception motivation, user experience, and its learning capacity, in function of student's profile. For the evaluation of the game, an adaptation of the model proposed by Gresse Von Wangenheim et al. is used. It is based on the Kirkpatrick four levels model [13], which is focused, after a discussion of its validity, in the evaluation of the level one: the way in which the students perceive the experience. This research strategy is based on the students' perceptions through questionnaires after the game application.
Gresse Von Wangenheim et al. discuss the validity of this method to evaluate the effectiveness of training, and he concludes that this method can provide "reliable, valid and useful results", and that it involves "a simple, quick and less intrusive alternative to obtain feedback.

Evaluation measures.
Gresse Von Wangenheim decomposes the three objectives of evaluation, motivation, experience, and learning, in hierarchical subcomponents based on the GQM model, [14] as a general framework. For motivation he uses the ARCS model [15], focused on the dimensions of attention, relevance, confidence and satisfaction. For the experience, it is based on the model of Tullis and Billa, [16] and the contributions of other authors, and focuses on the user's interaction with the game and his companions. He defines for it the dimensions of immersion, challenge, competence, fun and social interaction. The learning subcomponent is measured by taking the contribution of the game to training in the short and long term, based on the Moody and Sindre model [17]. The model finally adopted for the investigation can be seen in figure 2.

Hypothesis
The basic hypothesis is that there are notable differences in the perception of how a game contributes to improving the results of training among people with different educational levels and professional experience.

Research strategy
The research strategy is based on the feasibility of the evaluation objective in the context of the course, which groups heterogeneous students in their training and experience, around a specific professional profile. This is a consequence of the genesis of the course and its objective, very focused on a specific professional profile, the automotive team leader.
We use the case study defined as the application of an educational experiment than gamifies learning to an individual, group or event. After it, an individual survey is carried out, immediately, and through an online form.
The proposed questionnaire is designed by simplifying the Gresse Von Wangenheim proposal [19], since the final objective of this experiment is to measure differences in homogeneous groups, and not so much the evaluation of the quality itself. For simplifying, eighteen items are selected from the original IOP Publishing doi:10.1088/1757-899X/1193/1/012128 5 twenty-seven, divided in three subscales and eleven dimensions, as discussed previously. The Likert scale is used, which evaluates between -2 (strongly disagree) and +2 (strongly agree).
The questionnaire is anonymous, inclusive, and the subsequent treatment of the results is guaranteed by personal data protection laws.

Game application
The course was planned in the 2020-2021 professional training program of the department of education of the government of Aragon from September 2020 to June 2021, with the specialty code PEE01. Sixteen students have enrolled in the course. Some students have enrolled on their own initiative, either directly from the educational system, or from some companies. Others have been enrolled by indication from their companies. These profiles have been classified by degree (basic studies, VET, University); and for years of experience in the company (No experience. Less than 6 years, 6 or more). The group composition can be seen in Table 1. The game took place on January 12, 2021 (figure 3)  The cluster and the School of Engineering and Architecture at the University of Zaragoza have a collaborative program for research on educational methodology, in particular, gamification through learning factories. This same experiment is expected to be repeated with other training groups from both the school courses and the courses offered by Caar Academy, the educational area of the Cluster. In particular, the school and the cluster are currently collaborating in the revision of the official master's degree in Mechanical Engineering. This collaboration has, among other, the objective of improving the results of training in personal skills, through innovative methodologies, focused on learning factory model.
The size of the sample advises not to examine each item evaluated by a question. It is considered better to add the results of the questions referring to each of the items, motivation, experience, and learning. Total results are first compared with those of each of the three levels: basic studies, vocational education training (VET) and university degree (Figure 4a). Secondly, for the three levels of experience: they are unexperienced, less than six years of experience, six years or more (Figure 4b).  In general, the evaluation of the experiment is good. However, motivation has worse results than experience and learning categories. The analysis of this fact will be discussed in the section on results by levels of experience, since it may have its origin in the degree of freedom that each student has had to enroll in the course. Experience is the one that has the best results. The game seems attractive, the students value the experience and have fun, interact, and socialize.

2.6.2.
Results by training levels. The results can be seen in the Figure 4(a). The degree of motivation, paradoxically, increases in relation to the level of training. It is more dispersed in participants with basic training, and more homogeneous for the two upper levels, although it does not increase in a correlated way. In any case, the result suggests that people with more intense experiences in technical training, value the training itself more positively, and therefore, feel more motivated towards this type of experience.
In the area of experience there is much more homogeneity in the results. They practically coincide for the basic level and VET, and they are also positive for the university level. It can be inferred that a different training methodology provides a very attractive experience for students of any educational level. They welcome any project that involves an "out of the box" experience. In the area of learning outcomes, there is a clear inverse correlation between educational level and perceived outcome. An explanation may be the level of the training offered. The course is designed for the VET level. More precisely, for a level that we can denominate VET +, between VET and university. In Spain there is a gap between the maximum VET level (EQF 4) and the minimum university level (EQF 6). This course is designed to reduce that gap. This can cause it to provide a better experience of learning in people without technical training.

Results by experience levels.
Using the same comparison methodology, the results are shown according to the level of experience in Figure 4 A pattern is observed in all the factors under evaluation. In all of them the results worsened according to the level of experience in the position. It is more pronounced at the maximum experience level. The game experience continues to be the element that obtains the best results, and in which the difference by experience level is less than the global one. Depending on sample size, it can be stated that there are no substantial differences. This is not the case with motivation and learning, where the same pattern occurs: the more experience, the lower the level of satisfaction. This can be a consequence of: • Course level. The course is designed for developing the team leader role. It is a complementary VET training course for inexperienced students. The fact that people with up to 20 years of experience participate means that some content and the gamified experience do not arouse the same interest as in recent graduates. This aspect must be taken in account when designing serious games for experienced people. • The enrolment process in the course: as mentioned before, some of the students have voluntarily enrolled in the course. Specifically, all students without experience, and many of those with less experience. Some more experienced students have been enrolled by the companies. This can influence the way they perceive the experiment.

Discussion and future development
In this work we wanted to establish a line of work that allows evaluating the results of gamified training based on the profile of the group that participates in it. It is important to evaluate whether the selection criteria of the participants, based on their professional role, produces homogeneous results. This will allow to improve the design of the course and its target profile. In this first experiment, the sample is very limited, but it can be established that there is a trend of disparate results depending on the qualification, and specifically, on experience. In this group there are people without experience, with some workers who accumulate more than 20 years of professional activity.
During the next few years, the Automotive Cluster will develop new gamified educational experiences. In collaboration with the School of Engineering and Architecture at the University of Zaragoza (EINA) Caar will develop skills training that complements the Master of Mechanical Engineering at the school. This training will include the application of these methodology in its development. Likewise, the possibility of submitting a project to the 2021 call of the Erasmus + 21-27 course is being studied. The project will develop, in collaboration with other European schools and organizations, a structured personal skills model for students of technical. This line of research will contribute significantly to achieve good results in this work.
To improve course evaluation, we must consider the Kirkpatrick's level discussion, and the necessity of measuring in a deeper way the training results in terms of technical skills acquired. The gamified environment for training is conducive to a good experience as demonstrated by the results of this experiment. It is therefore necessary to take advantage of this context to ensure that the training achieves its results, and the students learn the use of the tools developed during the training. This is the main IOP Publishing doi:10.1088/1757-899X/1193/1/012128 8 objective of the training. For this reason, the objective of this line of research is to add tools for evaluating the results of the training that make it possible to objectively measure the degree of training effectiveness. It is important to create effective and engaging games, properly balancing utilitarian and hedonic systems in game, improving training results evaluation tools. And improving game design to create more realistic experiences for the students in the field of manufacturing processes [18].