Automatic Question Generator System Conceptual Model for Mathematic and Geometry Parallel Question Replication

Weaknesses in the paper assessment process have been able to overcome using the Computer Assisted Assessment (CAA) objective question. The weakness that can be overcome is the reduction in time for correction and the process of handling thousands of participants simultaneously. However, the process of preparing objective questions is still constrained by providing parallel questions. Parallel questions are made so that each examinee gets a different question but has the same level of difficulty. Making parallel questions manually requires expensive costs and the consistency of the difficulty level must be the same. Some researchers propose an Automatic Question Generation (AQG) system to overcome these problems. In this study an AQG system will be developed to make parallel questions for the material tested at each school level. The process of previous generating questions that can only detect text and numeric variables, will be developed so that they can detect image variables and can detect geometry types based on numerical variables. The system is expected to work like humans who can make questions when given a text, numeric, image, or mathematical notation. By combining the Stem, Multi Part Parser Algorithm and the Custom Media Type Parser Algorithm in the Python-Django framework, the system can provide question recommendations according to mathematical geometry-based input. The process of generating items in this study will use the concept of manipulation of keywords represented by variables. This variable stores values in the form of text, numeric, and image. The ability to manipulate text, numerics, and images is applied to the developed AQG system so that it can be used to generate questions on the material to be tested. AQG is expected to be able to detect text, numeric, notation, and image input and produce output in the form of geometry-based mathematical problems according to text, numeric, notation, and image input with a better level of accuracy.

methods or patterns. The system is expected to work like humans who can make questions when given a text or graphic. Humans can make questions because humans can understand the text and graphics provided and based on the knowledge they have [3]. AQG system is one of the most important components in learning technology at the knowledge domain stage. At the knowledge domain stage, the AQG system is used as a practice concept that is generated based on visual or text knowledge. The AQG system has developed into an important research because it requires insights from various sciences of Artificial Intelligence (AI), Image Processing, Natural Language Processing (NLP), Machine Learning (ML), etc. [4]. This automatic question generator aims to develop a framework that can be used to generate mathematical and construction geometry questions based on certain inputs, such as geometry, theorems and construction objects. For mathematical sets, algorithms can produce a set of questions and with solutions to those questions. The solution or answer will use the theorem the user wants directly or indirectly. In addition to helping users, this question generating framework has scientific contributions in other research fields, such as Intelligent Tutor System (ITS) and Massive Online Open Courses (MOOC). There are several studies related to mathematical geometry-based AQG, but the framework that is built is still fixed on the triangular flat figure, and is still less adaptive to the number of variations of the problem. The result of previous research haven't been able to process the image variable and automatically detect the type of two-dimentional figures based on the numeric or image variable input. This research is directed to develop and refine a framework that can be used to generate mathematical geometry-based questions based on certain inputs that are more adaptive and varied. In addition this study also aims to improve the level of accuracy and appropriateness of the questions that arise with numeric, text, images, and mathematical notations inputted from previous studies which reach an accuracy level of 70%.

Literature Review
There are various types of objects in the problem generator system automatically, including those based on text, graphics, HTML files, and mathematical notation. Many researchers have carried out the research and are now growing. The automated question generator system that is the most researched is the text based one. Whereas those which are still rarely studied are those based on mathematical notation, geometry and vectors. Especially those that use a combination of stem and parallel items methods. Assessment in learning mathematics is one of the important components in a learning process [5], because it is used to determine achievement in a learning process [6]. The usual process carried out in assessment activities is students are asked to answer questions related to the material that has been given. These questions can be divided into two types namely objective questions and subjective questions. The form of objective questions will include the choice of answers in the set of questions, while subjective questions ask participants to develop their own sentences to answer the questions that have been given [6]. Objective type questions are still the most widely used to conduct assessments on a broad scale [7]. Some researchers claim the superiority of objective questions is being able to provide accurate and consistent assessments [8]. The process of correcting this type of question is also relatively easy. One drawback of this type of question is that it costs a lot of money to prepare a test kit. Making parallel questions by involving humans does not guarantee consistency of the equivalent level of difficulty [7]. Some researchers have tried to develop Automatic Question Generation (AQG) as a solution to the problem of making objective parallel type problems. Gierl [9] defines AQG as a process used to generate test items. The generation process is carried out with a computer model and technology. Aside from being a question item generator, AQG is also intended so that the resulting questions have the same level of difficulty [9]. There are several terminologies known in AQG, namely stem, variable, and option. The term stem according to some previous researchers [9][10] is defined as a model (template) of a problem. Stem has two parts, namely variables and statements. Variables are words or phrases that will change with each question, while statements are sentences supporting the questions. The processing of AQG components (stem, variable, choice) by involving computer technology will produce a process of generating as many questions as desired. The number of questions that can be raised is the result of the manipulation of the stem and its variables. The items raised are also accompanied by the generation of key answers and distractors (deceptive answers). The process of determining the key answers and the factors are influenced by changes in the variables attached to the questions raised. Some of the AQG research that has been done can be grouped into two, namely research on the AQG framework [9][10] [11] [12][13] [14] and research on the development of AQG systems aimed at raising items on certain material [7][9][15] [16]. Through the study of previous studies it was found that the AQG framework can be divided into two, namely the generation of items automatically and semiautomatically. Frameworks with automated schemes use rules-based concepts such as the application of ontology methods as a medium to generate questions. Meanwhile in the semi-automatic framework, it uses the help of a material expert to form the stem first. Through the stem at the same time determined the level of difficulty and variables that will be manipulated by the computer. Two AQG frameworks with automatic and semi-automatic schemes have advantages and disadvantages. The advantage of the automatic AQG framework that applies the rules based concept is that it can be applied to adaptive learning systems to form questions with different levels of difficulty according to the ability of students during the learning process with an online system [10]. While the weakness is not easy to apply to different materials, because they have to build rules in accordance with the output to be achieved on the material. Meanwhile, the semi-automatic AQG framework has the advantage of being easily adaptable because this framework only needs to make the stem accompanied by variables to be used as a differentiator between items. The weakness is that each will create problems with different levels of difficulty must create a new stem [9]. Then a recent study related to Automatic Question Generation was conducted by Feddy S. et al. [17]. AQG was developed using the basis of Computer Assisted Assessment (CAA) with the Parallel Mathematic Items method. The AQG system developed in the study is used to make parallel questions for mathematical problems. The parallel nature of the questions raised is possible because the AQG System has 3 components, namely stem, variables and options that can be dynamically changed by mapping, combination and permutation. The result is that the system is able to generate 70% of the question variants from a question dataset that has been tried. This research will develop an AQG system that is able to be used to generate multiple choice parallel questions on the material tested in schools. Every subject in the school, whether elementary, junior high or high school has a variety of materials and each level of school has a different level of difficulty. AQG which will be developed in this study will adopt a semi-automatic AQG framework. The development of AQG as one of the exam tools was developed as an effort to enrich the questions that were tested on the material in schools. The questions raised will be parallel or equivalent. The process of making question equality with a sufficient number of variations to facilitate the questions on the school exam is a problem that will be solved through this research.

Methods
The development of the Automatic Question Generation (AQG) system that was developed based on the model proposed by Mark et al. [18]. and Susan et al. [19]. In essence, the AQG system has four main components: 1. Stem / Template / Question Model 2. Mathematical Equations 3. Variable (text, numeric, image) 4. Algorithms for manipulating Stem (Multi Part Parser Algorithm and the Custom Media Type Parser Algorithm) The dataset used to measure the level of accuracy of the AQG system is a dataset of National Examinations for Junior and Senior High School mathematics subjects. AQG system flowchart in this study is shown in Figure 1  The main components of the model are statements and variables. Variables are components that will be manipulated to generate items that are parallel using a computer program. Figure 4 also shows that the AQG system developed in this study will be able to process two types of questions namely essays and multiple choice. Generating the type of essay questions will produce items accompanied by 1 key answer. Meanwhile, multiple choice questions will be equipped with 4 answer choices consisting of key answers and deceptive answers. The development of the AQG system in this study will use mathematical problems as test data. Selection of mathematical problems as test data because it is able to represent the form of questions in other materials, such as involving simple calculations, complex calculations, and questions that only involve text. Through these problems, the developed AQG is expected to be able to generate items that require calculations and which do not require calculations. Figure 2 shows the process of mapping a variable into processing functions. The schematic variable processing function is shown in Figure 2 below:  Figure 2 explains that the variables included in a stem will be divided into 3: the variables to be processed with a scientific calculator, the variables to be processed as symbolic calculations, and the variables to be processed by a text processor. This process can be done using a programming language editor. In other words, the system developed is able to behave like a programming language editor that is very flexible for users. For manipulating image variables, Multi Part Parser Algorithm is used. This algorithm is parsing multipart HTML form content, which supports file uploads. Request data will be filled with Query Dict as below:  Tables 1 and 2 an example of a stem flow will be displayed producing problems and analysis in the Geometry and Trigonometry chapters.   Table 1 is a stem table where the subject is text and mathematical notation, while Table 2 is a stem table where the subject and variables to be manipulated are text, images, and mathematical notation. There are various combinations of formulas that will serve as input to produce various question outputs and answer options. One formula will produce the correct option, another formula will be deceptive in the answer options that appear. AQG system developed is a web-based system using the Python and Django programming languages. Python is a programming language that has many libraries such as NLTK, NUMPY, and SYMPY which are very powerful in manipulating text and numbers. The three libraries are very helpful in developing AQG for mathematical subjects.
The system developed will then be validated using a review process by stakeholders and material experts. This review process will determine that the system developed is suitable for use. After passing    Figure 3. m the system validation session further trials will be carried out on a limited and broad scale. Testing at this stage is carried out to find out about the readiness of the AQG system to be used by users (teachers) in creating questions to be tested.

Result
AQG can detect two-dimentional figure types automatically according to text and numeric variable input variable input  Figure 5 explains the automatic input of image variables. Image variables will be generated according to the name of the image that is called as a variable. Multiple choices consisting of 1 correct answer formula and 3 deceptive answer formulas are also inputted as shown in Figure 6. A list of questions in the form of mapping or a combination of generating text, image, and numeric variables has been formed as shown in Figure 7. Figure 8 is a *.doc file display of the results of export items that can be used as feeders to larger applications. Table 3 show result of AQG system comprasion tests on math National Exam.  Table 3 shows that of the 40 items tested on our research, 31 questions can use the stem on AQG to generate parallel questions. While 9 other question cannot be generated using AQG. That means our research has an accurary rate of 77.5%, better than previous research with an accurary rate of 70%.

Conclusion and Discussion
About 77.5% of the mathematical datasets tested, proved to be generated by the AQG system development. Significant improvements were found in mathematical items with geometry and trigonometry types because the majority of the questions used image variables. This is supported by the algorithm applied to the stem, the Multi-Part Parser Algorithm and the Custom Media Type Parser Algorithm. These two algorithms are proven to have an important role in the manipulation of image variables and the detection of image types in two-dimentional figures.
This research is an improvement from previous research and has shown improved results. But this research has many limitations, one of which is a limited image variable for one variable per stem. This system is still limited to the use of schools in Indonesia so it's still limited to language support. It also needs to increase the level of flexibility and more dynamically, so that more complex and complicated math items can be generated.

Acknowledgement
This study was supported by the Ministry of Communication and Informatics of Republic Indonesia scholarships.