Comparing Simulations to Improve Physics Students’ Education

Deepening the authentic inquiry process framework, we analysed and compared some simulations used in physics teaching/learning activities and covering different topics in secondary schools’ curricula. The analysis focused on cognitive processes activated by exploring the simulation and using the proposed material for teachers/learners. While evaluating the inquiry tasks in the simulations analysed, we recognised some features that could become a starting point for identifying simulation patterns targeting learning outcomes and scientific abilities. But more interestingly, we tried to focus on which could improve physics students’ education in an epistemologically authentic inquiry process. With the performed analysis and the collected data, we chose some simulations that better fulfilled the inquiry goal. Then, we tried to develop teaching/learning materials based on the ISLE (Investigative Science Learning Environment) approach. We adopted this framework because it is an example of epistemologically authentic inquiry. Lastly, we shared the results of our analysis and the developed materials with a community of in-service physics teachers to collect their feedback and reflections on this use of simulations.


Introduction and Theoretical Framework
Simulations have become useful and well-known tools in today's Physics Education [1,2,3,4].They paved the way to explain complex phenomena illustratively and vividly [4].Simulations provide interactive experiences and help teachers explain and deepen theoretical knowledge about certain physics topics [4,5].Also, real-world scenarios can be observed in a safe and user-friendly environment [3].The simulations create visually compelling and physically precise representations of fundamental physics principles.Additionally, they are intentionally crafted to establish clear connections between students' everyday perceptions of the world and the foundational principles of physics, often by rendering these physical models visible [1,4].This helps students to gain knowledge not only about the phenomena itself but also about the way scientists work and observe experiments [2].
Furthermore, conceptual simulations foster critical thinking among students, facilitating effective learning [4,6].Within these simulations, students can modify variables to observe the outcomes, enhancing their comprehension as they continue to experiment with these variables.Computer-based simulations provide students with hands-on experiences to apply their knowledge, enhancing their critical thinking and higher-order cognitive abilities [6,7,5,8].For this reason, computer-based simulations and inquiry-based learning [9] are promising for a conceptual change in science education [10,11].Those kinds of simulations could be wellidentified by some featured components: • embracing educational scenario which promotes investigative learning environment; • creating an immersive, realistic scenarios which replicates real-world phenomena [2]; • activating authentic inquiry through multi-levels and multitasking activities [11]; • promoting working team-group helping students to learn how to collaborate and communicate making science [10,11].
Therefore, it is possible to recognise in inquiry-based simulations some components featuring these activities (Table 1) for successful learning outcomes: Although numerous inquiry-based simulations are available for educational purposes, many require students to complete scientific inquiry tasks that do not reflect the core aspects of authentic scientific reasoning from a cognitive and epistemological point of view.The cognitive processes activated for using these simulations often differ significantly from those necessary for scientific activities.Indeed, the underlying epistemology of these inquiry tasks works antithetically against the principles of authentic science.Noticing this discrepancy, we were guided to formulate two research questions: (i) To what extent do inquiry-based simulations resemble authentic scientific inquiry?(ii) How could we improve teaching/learning materials for inquiry-based simulations which resemble authentic scientific inquiry?
As reference frameworks, we referred to the work of Chinn & Malhorta [12] to answer the first research question.They identified a theoretical framework for evaluating educational inquiry tasks' similarity to authentic science.They defined authentic scientific inquiry as the process which encompasses the actual research conducted by scientists, involving complex activity with expensive equipment, sophisticated methodologies, highly specialised knowledge, and advanced approaches for data analysis and modelling [12]: The cognitive models that underlie authentic experiments are fundamentally different from the cognitive models that underlie simple experiments, and the differences in models help account for why there are differences in cognitive processes and epistemology.
In school activities analysed, they found there are mainly three prominent types of school inquiry tasks, collectively called simple inquiry tasks: The simple inquiry tasks differ from authentic ones: as we move towards authentic procedures, there is an increase in the activation of cognitive processes involved in reasoning tasks.Therefore, utilising Chinn & Malhorta categorisation of these inquiry typologies, we employed this framework to analyse inquiry-based simulations encompassing various physics topics, comparing among different existing ones.The main aspects of the cognitive processes investigated are the following: • generating a research question; • designing a study to address the research question; • making observations; Table 1.Main components featuring inquiry-based simulations and their relation with learning goals [2].

Features of inquiry-based simulations
Learning Outcomes (students should)

Practical Application
• Bridge the gap between theory and practical application.

Engagement and Motivation
• Be active learners in an interactive learning environment.• Be actively engaged in the learning process.• Encourage to investigate simulations independently.• Feel motivated and fully engaged in this learning environment.

Transferable Skills
• Be able to transfer learned techniques, skills, and knowledge.• Apply learned concepts to solve specific tasks.• Be able to utilize their knowledge across diverse fields.• Apply acquired knowledge to address problems in various domains.
For each of them, well-defined cognitive sub-processes delineate the tasks scientists and students undertake.These categories referenced our comparison to answer the first research question.
For the second research question, we chose for each topic the inquiry-based simulation with the higher score towards authenticity (obtained in the previous analysis), and we used the ISLE (Investigative Learning Science Environment) [13] approach as the reference framework for the development of new teaching/learning materials to empower its use towards a more authentic practice.The ISLE process promotes and activates authentic inquiry cognitive tasks [14], ensuring we try to increase the inquiry potential in the existing simulations and the activation of scientific abilities [15].

Method
To effectively address the first research question, we employed a comparative research design [16], which enabled us to meticulously examine and contrast the inquiry cognitive aspects of the web simulations we explored across seven distinct physics topics: energy, dynamics, force, geometric optics, heat thermodynamics, charge, DC circuits, and magnetism.This methodology allowed us to unveil the nuanced cognitive processes engaged by students during their interactions with these web simulations, providing valuable insights into their effectiveness in fostering inquirybased learning.As outlined in curriculum instruction features [17], these seven topics represent the most significant ones in middle and high school education, making them crucial for developing students' scientific literacy and problem-solving skills.To improve the teaching/learning materials (addressing the second research question), we designed new ones ISLE-based for those simulations with high ranking scores in the cognitive processes activated (as described in the following Implementation subsection).

Sample
The sample for the comparison consisted of fifteen simulations, chosen among a huge variety of ones under the following conditions: they should have been developed by a known institution with accredited experience in Physics Education Research and proven tested/used by researchers, teachers and students for different contexts, curricula and instruction.Chosen simulation details are provided in Table 2.
We compared two simulations for each topic, except for the energy-related topics, for which we examined three.We did not find three simulations for each topic because the ones we faced did not satisfy the conditions required for our sample.

Data Collection
Before analysing each simulation from the cognitive processes activated -according to the Chinn and Malhorta scheme [12]-, it was necessary to get a deep overview of all the simulations, including the teaching/learning materials related (only the ones created by simulation authors).
The ranking scale in Table 3 is designed to assess the degree to which a simulation aligns with authentic inquiry.Authentic inquiry simulations should provide students with opportunities to manipulate variables, collect data, and analyse results to draw their own conclusions.Simple experiments simulations provide students with some opportunities to manipulate variables and observe results, but they may not provide as much freedom for experimentation and analysis.Simple observations simulations provide students with opportunities to observe phenomena, but they may not allow for manipulation of variables or data collection.Simple illustrations simulations provide visual representations of concepts, but they may not allow for any interaction or experimentation.Simulations that are not inquiry-based do not provide any opportunities for students to engage in scientific inquiry.
We investigated each simulation and its related teaching/learning material based on the main aspects of the cognitive processes activated.Then, we compared the collected data into tables highlighting the kind/level of inquiry they showed.We used a coloured scale for ranking how closely each simulation aligns with authentic inquiry, with green indicating a closer alignment and red indicating limited authenticity.We assigned each voice a score for data analysis (Table 3).A shortened example of the data collected for the simulation and teaching/learning material could be found in Tables 4 and 5.

Data Analysis
Each simulation was evaluated by a panel of three experts in physics education.The evaluators used a rubric based on the Chinn and Malhorta scheme [12] to assess the simulations' alignment with authentic inquiry.The evaluators discussed their ratings and reached a consensus on the score for each simulation.
Every chosen simulation was reviewed for every topic to confirm which fits closer to the most authentic inquiry.This review process also happened after every analysis of every simulation and all the teaching material.To quantitatively measure how close each simulation got to authentic inquiry, we calculated the mean value of all the cognitive processes investigated using the ranking score of the coloured scale (Table 3).Then, for each topic, we compared the mean value to choose which simulation had the higher level of inquiry.

Implementation
Addressing the second research question, the selected simulations were the ones to undergo enhancements within the ISLE framework.To design a new teaching/learning material, firstly, we deeply analysed the ISLE materials [18], then focused on the cognitive processes activated in the ISLE activities.Secondly, we designed new activities aligned with the ISLE process [13] and targeted the development of scientific abilities associated with specific "verbs of process" in the materials themselves.We identified the main ones in observing, describing, drawing, applying, finding patterns, testing, adjusting, and generalising.These verbs were incorporated into the new materials and thoroughly analysed through the Chinn and Malhorta framework.Lastly, we administered the teaching/learning materials to a group of physics teachers, experts in the ISLE approach adoption in their classroom activities.All the teaching/learning materials are available online1 .

Results
Table 6 presents the mean scores for the cognitive processes activated by each simulation, considering both the simulation itself and the associated teaching/learning materials.The mean scores represent the average activation of each cognitive process across all the analysed

Observer bias does not play a role
Observer bias does not play a role simulations for a given topic.The selected simulation is closest to authentic inquiry and has been chosen for developing new teaching/learning material.
Indeed, figure 1 compares the simulation overview analysis results with the corresponding teaching/learning materials.
Another interesting result is depicted in Table 7.This table presents the mean scores for the cognitive processes activated by each topic's teaching/learning materials.The "Existing material" column represents the mean score for the existing teaching/learning materials, while the "Newly created material" column represents the mean score for the new teaching/learning materials developed using the ISLE framework.The "Incremental Percentage value, existing-new" column shows the percentage increase in the mean score value from the existing materials to the new materials.For example, in the "Energy" topic, the incremental percentage value is 16.25%, indicating that the new teaching/learning materials activated cognitive processes to a greater extent than the existing materials.
Since the new materials are all prepared in a way as close as possible to authentic inquiry, they all have the same mean value, which is 2.86 ± 0.53 (not all the cognitive processes of authentic inquiry can be activated with these simulations in classroom practices).

Discussion and Conclusions
We analysed some inquiry-based simulations by focusing on the cognitive processes activated in their use through the Chinn and Malhorta reference framework [12] compared to authentic inquiry.
We essentially discovered that in various physical topics, most of the simulations analysed promote cognitive inquiry processes, which closely resemble those associated with simple observations.This holds true in both a general simulation overview and the teaching/learning materials we examined.We also noticed, as shown in Figure 1, that a higher score mean value in the simulation does not mean the same one in the teaching/learning materials.
For this reason, in order to design a new teaching/learning material with higher inquiry potential toward authentic practices, we chose the simulation with the higher score even if its teaching/learning material was not the same.Developing this new material, we achieved an improvement in all the considered physics topics (Table 7).Therefore, the ISLE framework used for the design of new materials satisfies the condition to enact students' tasks, mirroring what physicists do [13].Moreover, we suggest the conducted analysis could be useful either for researchers and teachers.Researchers could take support by analysing their products with the lens of cognitive processes activate in order to improve their simulations toward a more authentic inquiry environment.Indeed, when teachers select which simulations to adopt in their classroom activities, they could try to explore which level/type of inquiry is activated using the teaching/learning materials available.
Even if the freely available teaching/learning materials do not closely resemble authentic inquiry, we suggest that creating or designing materials using the ISLE framework and process is a possible way to activate learners' cognitive processes as the ones of scientists in their reasoning tasks.This may enhances all the features of inquiry-based simulations.
The simulations' new teaching/learning materials have been tested on university students and high school teachers, not high school students.The feedback had been very positive even if teachers stressed the need to reduce the length of materials used in classroom practices and timing.

Figure 1 .
Figure 1.Mean value of cognitive processes activated in simulation overview and teaching/learning materials for all topics and simulations investigated.Comparison with the ranking score scale addresses the kind of inquiry achieved (empty bars correspond to the results for the simulations' overview, full bars for the teaching/learning materials).

Table 2 .
Sampled simulations (we extracted the Instruction Level as defined by simulations' developers).

Table 4 .
Inquiry ranking scale for simulation comparison.

Table 5 .
Inquiry ranking scale for teaching/learning material comparison.

Table 6 .
Cognitive processes analysis results through simulation overview.

Table 7 .
Mean scores for the cognitive processes activated by each topic's teaching/learning materials.