Evaluation of Croatian students’ physics knowledge during the Covid-19 pandemic

The paper presents the results of the study in physics learning during the pandemic in Croatian schools. The aim of the study was to identify students’ views on physics lessons and problems with the online study process, as well as their knowledge. Data were collected by an online questionnaire in the school year 2020/2021 and knowledge tests for each level of pre-university education for two testing cycles, school year 2020/2021 and 2021/2022. The results of the questionnaire show that, from the students’ point of view, teaching during the pandemic was as detailed as in regular classes, and experiments were conducted very rarely. The knowledge test results show the difference in the level of physics knowledge in the pre-pandemic period, during online and hybrid classes.


Introduction
Physics education develops cognitive abilities as well as scientific and creative thinking, which are basic competencies in science and technology.Key educational goals of physics education are the development of a scientific and inquiry approach, reasoning and experimental skills, and formal critical, logical, and systematic thinking.Therefore, students should learn not only scientific results but also scientific methods and combine experimental knowledge with theoretical, analytical, and synthetic thinking.All these activities are of great value and help to develop students' intellectual abilities, promote their transition from concrete to formal thinking and, most importantly, motivate them to participate in classes and encourage them to explore physics laws [1].
Physics education in Croatia begins in the 7th grade of elementary school and extends to the 4th grade of high schools (gymnasium programs).At the end of the gymnasium education, all students must pass the State Matura.The aim of the State Matura is to objectively verify and evaluate the acquired knowledge and skills of students according to the prescribed curricula and programs.The State Matura examinations are standardized written knowledge tests prepared according to a strictly defined procedure and proper rules and administered throughout the country at the same time under the same conditions and criteria for all students.The physics on State Matura is an elective subject.Whether it must be taken or not depends on further education and the choice of faculty.The results are important not only for the students, because they have the necessary prior knowledge for further professional education, but also for the teachers, the schools, and the system at the national level, so that they can all plan further educational work appropriately to improve the efficiency and quality of high school education in the country [2].
Evaluation of educational outcomes and student knowledge is part of the educational process and reflects the achievement of learning and teaching objectives.Evaluation means the systematic collection (before, during, and at the end of learning) of information about how the student is achieving the learning objectives or knowledge standards from the curricula.The elements of evaluation in physics education are knowledge and skills, conceptual and numerical tasks, and investigation of physics phenomena.It is important that teachers review all of these elements and adapt the testing procedure to the teaching methods used.Both teachers and students benefit from knowledge testing.The teacher receives feedback on how successfully he or she has taught the students the content based on the data obtained and receives a basis for planning further learning.The student receives feedback on his or her acquired knowledge and understanding of the learning content [1,3].Knowledge testing can be conducted synchronously or asynchronously, must be continuous, and must meet the measurement characteristics of validity, objectivity, and reliability.They should also take into account the differences between students and provide concrete feedback at the end [4].
After 16.3.2020,when all schools in Croatia were closed because of the Covid-19 pandemic (COVID -19), not only the way of teaching physics changed, but also the way of learning physics.Students accepted the teaching methods presented by teachers, but not all of them have the same effect on the acquired knowledge and skills [5,6].At the beginning of the pandemic, face-to-face instruction was completely replaced by online non-contact instruction, which evolved into hybrid instruction in the 2021/2022 school year because the conditions under which instruction takes place required it.During these transitions, many alternative teaching methods were used, including experimental work, in which technology plays a key role by serving as the main communication channel and source of learning materials.Online teaching is more challenging in terms of providing hands-on work, feedback, and motivating students to work.As the use of technology in the classroom increases, so do the requirements for knowledge of its use and the digital skills acquired [7].All these teaching methods need to be structured [8] and have well-prepared teaching materials [9].The question arises about the learning conditions and the performance of students who have experienced this way of teaching physics.
Since the situation was the same all over the world, the same problems occurred depending on socioeconomics, health, ICT knowledge and skills, and many more [10].
All this has shown that a longitudinal study is needed to investigate the way of learning physics and the level of physics knowledge acquired by primary and secondary school students in special conditions such as the pandemic COVID -19 compared to normal conditions, in order to apply appropriate teaching methods in similar circumstances in the future.

Research questions and methodology
Two research questions were set: How does COVID-19 pandemic affect physics learning, from students' perspective?How does COVID-19 pandemic affect the acquired physics knowledge?A study was carried on the pre-university level in Croatia.
Before the study was carried out, according to Ethical code of research with children [11], the consent of the principals', teachers, parents (for students age under 14) and students themselves was collected.
Anonymity of the data was guaranteed for research purposes when processing the data.The data was collected in Microsoft Office Excel and statistically processed in SPSS.Descriptive and inferential statistics were used to describe the data.Classical test theory and Rasch methodology were used to describe the results of the knowledge tests.

Online questionnaire
To answer the first research question, the online questionnaire (instrument) was prepared.It consists of 5 parts: general data, conducting physics teaching during the pandemic, experiments in physics teaching, sociological component, exchange of experience.The data were collected via Google forms, and statistically processed in SPSS.

Sample
The questionnaire was completed by 1363 students (convenience sample) in the 2020/2021 school year for the previous school year (regular classes in the 1st semester and online classes during almost the entire 2nd semester): 61.9% of female students, 79.9% of high school students (58.6% of gymnasium students).The vast majority of students achieved excellent and very good grades in physics and mathematics in the previous semester (semester before filling out the questionnaire) (Figure 1).

Knowledge tests
To answer the second research question, a knowledge test was created for elementary and high school students who have physics: two last grades of elementary school, four-year high school with physics classes during all 4 years.For this purpose, 6 different online physics knowledge tests were created, each with 20 conceptual questions based on the regular questions of the national knowledge exam (State Matura).They were anonymously administered via Google forms, the link to which was sent to teachers, who agreed to participate in the research with their students.

Sample
The tests were administered in two cycles at the beginning of the school year: Test 1 -2020/2021 (under online teaching) with a total of 1303 students (convenience sample) (39.8% of elementary and 60.2% of high school students); Test 2 -2021/2022 (under regular/hybrid teaching) with a total of 704 students (convenience sample) (41.6% of elementary and 58.4% of high school students).Tests were assessing knowledge for the previous school year.

Students' views and challenges in physics learning during pandemic
Students are convinced that physics classes were conducted in the same details as in regular conditions, most often via Teams channel, Yammer or e-mail (Figure 2), like teachers [12].Topics and how detailed students think they were taught during online teaching.About 40% of students reported that their workload in online classes was the same as in regular conditions, and that homework and writing formulas and definitions were the only required assignments (Figure 3).From the results it is also evident that 67% of students were using just those materials for learning physics that was given during online classes, via mobile phones, and that during online classes they were spending time for learning physics around one hour per week or less.37% of students said that they had synchronous physics classes in the same time as during the regular classes, and 30% said that they had just asynchronous classes.Results of the study show that students also experienced problems due to technical difficulties, insufficient space to participate in online classes, and lack of direct communication with teachers and other students, similar was reported by teachers as presented in the study of Štibi et al. [12].The result is similar to other research pointing to the need to use appropriate interactive online learning opportunities [13].However, regardless of the benefits of online classes in terms of the convenience of participating in online classes from home, over 65% of students indicated that the regular form of classes was better.63% of students of primary and high school together reported that they very rarely had experimental work (Figure 4).Divided data for elementary and high schools show there is not much difference in frequency during the pandemic, but in elementary schools the situation is slightly better for the prepandemic period (regular classes), students indicated that experimental work was done more often than in high schools.Comparing the periods before and during the pandemic in terms of the type of experimental work (Figure 5), it is noticeable that all types of experimental work were shifted to "Never", except for those carried out with the help of ICT or any type of multimedia.

Students' physics knowledge
Results for the 5 tests in total for both testing cycles are presented in the Table 1.From the Table 1 it can be concluded that all the knowledge tests are particularly difficult (Kuder-Richardson (KR r)  < 0.8), and that students solved 6 to 9 items correctly in average, over 20 of them (maximum number of points of the test is 20).Total result of the tests for all grades is comparable to level of students' scientific literacy which is, based on PISA, at level 2 (or 28,4%) [14].It means that "at Level 2, students are able to draw on everyday content knowledge and basic procedural knowledge to identify an appropriate scientific explanation, interpret data, and identify the question being addressed in a simple experimental design.They can use basic or everyday scientific knowledge to identify a valid conclusion from a simple data set.Level 2 students demonstrate basic epistemic knowledge by being able to identify questions that can be investigated scientifically" [15].When the overall results of Test 1 and Test 2 are compared, only tests for 8 th and 3 rd grade have statistically significant difference of means, and for all grades magnitude of the differences in means is very small (η 2 <.05).
Results that are presented in the Table 2 and Table 3 shows that 9 items out of 100 items has Point biserial correlation coefficient (rPBK) lower than 0,19, and according to Item response curves theory (IRCt) [16], it can be seen that just five items, among all 5 tests had to be improved (3 items have discriminatory index of the items (IDis) < 0,2 from the topics of the 3 rd grade).During the administration of the tests, 4th year high school students enrolled in the faculty, so there are few responses, which is not enough for statistical analysis.
Figure 6 and Figure 7 present some items, in this case 1st and 20th for the 1st grade of high school, which Classical test theory (Table 3) flagged as problematic.From Figure 6 it is evident that items don't discriminate less able students from more able ones.If those results are compared to Rasch analyse of the items (Figure 7), it is evident that there is also problematic aspect of those items.A, B, C, D) of 1 st item and 20 th item for the 1 st grade test of high school (X stands for nothing is chosen; red distractor is correct answer).
There is no droping down probability for the incorrect answer (and vise versa).

th item
More able students have the same probability of solving this item corectly and incorrectly.Legend: Students' probability (according to their ability) of solving specific item: • smooth curves present theoretical data; crosses connected with lines present experimental data • red lines present students that incorrectly solved this item, blue lines present students that correctly solved this item Figure 7. Rasch analyse for the 1 st and 20 th item for the 1 st grade test of high school.Table 4 shows the item difficulties for two grade levels, 8th grade elementary school and 3rd grade high school, where there is a statistically significant difference between Test 1 and Test 2. From the curricula [1] and the questionnaire for students, the items marked in red are from the topics taught in the second semester of 2019/2020 (COVID), and those marked in black are from the topics taught in the first semester of 2019/2020 (strike in Croatian elementary and high schools).
When IDif's are compared across testing cycles and semesters, it can be seen that IDif's are lower for both testing cycles compared to the State Matura (Test 1 and Test 2: 7th grade t(38)=2.02,p=.000 both; 8th grade t(38)=2.02,p=.006, p=.008 respectively; 1st grade t(38)=2.02,p=.003, p=.007 respectively; 2nd grade t(38)=2.02,p=.000 both; 3rd grade t(38)=2.02,p=.04, p=.006 respectively), which means that their knowledge acquired in online classes is lower and statistically significant.However, the IDif value for 2nd cycle is slightly higher than that for 1st cycle, which means that the level of knowledge acquired in hybrid classes is higher, but not statistically significant (t(38)=2.03,p >.05), for all grades.
The number of tasks taught in each semester varies, as the number of tasks on a given topic in the State Matura is limited.Also, there are tasks that have been repeated with changed numbers, but the instructions on the tasks have remained the same.We must emphasize that due to the strike in the entire Croatian school system in the first semester of 2019/2020, which preceded the corona, not all teachers covered the same topic.This is evident from our questionnaire, but the majority of students, despite the mentioned differences, answered as the items were marked in the paper.

Conclusion
From the results of the longitudinal study, it is evident that the lower level of physics knowledge of the students of all the studied classes may be related to the implementation of the teaching process, experimental work and the obligation to participate in the teaching process during the special time mentioned above.Moreover, the results of the questionnaire show that in the school year 2020/2021 only 1% of the students do not have a computer and 48% have their own computer.Regarding the implementation of physics lessons, 44% reported that they had no direct lessons or less than regular conditions (before spring 2020) (2 hours per week), and those who have direct lessons participated every hour.Nearly 29% of students took no physics in online classes, while 60% studied one or two hours per week.
Students believe that physics classes were conducted in the same detail as under regular conditions, most often via the Teams channel, Yammer, or e-mail.63% of the students reported that they very rarely did experimental work.The most common means of experimentation was through ICT, videos, or pictures of the experimental setup.
About 40% of students reported that their workload in online classes was the same as in regular conditions and that homework and writing formulas and definitions were the only required tasks.Students experienced problems with technical difficulties, insufficient space to participate in online classes, and lack of direct communication with teachers and other students.Regardless of the advantages of online instruction in terms of the convenience of participating in online classes from home, over 60% of students indicated that the regular form of instruction was better.
The analysis of the physics knowledge tests shows a negative trend to the test scores prior to the 2020 spring State Matura.In the second testing cycle, hybrid instruction, (2021/2022) there is a slight positive shift compared to the first, online instruction, (2020/2021), as in the overall physics knowledge test scores, i.e., down to the difficulty index of individual items.This is very important because it shows the necessity of the experimental work of the students and the interactive teaching methods that were partially implemented during the hybrid teaching.
However, the data collection method and instruments, as well as the sample size, limit the research results to some extent.It is noted that this research shows that as the educational system changes, the development of ICT-based teaching methods and materials that actively engage students in distance education, including the experimental part of physics education, should be considered.

Figure 1 .
Figure 1.Distribution of grades in physics and mathematics of the previous semester (before filling out the questionnaire).

Figure 2 .
Figure 2.Topics and how detailed students think they were taught during online teaching.About 40% of students reported that their workload in online classes was the same as in regular conditions, and that homework and writing formulas and definitions were the only required assignments (Figure3).From the results it is also evident that 67% of students were using just those materials for learning physics that was given during online classes, via mobile phones, and that during online classes they were spending time for learning physics around one hour per week or less.37% of students said that they had synchronous physics classes in the same time as during the regular classes, and 30% said that they had just asynchronous classes.

Figure 3 .
Figure 3. Students workload during online teaching.
time required for doing (homework) homework Time required for learning (evaluation of knowledge)

Figure 4 .
Figure 4. Frequency of conducting experiments (for both levels of education).63% of students of primary and high school together reported that they very rarely had experimental work (Figure4).Divided data for elementary and high schools show there is not much difference in frequency during the pandemic, but in elementary schools the situation is slightly better for the prepandemic period (regular classes), students indicated that experimental work was done more often than in high schools.Comparing the periods before and during the pandemic in terms of the type of experimental work (Figure5), it is noticeable that all types of experimental work were shifted to "Never", except for those carried out with the help of ICT or any type of multimedia.

Figure 5 .
Figure 5. Types of experimental work before and during pandemic (both for elementary and high school).

Figure 6 .
Figure 6.Distractors (A, B, C, D) of 1 st item and 20 th item for the 1 st grade test of high school (X stands for nothing is chosen; red distractor is correct answer).

Table 1 .
Results of the knowledge tests (maximum score for each knowledge test is 20).

Table 2 .
Discriminatory index of the items (IDis).

Table 4 .
Item difficulty (IDif) of items on the State Matura and each testing cycle.Year of appearance stands for the school year of the State Matura in which the selected item was published.Items marked italic belong to the topics that were covered in online classes.