Assessing the prior knowledge of students in physics minors

The number of students who give up science and engineering in the first semesters is regrettably high. While there is empirical evidence of the relationship between (lack of) prior knowledge in mathematics and dropout among students with a compulsory minor in physics, there are - to our knowledge - no studies on the relevance of prior knowledge in physics. Therefore, we constructed a knowledge test and statistically validated the results of two pilot studies in consecutive semesters with students from different physics minor programs (n = 643). In this article, we briefly introduce the structure of the test, consisting of three sections (knowledge of facts, knowledge of meaning, application of knowledge) and the corresponding test parameters. Furthermore, we summarise the first empirical results using descriptive methods. For this purpose, violin plots were created to analyze similarities or differences in the response behaviour of the subgroups. Moreover, an analysis of classical statistics was carried out, e.g. pairwise Mann-Whitney-U tests for comparing sample subgroups. Our results show that subgroups in the overall sample significantly differ in their test results depending on their faculty affiliation.


Theoretical background
The dropout rate at German universities in science and engineering subjects is recurring excessively high at around 40% [1,2,3].In surveys, students report content-related requirements as the main reason for struggling in successfully participating at university [4].Likewise, many empirical studies have been conducted to investigate the relationship between prior knowledge and study success.These repeatedly show that prior knowledge in mathematics predicts students' academic success in physics, namely for students in their major and for students in their minor [5,6].Accordingly, some lecturers are convinced that the relationship between prior knowledge in mathematics is considerably more important for students' success in any physics or science course than subject-specific prior knowledge [7,8].At universities, the wide range of well-established and researched support programs in mathematics for university freshmen depicts this perspective [9].
In contrast, a general claim to offer students a support program in physics does seemingly not exist at universities.In international studies, the investigation of prior knowledge in physics gets more attention explicitly in research about how it influences the understanding of physical concepts and allows for conceptual change [10].Since physical concepts are complex, prior knowledge seems to be the foundation on which existing concepts can be restructured or new concepts can be acquired in the learning process [11].Furthermore, findings of prior knowledge studies by Dochy and Alexander in the 1990s, where students' learning processes and prior knowledge levels were analysed, suggest that additional studies about subject-specific prior knowledge in physics could lead to a better understanding of how students can improve on successfully achieving academic requirements [12,13,14].
Even though the research about the predictive power of physics prior knowledge for study success in physics gets only a little attention in Germany, some recent studies show the incremental validity of prior knowledge in physics for physics majors as a predictive factor for success [15,16,17].However, there are -to our knowledge -no studies on the relevance of prior knowledge in physics for physics minors, only one German study that shows a trend of 'physics knowledge' in STEM faculties of first-semester students [18].With the research results about physics majors in mind, this is surprising since physics is also a compulsory minor for many subjects with high dropout rates, e.g.engineering.This leads to the need for further research taking students in physics minors into account.In particular, firstly, a detailed view of their prior knowledge in physics in different task scenarios is missing.Secondly, there is a lack of data about the importance of prior knowledge for the success of different subgroups of students.

Research goals
The research goal of this particular study is to point out statistical differences in the prior knowledge of participants from physics minors.Therefore, we use a prior knowledge test, which we constructed and validated before, to describe the target groups by their test results.In conclusion, we want to show the differences and commonalities between students' test results in different knowledge types from several physics minors at TU Darmstadt.

Instrument
Similar to existing prior knowledge test instruments for physics majors from Binder et al., our test is grounded on the theoretical framework of Hailikari's [19] prior knowledge model.The model itself is founded on a more coarse differentiation of general knowledge from Alexander [14].
For the construction of our test for physics minors, we have taken the tests and the latest research results from physics major students into account and have developed them further, f.e. by adding new questions to cover up all contents in every test section and take known misconceptions into account.
Same as in studies from Dochy further back in time [20], two knowledge types are used for our considerations: declarative and procedural knowledge.However, following Hailikari, we assume that a more dimensional model is needed due to the high complexity of physics knowledge.Therefore we implemented a smaller segmentation of the two knowledge types where declarative knowledge splits up into knowledge of facts and knowledge of meaning and procedural knowledge into integration of knowledge and application of knowledge.In the model and latest tests, the level of difficulty increases from knowledge of facts to application of knowledge in the order given above.
Since Binder et al. [21] could show the incremental validity of knowledge of meaning and application of knowledge in predicting study success, we will evaluate these two types in the test.In addition, we implemented a less difficult test part for knowledge of facts because the prior knowledge of physics minor participants is expected to be much lower than that of physics majors.We did not test for integration of knowledge since it was not a predicting factor for success for physics major students.In addition, we wanted to limit the timeframe of the test, which is already high.Therefore, the final test consists of three sections (knowledge of facts, knowledge of meaning, application of knowledge), each assessed separately.All three test sections cover essential contents in mechanics, electricity and optics with different question formats.
Knowledge of facts consists of multiple-choice questions with a single correct answer and are rated by right 1 or wrong 0. The section knowledge of meaning covers up questions about physical concepts or facts.Students must describe the answer in one or two sentences of freetext input [22].To assign appropriate scores for the answers, we used a rubric containing all of the core aspects of the physical concept or fact.Depending on how well or if the core aspect is described in the answer of the student, students get a score of 0 -core aspect not mentioned, 0.5 -core aspect mentioned, but not precise enough or incorrect or 1 -core aspect described correctly.The maximum number of core aspects of a single question is two.In the section for application of knowledge, students have to name the correct solution approach [23] adapted after Chi et al. [24], which have an arithmetic task framework.Therefore, this last test section was also rated dichotomously.

Study design and sample group
The prior knowledge test is fully implemented and administered in an online format.Our study design follows a two-step approach in order to derive information for test improvements from data of two pilot studies: At the beginning of the fall semester 2021/22, the first pilot study was conducted with physics minors in four different lectures, "physics for biology" (students' first term of study), "physics for electrical-engineering" (students' first term of study), "physics for chemistry" (students' third term of study) and "physics for mechanical-engineering" (students' third term of study).We conducted the second pilot study in the following spring semester 2022 in one remaining physics minor, "physics for civil and environmental engineering", intended for the students' second term of study.
The test was part of the students' homework assignment and took about 60-70 minutes (pilot study I) or rather 50 minutes (pilot study II).In total, pilot study I comprises a sample of n = 643 students (biology n B = 28, electrical-engineering n EE = 226, mechanical-engineering n M E = 198, chemistry n Ch = 78) and pilot study II of n CEE = 113 students (all of them civiland environmental-engineering) who fully completed the tests.

Results
Here we offer an in-depth view of the statistical values of both pilot studies.However, it is important to emphasize that the results of these studies are only comparable to a certain degree because of the modifications we made in the test instrument from pilot study I to pilot study II.Therefore we consider the results of both studies as preliminary, and they remain to be confirmed in the planned main study.

Test parameters
From pilot study I to pilot study II, several adjustments were made in the test instrument in order to improve test parameters for our sample of physics minors.Table 1 gives an overview of these adjustments and the instrument's characteristics.
On the one hand, we aimed at shorting test length.On the other hand, we had to revise the task format of the test part application of knowledge since the difficulty was too high for our sample group compared to physics majors [22].While in pilot study I students had to give a solution approach in a free-text answer, in pilot study II they could choose a solution approach from a drop-down menu.
To indicate test reliability, Cronbach's alpha was calculated individually for each test section of both pilot studies I and II.Internal consistency is considered acceptable or good for all test sections, with the most critical value at the application of knowledge section in pilot study II [25].This value can be traced back to the much lower and, therefore, a more outlier-sensitive sample size of the second pilot study.In addition, only 12 tasks (pilot study II) were conducted instead of 18 (pilot study I), which affects Cronbach's alpha as well.

Descriptive results
The following descriptive results are separated for the different test sections.To better visualise statistical data, we created violin plots presented in the following.These show boxplots with the associated distribution functions.In addition, tables 2-4 give a detailed view of all exact data supplemented by the calculated means and standard derivations.

Knowledge of facts
For the comparison between the students participating in the different lectures, the plot in figure 1 shows an uprising trend in the achieved score percentage with biology participants at the lower end followed by civil-and environmental engineering, chemistry, electrical engineering and mechanical engineering at the upper end.The mean values in table 2 reflect these visible results additionally.
In addition, it can be seen through the distribution functions that the results of chemistry, mechanical engineering, and civil-and environmental engineering lectures accumulate at two different values.The distributions from electrical engineering and biology physics minors seem to have only one peak score percentage.
The overall results in this section are relatively high, with no ceiling effects at the upper end of the scale for all of the sample groups.Table 2. Descriptive statistical data for knowledge of facts."q -1" and "q -3" are the first and third quantile of the sample group.

Biology
. shows similar results for the arrangement of achieved scores between the groups.In the lecture for biology students, participants performed worst and mechanical engineering best.The chemistry and electrical engineering order has changed compared to the first test section.Furthermore, in contrast to knowledge of facts, only the electrical engineering lecture sample group seems to have two peak scores.One is more in the upper part of the distribution, close to the median of the best-performing mechanical engineering students.The other one is placed in the lower part of the overall distribution.The scores achieved in this second test section are lower overall than in the first.
As indicated by the distribution functions in the chemistry, electrical engineering and civiland environmental engineering lectures, the test section shows a small ground effect at the lower end of the scale.Additionally, the distribution of achieved scores in civil-and environmental engineering is wider than within other physics minor lectures.This also appears in the difference between the quantiles and the high standard derivation of the mean [M = 0.44 SD = 0.27].Table 3. Descriptive statistical data for knowledge of meaning.

Application of knowledge
In the last section, application of knowledge, participants of the first pilot study (the four plots on the left) show a remarkable ground effect.For example, in physics for biology students, the median of the achieved score is just slightly over zero (M D = 0.03).Therefore it shows that the section is the most difficult in the overall knowledge test, as intended.Scores are significantly lower than in the other two test sections.Accordingly, the scores are also significantly lower in this test section than in the first two test sections for chemistry, electrical engineering and mechanical engineering students.Their distribution functions and boxplots are similar, with mechanical engineering being a bit ahead of the other lectures.Therefore it shows that the section is the most difficult in the overall knowledge test, as intended, but might even be too difficult for the sample of physics minors.The wide range of the distribution of the results and many students achieving scores around zero suggests that the section was too challenging for the sample group in pilot study I.The only hinted peaks of distribution functions at 0.00 percent of the achieved score display this fact additionally.
Since the second pilot study was conducted with a modified task format, the results in this section cannot be adequately compared.Nonetheless, the plot indicates that civil-and environmental engineering students have way better results in the percentage of the achieved score than the participants in the first pilot study.Thus the distribution function of the second pilot study shows similar behaviour to the other test sections, with a downward shift on the scale.

Statistical analysis
In this chapter, we evaluate if there is a correlation between participating in a specific lecture and the achieved score in each test section.The means were analysed with non-parametric significance tests (pairwise Mann-Whitney-U) to make valid statements about differences and similarities between the descriptive results of the different samples [26].In addition, we calculated the effect sizes of the significance tests via Pearson's r correlation coefficient [27], shown in table 5.
To a certain degree, the results of the correlations replicate the mentioned results from descriptive statistics.We exclude the comparison of pilot study I and II results for application of knowledge in this entire section because of the high discrepancy in the task format.
No correlation coefficient shows a high degree of correlation between the sample groups (r < .50).The most significant results of the calculated values are detected between mechanical engineering participants and all other lectures.For example, there are significances with a moderate effect size between mechanical engineering and biology (r = .40)as well as civiland environmental engineering (r = .45)participants and with a small effect size for chemistry (r = .25)and electrical engineering (r = .22).The high differences in medians can underline these results: f.e. the median for knowledge of facts for mechanical engineering participants has a maximum difference of 0.30 to participants of the physics for biology students.
The correlations are decreasing for the knowledge of meaning section but are still present.In fact, the differences of the medians show that mechanical engineering participants still lead by maximum differences of up to 0.21 compared to the biology group, which is meaningful high.
Taking a vertical view of the values within the table, the results for significance concerning biology participants stand out.In almost every test section, they reach different results with mostly high significance and low to moderate correlations, underlining the lower achieved scores in every test section, which we already discussed in the previous chapter.
The least significances appear between physics for chemistry participants and the other lectures.
Table 5. sizes calculated via p-values of pairwise Mann-Whitney-U tests.The values in the table show the effect sizes in form of Pearson's r: 0.00 indicates no correlation, < .29 low degree correlation, .30< r < .49moderate degree and > .50high degree.Significance level is shown by: * p < 0.05, * * p < 0.01 and * * * p < 0.001 [28].

Knowledge of facts
Biology Chemistry Civ.Eng.El.Eng.

Discussion
Overall the data can replicate the theoretical knowledge model of Hailikari since, for all groups of physics minors: knowledge of facts was the easiest test section, increasing from knowledge of meaning to application of knowledge, which was the most challenging section.We showed that the respective course of study correlates with the prior knowledge evaluated by achieved score percentages of the test sections.One additional factor seems to be the semesters in which the five subgroups of our sample attend the physics lecture.Advanced students have probably acquired knowledge in other courses relevant for the physics lecture, which should positively influence their test results.However, this can only explain part of the findings since the electrical engineering lecture participants show better results than civil engineering students, even though the latter are more advanced.Pointing out the third-semester lectures for chemistry and mechanical engineering students, there is a significant difference in the test section for knowledge of facts.This effect may occur because mechanical engineering students might have to make more calculations, especially in subjects like technical mechanics.
There are two limitations of the test to point out.At first, the different groups of students in the sample differ not only in the choice of their course of study but also in their study progress, as mentioned above.Groups are, therefore, comparable only to a certain degree because of the, in our case, not measurable conditions students bring before participating in the prior knowledge test.Second, the test only covers contents in mechanics, electricity and optics and leaves out other content areas such as e.g.thermodynamics, which can also be content in some lectures, depending on the lecturer.Since the test was constructed for measuring the prior knowledge in any physics minor, we had to determine specific contents to cover up the sum of curricula of physics minors as best as possible.Consequently, we had to accept losing lecture-specific information in some physics minors, such as prior knowledge in thermodynamics, which, e.g.comes into account in the physics for chemistry lecture.
With a view to already preceeded studies in Germany from Binder et al. [15,16,17], we could show that the structure of the knowledge test also fits the needs to examine prior knowledge for physics minors.The statistical analysis led to sufficient results to confirm the validity of our final version of the prior knowledge test.Especially in the test section application of knowledge, much more information could be generated in pilot study II than in pilot study I.Even though the test reliability for this section decreased, explained by the assumptions in previous chapters.
The first preliminary findings concerning our research goal could also be made in the pilot studies by examining significant differences between the sample groups in the three test sections.

Conclusion and outlook
Our results can lead to the conclusion that the different physics minor lectures, all administered by the faculty of physics, must be accommodated to the differing needs concerning the prior knowledge of the sample groups.This could mean that lectures must adjust the given learning speed and the curriculum.Thought ahead, another option leads to specific support measures, where students with less prior knowledge could approach the minimum needed prior knowledge to participate in physics minors successfully.Considering the alarming low success rates, particularly in physics minors, supporting students' physics prior knowledge should also be focussed next to the much higher valued mathematical prior knowledge.
As the next step in our project, from the end of 2022 till the summer of 2023, the main study will be completed for further examination and more coherent results for all physics minors at TU Darmstadt.To generate a more informative value about the influence of prior knowledge in physics in relation to the success in physics minors, the results will be evaluated with methods of Item Response Theory and be investigated for correlations to the final exams of the lectures.It is also to be seen if the revised test section for application of knowledge shows better test reliability for the bigger sample group than for the sample in pilot study II.Additionally, the results of the two pilot studies led us to assume that there is information within the free-text answers, which could lead to students' preconceptions.Therefore, we want to analyse the results of the main study for content-related particularities, especially in the section knowledge of meaning.

Figure 1 .
Figure 1.Violin-Plots for knowledge of facts in all five physics minors showing boxplots wrapped up by the distribution functions.

Figure 2 .
Figure 2. Violin-Plots for knowledge of meaning in all five physics minors showing boxplots wrapped up by the distribution functions.

Figure 3 .
Figure 3. Violin-Plots for application of knowledge in all five physics minors showing boxplots wrapped up by the distribution functions.

Table 1 .
Contents of different test versions with calculated Cronbach's alpha.

Table 4 .
Descriptive statistical data for application of knowledge.