Gender differences in conceptual understanding of Newtonian mechanics: a UK cross-institution comparison

We briefly summarize the educational background from which the vast majority of our undergraduate intake is drawn for readers not familiar with the UK system. There are three main sets of qualifications for our intakes.

• A-levels, taken by the majority of students from England, Wales and Northern Ireland, and some from Scotland. The final two A-level years follow on from six years of primary schooling and five years at secondary school where a broad set of subjects is studied. Students generally take a minimum of three subjects for these final two years before going to university. The results for each subject are reported as letter grades (highest grade A*, other pass grades A–E, and U='unclassified' as a failing grade) and are obtained from a combination of the results from a set of modular exams. Within this system there are various syllabuses for both mathematics and physics, and physics itself is taught in a manner which largely avoids the use of calculus. Students are exposed to more of the mathematical rigour of physics if they choose so-called 'mechanics modules' (elective modules on the mathematics of Newtonian mechanics) as part of their mathematics A-level.
• Scottish Higher qualifications, taken by most Scottish students, since Scotland has a separate education system. Entry to universities in Scotland is based on performance in these qualifications, taken at the end of year five of secondary level. Typically five subjects are studied to Higher level. Most students stay on for a sixth year at secondary level, taking (usually) three subjects at Advanced Higher level.
• Finally, a growing fraction of our students enter with the UK International Baccalaureate. It consists of three specific core elements, and study of six elective subjects, three of which are at higher level.

The typical age of an incoming student to undergraduate physics programmes in the UK is 18 or 19 (occasionally 17 from Scotland). The school system in the UK is funded through a variety of sources, where the most important difference with the rest of Europe is the larger number of students studying at privately funded schools. First year students at the three universities in this study (two from England, one from Scotland) therefore comprise a diverse group in terms of their prior academic background and ability.

All three institutions have been utilizing the FCI (or a close variant thereof) as an assessment instrument within their introductory physics courses for a number of years. We all had slight differences in our approach to the FCI, but we have aligned our processes in the 2011–12 academic year, on which this work is based. Due to a substantial increase in tuition fees in England for 2012, this was also a year when entry qualifications were higher than in recent previous years.

All three universities require both physics and maths school-leaving qualifications for students wanting to study physics courses. There are, naturally, differences between the cohorts and courses at the three universities, which are detailed below.

2.2.1. Edinburgh

2.2.2. Hull

2.2.3. Manchester

A summary of the implementation procedures employed at the three institutions is presented in table 1. In the case of Edinburgh, where the FCI score contributed to approximately 3% to the course mark, it was the better of a student's two attempts that counted.

Since our data are represented in two two-way contingency tables (one dimension is male or female, the other their score on one of the FCI tests), the best approach to see whether the differences between males and female students are significant is the Pearson χ² test of association. This test is based on a test statistic that measures the divergence of the observed data from the values that would be expected under the null hypothesis of no association. This requires calculation of the expected values based on the data; the expected value for each cell (i, j) in a two-way table d_ij is equal to

$$\begin{equation*} {\bar{d}}_{ij}=\bigg(\sum _{i^{\prime }} d_{i^{\prime }j}\bigg)\bigg(\sum _{j^{\prime }} d_{ij^{\prime }}\bigg)/n, \end{equation*}$$

where n is the total number of observations in the table. The statistic is defined as

$$\begin{equation*} \chi ^2=\sum _{ij}(d_{ij}-{\bar{d}}_{ij})^2/{\bar{d}}_{ij} . \end{equation*}$$

This is distributed as a χ²-distribution with (N_column − 1)(N_row − 1) degrees of freedom, and we can test whether the observed value for the statistic is significant by looking at the P value for the χ² distribution.

The alternative of approximating the two sets of mark distribution as continuous is also attractive; the disadvantage of that approach is that the data are not normally distributed, mainly due to the fact that scores on the post-instruction test are close to the maximum, and thus fluctuations above the mean have only a limited range. Thus the standard t-test for the equality of the means does not apply. We have applied the Mann–Whitney U test to test the difference between the distributions of the independent data sets.

The FCI was administered to each cohort before and after relevant instruction. In the presentation of results that follows, only matched pairs of data (i.e. data for students who had taken both the pre- and post-instruction tests) are included. Thus the sample size for each institution is lower than total class size.

For the members of each cohort undertaking both a pre- and post-instruction test, we calculate the pre- and post-instruction average percentage scores 〈x〉_pre and 〈x〉_post. This allows us to calculate a cohort-averaged normalized gain, 〈g〉 defined as

$$\begin{equation} \langle g\rangle = \frac{\langle x\rangle _{{\rm post}}-\langle x\rangle _{{\rm pre}}}{100-\langle x\rangle _{{\rm pre}} }. \end{equation} \tag{ 1 }$$

This normalized gain is often considered as a measure of instructional effectiveness, representing the fractional improvement in understanding, as first described in Hake's study [2]. The normalized gain can be calculated for the entire cohort, or just the male or female sub-cohorts, using the appropriate mean FCI scores.

For the analysis of the male and female sub-cohorts, we can define a performance gender gap, G, as the difference between male and female mean scores, such that

$$\begin{equation} G_i = \langle x\rangle ^{{\rm male}}_{i} - \langle x\rangle ^{{\rm female}}_{i} \end{equation} \tag{ 2 }$$

where the subscript i denotes pre- or post-instruction, and the convention we have adopted is that positive gaps imply male students outperforming female and vice versa. We can define a change in this gap, ΔG, as the difference between values of G determined for post- and pre-instruction, such that

$$\begin{equation} \Delta G = G_{{\rm post}} - G_{{\rm pre}} \end{equation} \tag{ 3 }$$

where the convention here is that a positive value of ΔG denotes a gap that widens as a result of instruction, and negative ΔG denotes a gap that narrows.

Table 2 presents values for these quantities for all three institutions. Cohorts from all three institutions show substantial learning gains on the FCI, comparable with those seen on 'reformed' courses in studies reported in the literature [2], providing evidence for effective (even though they are all rather different) instructional methodologies. However, all three institutions show a consistent performance gender gap (G positive) on the basis of pre-instruction test results, ranging from +10% to +19%. Furthermore, this gap persists on the post-instruction assessment and is statistically significant (p < 0.05), but is reduced in all three cases. Figure 1 illustrates this, presenting the male and female sub-cohort data for pre- and post-instruction tests in graphical form.

Zoom In Zoom Out Reset image size

Table 2 illustrates that on the basis of the pre-instruction test, female students start the courses with lower FCI attainment. It is instructive to investigate the distribution of these students across the cohort and chart their later outcomes on the post-instruction test. To do this, we split the cohort on the basis of pre-instruction test performance into quartiles (of approximately equal size) at each institution. We then further separate each quartile into male and female subgroups. The performance on the post-instruction test of these gender-split quartile groups for each institution is presented on figure 2.

Zoom In Zoom Out Reset image size

For the data from Hull, there are no female students in the top two quartiles and the rather small sample sizes (particularly of female students) means it is not sensible to try and draw too many conclusions from the data in figure 2(b). For the larger sample sizes of the Manchester and Edinburgh cohorts, there was no statistically significant difference in post-instruction test scores of the male and female cohorts within each quartile group.

However, we do note that in case of the lowest quartile for all three institutions, the mean post-instruction test score for this quartile barely reaches the pre-instruction whole-cohort average for that institution. In other words, on average, students in the lowest quartile pre-instruction show the lowest normalized gains post instruction. Another way to analyse the data is to consider the 'churn' between students in the lowest ability quartile on both the pre-instruction and post-instruction tests. Considering the Edinburgh data, we find that approximately 70% of students initially in the lowest quartile are also found in the lowest quartile on the post-instruction test, with all of the remainder elevated to just the third quartile. For Manchester almost 60% of those students initially in the lowest quartile on the basis of pre-instruction FCI scores remain there.

Further insight into the consequences of these results can be gained if we consider the relative proportions of male and female students in each of the four pre-instruction test quartiles: so as not to overcrowd figure 2, this is presented separately in table 3. This illustrates that the fraction of male students in each of the four quartiles is approximately equal, and furthermore this is consistent for the male student cohorts from all three institutions. In other words, prior to instruction, male students are distributed approximately evenly throughout the ability range as determined by the FCI. In contrast, there is a starkly different picture for female students. Across all three institutions, approximately half the female students in each of the institutions are in the lowest ability quartile prior to instruction (final column in table 3). Taken together, figure 2 and table 3 present a worrying picture, for both the starting point and outcome for female students. Approximately half start in the lowest quartile, the majority remain there, and for these students, figure 2 shows that their post-instruction test performance remains, on average, the lowest of all eight sub-cohorts for the larger data sets from Edinburgh and Manchester.

It may be tempting to suggest that this is due to these students simply being weaker at the point of entry. We find no evidence for this in their prior qualifications, though it is difficult to obtain good, discriminating, quantitative data since most entrants arrive with very similar school leaving qualifications, frequently at or close to the highest grade bands. Furthermore, we have far from complete data on the number of mechanics modules taken by these students during their final secondary school study of physics and maths, so even comparing students with the same grades may not reflect their prior exposure of to Newtonian mechanics. By looking across a wider population of UK students, rather than simply the fraction that attend our institutions, there is clear evidence that female students outperform male students in school-leaving examinations, including physics [10]. In the US, the situation seems to be slightly different. Sadler et al, considering the impact of high school and other affective experiences [25] have reported that the overall background of female students entering college physics was stronger in most subjects, but not in physics. Nevertheless, they conclude that the stronger academic background of females entering college physics did not appear to help them perform better than males; in fact, they performed worse than their male counterparts with the same academic backgrounds.

Given these differences, it is reasonable to question whether they arise from a greater fraction of male students getting certain questions (items) on the FCI assessment correct, or whether the origin is a consistent outperformance across the entire test.

In figure 3, we plot the fraction of male students getting an individual item correct against the corresponding fraction of female students who do likewise, for each of the 30 items on the FCI instrument. Even though we already know we have a performance gender gap, and thus are not expecting the line of unit slope to represent a line of best fit to the data, the data confirm that a larger fraction of male students get a given item correct compared to their female counterparts, for almost every item on the instrument (i.e. the majority of points lie above the line of unit slope, with the exception of only a small number of items for both pre- and post-instruction assessments).

Zoom In Zoom Out Reset image size

The largest gender differences tend to occur for items that are generally more poorly answered by the entire cohort: this is particularly evident on the pre-instruction assessment scores (figure 3(a)) where the spread of total scores is wider. Although a complete item-by-item analysis of differences in male/female response choices is beyond the scope of this paper, we do highlight a few illustrative examples, chosen by considering those items that lie furthest from the unit line in figure 3, i.e. those items with the largest gender gap pre-instruction or post (or indeed both). These are in no way intended to be comprehensive, but rather representative of the complexity of the data.

3.2.1. Item 2

3.2.2. Item 13

3.2.3. Item 23

Of course, the FCI covers only part of the course material. Looking more widely, we may wish to consider the existence, or otherwise, of performance gender gaps on final examinations (typically the principal assessment component for all of these courses). In doing so, we accept the inherent limitations that exams often test a degree of knowledge ('bookwork') and other measures of proficiency as well as the conceptual understanding that forms the focus of the FCI. Nevertheless, for the same cohorts of students at each of the three institutions in this study, we find that there is no statistically significant difference in examination performance for male and female cohorts, as determined by a t-test at the 5% significance level.