Developing and tracking profiles of student conceptions of force through an engineering degree

Research over the last four decades has shown that common misconceptions about force can be particularly resistant to change. The development of Newtonian ideas of force is important in many branches of engineering, particularly in the understanding of structures and the mechanics of machines. While these ideas are important in engineering, evidence from assessment shows that while students are able to solve problems and perform at a relatively high level, they have often not fully understood the underlying Newtonian concepts. In order to detect and ameliorate these misconceptions, a number of excellent diagnostic tools have been developed. Probably the most well-known of these are the Force Concept Inventory (FCI) and the Mechanics baseline test (MBT), developed by David Hestenes and his colleagues at Arizona state university [1]. Both of these tests been statistically validated and used worldwide in many different contexts. Evidence from data collected in 2016 shows that students in their final year of study perform as poorly as those in their first year on the FCI. What is not shown by these data is the way in which the students’ conceptual profiles change as they progress through their courses of study. Conceptual profiles can be used to map the evolution of student conceptual thinking [2–3]. In this project, we use the FCI to develop conceptual profiles for individual students enrolled on two programmes of study; the Bachelor of Engineering Technology and the New Zealand Diploma in Engineering, both of which start with introductory physics courses, but rely on force concepts throughout the qualification. Our intention is to track the evolution of these profiles as the students’ progress through their programmes of study. The study is in the first year of a three year cycle and this paper outlines the research design and reports on the changes that have taken place in the first six months of the students’ qualification, after they have completed a basic mechanics course.


Introduction
The development of Newtonian ideas of force is important in the teaching of content in many areas of engineering, but particularly in structural and mechanical engineering. For example, a basic understanding of Newton's laws are imperative for the understanding of trusses and the operation of machines. These ideas are used in the engineering curriculum again and again, but evidence from assessment shows that while students are able to solve problems and perform at a relatively high level, In this paper, I outline the first part of a three year longitudinal study that seeks to track how engineering students ideas of force develop from entrance into both a two year diploma for engineering technologists (New Zealand Diploma in Engineering -NZDE) and a three year Bachelor of Engineering Technology (BEngTech). Both qualifications are accredited through international accords and as such represent a common international benchmark in terms of learned content as well as assessment practice, making a study such as this not only important in terms of the development of New Zealand Engineers, but also internationally.
There were two main aims of this research project. The first is to track the development of students understanding of force as they progress through their chosen qualification. This would be done using the development of individual conceptual profiles, for each student at various stages in their progression through the qualification and then tracking the way these profiles change.
The second aim of the project is to use research findings to develop a teaching tool that can be used alongside traditional courses to provide feedback to students on their progressive understanding of force. This would be done by using the above-mentioned profiles as a tool for amelioration.

Literature review -early misconceptions research
Over the last few decades, much work has been done investigating the learning of important physics concepts. In early work by for example; Gilbert and Watts [4], the identification of particular misconceptions in physics took place, with the concept of force being one of the most extensively researched. This has led in turn to the development of diagnostic tools such as the Force Concept Inventory (FCI) and the Mechanics baseline test (MBT), designed by David Hestenes and his colleagues at Arizona state university [1].
In the 1980s, it was noticed that many of these misconceptions that were identified, were similar to paradigms of thought that have been prevalent at various times over the last 2300 years of physics understanding and development. A good example of this is the idea of impetus, a conception that can be detected in student thinking today. The connection between force and motion in the impetus idea was initially identified by Aristotle but then later developed into the idea that the object in motion carries with it the impetus to make it continue on its trajectory. This idea continued to be the primary conception of force until at least the 14th century and many of today's students still find this quite acceptable as an explanation.
The mirroring of paradigms has been discussed at length in the literature and developed into a theory regarding the coherence of student views of the way things work that is best reflected by the work of Rosalind Driver [5]. The idea that misconceptions demonstrated by students are actually not a fragmented collection of "wrong ideas", but part of a more holistic alternative conceptual framework that necessitated a paradigm shift similar to the major paradigm shifts that have taken place in the evolution of science, was commonly accepted as can be found in much of the early work in the 1980s.
The existence of conceptual frameworks that mirror the historical frameworks of the different eras of the development of physics knowledge (Aristotelean, Galilean, Newtonian and Einsteinian) naturally leads to thought regarding how we can get students to change from one paradigm to another and the idea of conceptual change, something that has also been written about extensively [6][7][8].
As to whether students came into introductory physics courses with a coherent framework of ideas similar to the great paradigms of history, or whether they had a less holistic, fragmented view of each concept that was context specific became the big question toward the end of the 1990s. DiSessa, Gillespie and Esterly [9] have addressed this issue directly by replicating work by Ioannides and Vosnaidou [10] where they found that students' ideas regarding force are not necessarily incoherent and unconnected, but that neither do they exhibit a systematic framework of thought that undergoes a paradigm shift similar to the great paradigm shifts in history.
Using diSessa et al's "knowledge in pieces" view of conceptual change as a framework for describing and interpreting student thinking, data on student responses to the FCI test [1] will be used in this project to develop individual profiles of student thinking. Conceptual profiles can be used to map the evolution of this thinking [2,3] and it is hoped that by identifying and developing profiles of conceptual thought that might be context related and piecemeal for each student, we can track the way in which these profiles change as a student matures through an engineering degree.
Lastly, while there has been debate over the last few decades regarding the name given to students' wrong ideas in physics, for example, the use of the terms; misconception, alternative conception, preconception etc. [4], for clarity, I will use the term misconception in this paper.

Research Methodology
The design of this study is fairly simple and aims to track the development of each student's concept of force as they progress through their qualification. This will be done by periodic testing of different cohorts of students, intensively in their introductory mechanics course and then later, less frequently throughout the rest of their degree (See figure 1 below). This the data discussed in paper however, only reflects the initial six months of the project, which extended from March to June of 2017 and as such, results are formative. The project will extend until the end of 2019, a total of six semesters as indicated in the force concept tracking plan in figure 1.

The participants
The participants in the first semester of 2017 were:  16 students in their first year of the Bachelor of Engineering Technology degree (BEngTech), a three year degree. All of these need to take an introductory 1 semester mechanics course. Two of the 16 were female and the rest male.   53 First year NZDE (two year diploma at a lower level than the degree) students, all except one of whom were male The unusually high proportion of male students in this sample is common in engineering, particularly in the polytechnic sector by comparison to those who embark on a university degree, where although the proportion of female students in engineering is higher, they still do not usually make up 50% of the annual cohort. While data has been collected, for all 69 students, the detailed analysis has focused on the 16 students enrolled in the mechanics course for the BEngTech.

Using the Force Concept Inventory (FCI) as a research tool
In this project, I have used the Force Concept Inventory (FCI) as the primary diagnostic tool. The reason for this is that the test has been used in over 30 different countries and has been validated and refined for several years after its initial development. There are many other diagnostic tools available but this test has been the most used worldwide. The profiles discussed in this paper have been developed using the data gained from the FCI. While the same test was given to all students in the pre-test, post-test 1 and post-test 2, these were held six weeks apart and the test items were never discussed with the participants.
The force concept inventory consists of 30 questions that test for understanding of several important Newtonian mechanics principles, including Kinematics, Newton's three laws and the vector nature of force. The categories of misconception diagnosed by the FCI are listed in table 1 below. It must be noted that in this study, only six misconceptions regarding force were investigated. This means that a student might have given an incorrect answer that was not due to the six misconceptions above and while this will affect their score, it will not affect their tally of misconceptions.

Colour code Misconception
Kinematics concepts are vague and undifferentiated, leading to a confusion between velocity and acceleration The Impetus idea of force, namely that the force supplied by the "hit" will travel with the object being hit The idea of an active force, in other words that only active agents exert forces Students ideas about action / reaction pairs, for example that the object with greater mass will exert a greater force Various misconceptions regarding the vector nature of force Other influences on motion that have been detected

Discussion of the results
In this section, I discuss the formative data collected for the class of 16 BEngTech students and have analysed the results of the three tests in two ways; the first being a discussion of the changes that have taken place in terms of the profile of the group as a whole. The second is an attempt to categorize individual participant profiles of thinking and see if there are common individual changes to these profile categories that emerge from the data. Figure 2 shows raw data changes taken from a subset of the whole class improvement from pretest through to the second post-test. (In the diagrams, the columns A9 to A14 represent selected students who took the tests while the letter in the spreadsheet are the actual answers given for the test items.)This is a sample of the class and is used to illustrate visually the changes that occurred in performance on the FCI from the pretest through to the second posttest. At a glance, one can see from figure 2, that there is a progressive change from pre-test to post-test 2, both in number of misconceptions as well as the individual student profiles of response. Table 2 shows the changes in performance of the whole class (16 participants) over the initial phase of the project in terms of the number of instances the class as a whole exhibited in response to the FCI, normalized as a fraction of the maximum possible number of responses for each misconception, as well as the number of instances per question addressing each misconception. The reason for the normalization is that different misconceptions are tested with different questions and therefore the number of instances of the use of a particular misconception is dependent on the number of questions that address that misconception. For example, in order to test the kinematics misconceptions, there were only three questions and a maximum number of seven possible responses that indicated misconceptions in that area. The Impetus misconception was featured in a total of 14 questions and there were a maximum of 31 possible responses a student could make that would show the use of this misconception. Table 2 data is presented as a fraction of the maximum possible number of responses, rather than the number of questions that address each particular misconception. Both of these are rough indicators of relative performance, each with its own weaknesses, mainly because many questions address more than one misconception.    Table 2, instances of misconception use (upper statistic in each cell) normalized as a fraction of the maximum number of possible instances of misconception use (middle statistic in each cell) and as a fraction of the number of questions in the FCI addressing each misconception (lower statistic in each cell) Table 3 below shows the number of students using the different misconceptions detected by the FCI. This gives a different kind of picture of the class profile in terms of numbers of students having a particular misconception. In analysing the FCI responses, a student was deemed to have a misconception if they showed the use of a particular misconception at least twice in the test. This is to eliminate students who made silly mistakes, through carelessness. For example, in the post-test raw data shown in figure 2 above, A11 would not be deemed to have the impetus misconception, but A14 would. Table 3. Whole Class improvement: Number of students with different misconception types.

Test
Kinematics Impetus Active Force N3

Force as a vector Gravity
Pre-test 3 (19%) 10 (63%) 7 (44%) 9 (56%) 6 (38%) 2 (13%) Post-test 4 (25%) 12 (75%) 5 (31%) 9 (56%) 9 (56%) 3 (19%) Post-test2 1 (6%) 5 (31%) 2 (13%) 3 (19%) 4 (25%) 2 (13%) Two aspects of the "whole class" data emerge. The first is that in general, there is not much improvement between the pre-test and the first post-test, but there is a marked improvement between the first post-test and the second post-test, even though during this stage of the course, fundamental aspects of the force concept were not covered and the class rather focused on more engineering style algorithmic problems. This improvement manifests itself as a marked decrease in the number of instances each of the identified misconceptions is used (pre-test -99 instances, dropping to 91 in the first post-test and then to 56 in the second post-test), as well as a marked decrease in the number of students using these misconceptions.
The second is that some misconceptions appear to more frequent than others and more resilient than others. By looking at table 2, at first glance the impetus misconception (30 instances) is more common than say the undifferentiated use of kinematics concepts (7 instances), but when the data is normalized as a fraction of the maximum possible number of instances, the impetus misconception has an index of 0.97, whereas the difficulty with kinematics concepts an index of 1.00 not very different. Similarly, it seems that the use of the active force is as common as the impetus misconception. What appears to be a much larger problem are the misconceptions the class has around Newton's third law. The FCI, while an excellent tool for guiding teaching, does have a weakness in that it is biased towards testing for IOP Conf. Series: Journal of Physics: Conf. Series 1286 (2019) 012003 IOP Publishing doi:10.1088/1742-6596/1286/1/012003 7 particular misconceptions such as the impetus misconception and that looking at the statistics can lead to a misconception regarding the relative strengths and weaknesses of the class as a whole.

Individual student profiles
While the profile of the whole class can identify broad areas that need to be addressed in teaching and can be used to guide classroom activities, it is the changes in the individual profiles that are perhaps more useful in diagnosing students' problems and helping them develop their own learning programme. In analysing the conceptual profiles of individuals in the class, four different profiles were identified. The categories were based on the number of misconceptions used by a student as well as their score. This is fairly arbitrary, but serves to differentiate students initially. These categories are described below: 1. Has shown no evident misconceptions according to the FCI. Made a few mistakes in the test that could be put down to carelessness and which were not repeated 2. Has made repeated mistakes demonstrating one or two misconceptions. Typically scored between 10 and 20 out of 30 for the test 3. Has made repeated mistakes demonstrating 3 to 4 misconceptions and typically scored between 10 and 20 out of 30 for the test 4. Has made repeated mistakes demonstrating the use of 3 to 5 misconceptions and scored below 10 out of 30 for the test.

Individual student profile changes
Examples of two individual profile changes are given below. In the first example, the student changed from a category III to a category I conceptual profile and can be seen as an example of a successful transition. The second example is of a transition from a category IV to a category II conceptual profile and while there has been improvement in this student's ability, this is not regarded as a success.
Example 1: (Category III to category I) Figure 3. Individual raw responses and spider diagram depicting changes in responses for example 1.
In this example, the student's score for the test improved between tests each time, indicating that the programme of learning she embarked on was successful. This student is able to discriminate between velocity and acceleration, and had a strong misconception of impetus demonstrated in the pre-test (6 instances), which reduced to 1 instance in the second post-test. This student also had a relatively persistent misconception regarding N3, which has reduced with time. While there was one instance of What was noticeable in the data, was the dramatic decrease in instances of the use of the six identified misconceptions between the two post-tests. This is possible further evidence of conceptual development, rather than a paradigm shift, as some students still showed signs of context specific use of misconceptions in their reasoning. The student in Example 2 presents an interesting case where the students score marginally improves, but the impetus misconception continues to dominate their thinking.
While the data analysed thus far presents some evidence of individual conceptual profile change, conclusions drawn from these data need to be considered in the light of the fact that not all the data collected so far has been analysed and that data collection over a longer period might provide further insight into changes in the students' conceptual profiles over time.
In the light of the completion of this first part of the study, it may be necessary to modify our research plan and introduce some targeted "interviews about instances" [4] as a way to unpack the process students go through when they undergo these profile changes. This mixed methods approach will hopefully provide us with a richer understanding of the students' experience in changing their conceptual profiles about force.
Finally, while this research does not provide a "quick fix" for practitioners, it does suggest that by using the FCI test and creating profiles for individual students as well as for whole classes, secondary school teachers will be able to develop a better understanding of their own classes as well as provide their students with a tool to promote their own self-awareness of their understanding of foundational concepts. Using a conceptual profile approach to map students' understandings could of course be expanded to the use of other tests as well.