Analyzing the measurement error from false positives in the Force Concept Inventory

We analyze the measurement error due to false positives of the Force Concept Inventory (FCI) focusing on four questions (Q.5, Q.6, Q.7, and Q.16). We determine whether or not a correct response to a given FCI question is a false positive using subquestions. Using the data of 1145 university students in Japan from 2015 to 2017, we find that the sum of the systematic error from the false positives of Q.5, Q.6, Q.7, and Q.16 is about 10% of the FCI score of a mid-level student. We consider what degree the error influences the measures of the effectiveness of a course, namely, the average normalized gain and Cohens' d. Using a set of simulated data, we show that Cohens' d is less sensitive to the systematic error due to false positives than the average normalized gain.