Physics postgraduate teaching assistants' grading approaches: conflicting goals and practices

The data on TAs' goals for grading and grading approaches were collected using a group administered interactive questionnaire (GAIQ) which was previously developed and validated for use with TAs/instructors [63]. The GAIQ is comprised of a series of activities which use worksheets designed to clarify a TA/instructor's ideas about helping students learn physics content and desired problem solving practices. The GAIQ worksheets and artifacts encourage reflection on various aspects of teaching physics problem solving: designing problems on a particular physics topic with features effective for use in different situations (e.g. questions for peer and class discussions, homework, quizzes, exams, collaborative learning, etc), designing solutions to homework problems that will help students develop effective problem solving strategies, and grading student solutions. Questionnaires on each aspect of teaching problem solving (e.g. problem types, instructors' example solutions, or grading) involve three stages: (1) TAs/instructors are individually asked to solve a core problem (figure 1) suitable for distributing to their students and complete a worksheet eliciting TAs/instructors' initial ideas about teaching problem solving; (2) TAs/instructors work in groups of three to answer the same questions as in the pre-class worksheet and then each group shares their ideas in a whole class discussion; (3) TAs/instructors individually complete another worksheet in which they can modify their previous answers and connect their ideas to a list of pre-defined features about teaching problem solving developed by the researchers. As mentioned earlier, the GAIQ (data collection tool) was designed for professional development to help TAs reflect on their grading practices. However, it is designed in a way to provide feedback to professional development leaders/researchers and serves as a data collection tool on TAs approaches and attitudes related to grading student work. This means that the GAIQ had a dual role: both as a tool to help the TAs reflect upon and develop productive attitudes related to grading and also as a data collection tool which helped the researchers investigate TAs' grading approaches.

Zoom In Zoom Out Reset image size

The initial versions of the worksheets were designed using qualitative data from semi-structured interviews with faculty members based on an 'artifact comparison' approach [17]. In these interviews, faculty members were asked to make judgments about instructional artifacts which were similar to those they often use in their classes. The artifacts presented to the instructors during interviews were designed to reflect those that would be familiar to physics instructors. In particular, the three types of instructional artifacts were instructors' example solutions, student solutions, and problem types (e.g. problems in multiple-choice format, context-rich form, divided into sub-problems, with and without diagrams, etc). The artifacts presented to the instructors during interviews were designed to create a context which would activate beliefs that could influence decisions when they select instructional material or pedagogical techniques while teaching [17]. All of the GAIQ activities about instructors' solutions, student solutions, and problem types relate to a single core problem shown in figure 1 [17]. The core problem was designed, validated and approved by four physics instructors who taught introductory physics courses at the University of Minnesota and was used on final exams. The core problem was also sent to several other instructors of physics courses who reported that the difficulty of the problem was such that it required an average student to use an exploratory decision making process as opposed to an algorithmic procedure [17]. The problem involves synthesis of several fundamental physics concepts and principles. The problem included several features of a context-rich problem [17] (i.e. it was set in a realistic context, was not broken into parts, did not include a diagram, etc) and is rich enough to allow for interesting variations in students' solutions. Students could potentially solve the problem in different ways. Thus, the problem allows for a spectrum of more or less effective problem solving practices. The student final exam solutions were available, providing a source of authentic student solutions which were used both in reference [17] and in the present study. The specific artifacts involving grading activities included five student solutions (see an example of two student solutions in figure 2), which were based upon actual students' common answers in the final exam. The artifacts were chosen to reflect differences between expert and novice problem solving from the research literature such as including a diagram describing the problem, explication of sub-problems, justification of solution steps, evaluation of final answer, explication of the scientific principles used, etc [17].

Zoom In Zoom Out Reset image size

Instructors' responses to interview questions about the instructional artifacts revolving around the core problem and five student solutions were used to create the initial GAIQ worksheets including the worksheets on grading [63]. We note that the GAIQ is meant to take the place of individual TA/instructor interviews about the teaching and learning of problem solving. While the development and validation of the GAIQ was a very time-consuming process [17], the GAIQ requires significantly less time than interviews for data collection and analysis. Equally important, it avoids researcher intervention in the process of clarifying the interviewees' responses, and the inter-rater agreement on the coding of the data obtained and interpretation of the data is excellent. Thus, the GAIQ worksheets can be used by researchers and professional developers at different institutions to collect and analyze data and data across different institutions can readily be compared with relative objectivity.

The initial version of the GAIQ was iterated between the researchers and physics instructors and modified to a version which was administered in the context of professional development for Israeli pre-service and in-service teachers multiple times [64]. After each initial implementation and feedback from the teachers, the GAIQ was refined further until a version was developed that the researchers were satisfied with. The GAIQ tool was then adapted for a professional development program for physics teaching assistants in the United States. The TA professional development program in this study followed suggestions from prominent teacher educators to anchor professional development in collaborative reflection with peer instructors on classroom experiences [65–68]. Reflection on practice serves to enrich instructors' interpretations of classroom experiences, widen the inventory of possible actions instructors might use, clarify instructional goals and examine practice in view of these goals, and provide motivation for the adoption of new instructional strategies. Following these suggestions, the activities in the TA professional development program elicited TAs' initial ideas on different aspects of teaching and learning. Then, the instructor facilitated peer discussions about TAs' ideas, led class discussions in which the instructor provided ideas for 'best practices', and also provided opportunities for TAs to reflect on their ideas (for example, opportunities to think about discrepancies in their beliefs about teaching and learning and reflect on changes in their initial ideas).

The GAIQ including the grading activities were implemented in two different semesters of a TA professional development program in the U.S., and after each implementation, the researchers iterated the version several times between them. A postgraduate student researcher in PER observed the three semesters of the TA professional development program when TAs' worked on the GAIQ. The postgraduate student researcher and two of the authors (EY and CS) revised and iterated the GAIQ based upon the TAs' comments and responses. This validation process in the context of the TA professional development program ensured that TAs interpreted all components of the GAIQ appropriately as the researchers had intended.

The artifacts about grading have also been used as the basis of a previous paper on faculty members' grading practices [37]. In this study [37], faculty members were asked to solve the core problem (see figure 1) and compare and make judgments about two student solutions (see figure 2) to the core problem. These two solutions were chosen because they engender conflicting instructional considerations in assigning a grade [37]. In this paper, we present findings related to the same two student solutions (see figure 2) to enable comparison of the findings of the TAs and faculty members' grading practices and values [37]. We suggest that the readers examine the student solutions (see figure 2) and think about how to grade them. We note that incorrect aspects of the solutions are indicated by boxed notes. Both solutions arrive at the correct answer. SSD includes a diagram, articulates the principles used to find intermediate variables, and provides clear justifications. The elaborated reasoning in SSD reveals two canceling mistakes, involving misreading of the problem situation as well as misuse of energy conservation to imply circular motion with constant speed. On the other hand, SSE is brief with no articulation of reasoning, and it does not give away any evidence for mistaken ideas. However, the three lines of work in SSE are also present in SSD, suggesting that student E could have been guided by a similar thought process as student D.

The TA professional development program consisted of two-hour meetings held weekly throughout the fall semester. Three consecutive weekly sessions at the outset of the program revolved around the GAIQ involving grading activities. Table 1 shows the sequence of grading activities. The activities served as a learning experience within the professional development program as well as a data collection tool in order to study TAs' grading decisions and considerations in a simulated environment [54, 55]. Data regarding grading was collected twice, at the beginning (the second and third weeks) and end of the semester (the last class). Thus, prior to the first data collection, the TAs had only minimal teaching experience in recitations and labs.

The GAIQ sequence and particular worksheet questions are shown in table 1. The TAs completed a pre-lesson stage, in which they responded to the following questions in the form of an essay:

• (a)  
  What, in your view, is the purpose of grading students' work?
• (b)  
  What would you like students to do with the graded solutions returned to them?
• (c)  
  What do you think most of them actually do?
• (d)  
  Are there other situations besides the final exam and quizzes in which students should be graded?
• (e)  
  Does grading serve the same purposes for these situations?

In the pre-lesson stage of the GAIQ, TAs individually graded the student solutions for both homework (HW) and quiz contexts out of a total score of ten points, listed characteristic solution features, and explained why they weighed the different features to obtain a final score. The TAs were told to assume that (1) they were the instructors of the course and thus have ultimate freedom in structuring their grading approaches; (2) they had the authority to make grading decisions; and, (3) their students were aware of how they would be graded. An example response (transcribed) is shown in figure 3.

Zoom In Zoom Out Reset image size

During the in-lesson stage of the GAIQ (see table 1), the TAs worked in groups of three in which they discussed the student solutions and attempted to reach an agreement regarding how to grade them. Afterwards, a representative from each group shared their grading approaches with the entire class. Two of the authors (CS and EM) were present in the class. CS coordinated the class work and led a discussion at the end of the class which highlighted grading 'best practices'—i.e. grading approaches that promote effective problem solving. The instructor of the TA professional development program also noted the disadvantages of grading which focused exclusively on correctness. EM observed and documented the TAs' comments during the group and whole-class discussions.

The post-lesson stage of the GAIQ (see table 1) examined the effect of the individual activities, as well as the group and class discussion on TAs' perceptions about grading. The TAs completed an individual worksheet in which they related their initial features to twenty solution features that were defined based on the analysis of TAs' answers to the pre-lesson activity. They were also asked to consider changes in their consideration of features in re-grading the student solutions. This stage intended to allow TAs to rethink their choices as well as to allow the respondents to take part in the categorization of their responses regarding features by using a predetermined set of features.

All components of the GAIQ sequence shown in table 1 were completed by the TAs in the second and third weeks of the TA professional development program when the TAs had very little teaching experience. The end of semester task (see table 1) was administered to the second cohort of 18 TAs in the last class of the TA professional development program. The in-lesson stage of the end of semester task included the same essay and grading activity as in the beginning of semester pre-lesson stage. In addition, TAs were given copies of their pre-lesson activities from the beginning of the semester and were asked to reflect on how their grading approaches evolved throughout the semester.

Analysis. The pre-lesson stage of the GAIQ sequence asked TAs to grade student solutions SSD and SSE, list solution features, and explain their reasons for why they weighed the different features to arrive at a final score (see figure 3). Data analysis involved coding the features listed by TAs in the worksheets into a combination of theory-driven and emergent categories. Twenty-one solution features were identified. We made a distinction between features that were merely mentioned or weighed in grading. For example, in figure 3, the sample TA listed 'no figure' as a feature in SSE (solution feature 'figure' was considered 'mentioned'), but when assigning a grade, s/he did not refer to this feature when explaining how s/he obtained a score (solution feature 'figure' was not included in grading). Thus, the sample TA would be counted as mentioning solution feature 'figure' but not counted as grading on it. A TA who graded on a solution feature was counted as both mentioning and grading on it. For example, if the sample TA had not written 'no word explanation' in the feature column, the feature 'explanation' would have still been considered to be mentioned because this TA wrote 'there are no explanations in this solution' as a reason for why he/she weighed the feature to arrive at a final score he/she would assign to this solution (the feature would also be considered to be graded on). The coding was done by four of the researchers. In cases where disagreement occurred, this was usually due to vagueness in the wording of TAs' written statements. The researchers made use of the TAs' answers in the post-lesson stage (see table 1) to clarify vague statements made by TAs. After comparing codes, the researchers discussed any disagreements during multiple meetings until agreement better than 90% was reached.

To help interpret the data collected, the features were grouped into the 5 clusters shown in table 2. Each solution feature listed by a TA was entered into only one cluster. Cluster 1 (C1) includes features related to promoting effective problem solving practices [7–13] (i.e. initial problem analysis as well as evaluation of the final result). Cluster C2 involves features related to explication of reasoning (i.e. articulation and justification of principles). Features in C2 are written statements in the student solution that clearly invoke principles used to solve the problem and justify why the principles were used. Cluster 3 (C3) includes domain-specific features, such as invoking relevant physics principles and applying them correctly. Cluster 4 (C4) includes features related to elaboration which emerged during the coding process, e.g. 'written statements,' 'good presentation,' 'solution in steps,' and 'conciseness.' These features were not assigned to the 'explication' category C2 because they were unclear. Cluster C2 is focused on the explication and justification of the physics principles, whereas C4 is more about general communication of the solution. For example, we could not differentiate whether a TA who wrote 'written statements' meant that the student solution includes an explicit statement of a principle in writing, explicit justification of a principle in writing, or simply a written statement. Thus, we coded 'written statements' as belonging in the general category C4 elaboration. Similarly, if a TA says that a solution is 'organized' he/she could mean that the solution is neatly written or that it is systematic. Cluster C4 also involves solution features related to lack of elaboration, e.g. conciseness (this feature was mentioned most often by TAs when they graded the brief student solution SSE). Finally, cluster 5 (C5) focuses on correctness of algebra and final answer.

TAs' grading decisions were somewhat different in a quiz and a homework context, but we found little difference in the feature clusters they considered in these contexts. Thus, the findings presented relate to grading considerations for the quiz context (the percentages of the TAs who mentioned and graded on clusters in the homework are shown in the appendix, figure A1).

Findings (see figure 5). We found that for clusters C1 and C2, many more TAs mention these features than actually use them when assigning a grade. In particular, roughly 50% of the TAs mentioned features from the problem description and evaluation cluster (C1), but less than 20% considered these features in grading, regardless of whether they were present (as in SSD) or missing (as in SSE). Similarly, a larger percentage of TAs mentioned features involving explication of reasoning (cluster C2) than those who graded on explication. However, in contrast to the description and evaluation cluster that was treated similarly whether it was present (SSD) or missing (SSE), a larger portion (26%) of TAs took the explication cluster into account when they graded SSD (where explication was present) than when they graded SSE. Only 14% of the TAs graded on explication in SSE, where it was missing. Similar to TAs' considerations involving C1 and C2, more TAs mentioned features from C4.1 (i.e. explanation, written statements) than graded on these features (see figure 5). Very few TAs (∼10%) considered explanation and written statements in their grading of SSD, a solution which includes many written statements. Somewhat more TAs (∼30%) considered missing explanations and written statements when grading SSE. These data suggest that while TAs may be aware of features related to explication of reasoning and desired problem solving practices, they were not committed to grade on these same features.

Zoom In Zoom Out Reset image size

The cluster most often considered in grading was domain knowledge (C3). Over 80% of all TAs claimed that they grade on features related to domain knowledge (C3) in SSD, deducting points for physics concepts and principles that were inadequately applied. Additionally, approximately 50% of all TAs said they grade on correctness (C5) in SSD, deducting points for explicit algebraic mistakes. Fewer TAs said that they grade on domain knowledge (∼30%) or correctness (∼20%) in SSE, where no apparent mistakes were evident. These data suggest that TAs used a subtractive grading scheme, removing points from SSE for missing explanations (C4.1), but not weighing this cluster in grading SSD, where it is present. Similarly, TAs deducted points from SSD for lacking domain knowledge (C3) and correctness (C5), but did not weigh this cluster when grading SSE, where there are no apparent mistakes.

The data for the analysis of TAs' reasons for grading was collected in several parts of the pre-lesson stage of the GAIQ sequence. First, TAs discussed the purposes for grading in the essay they wrote. TAs were also asked to explain their reasons for weighing different solution features the way they did to arrive at a final score. Open coding was used [69] to generate initial categories grounded in the actual data. The coding of reasons and purposes for grading was completed by four of the researchers. After comparing codes, any disagreements were discussed and the categories were refined until better than 90% agreement was reached.

The reasons for the grade on SSD mostly reiterated the importance of the features included in C3 and C5 (physics knowledge and correctness) and very few TAs mentioned other reasons for scoring SSD in a particular manner. However, many reasons surfaced when TAs graded SSE and these are shown in table 3 (we do not show reasons for the grade on SSD since TAs mainly focused on physics knowledge and correctness). TAs' reasons for grading SSE included 'adequate evidence' (i.e. whether the solutions allowed the TA to understand the student's thought process), time/stress (i.e. on quiz students do not have time and are too stressed to elaborate their reasoning), and aesthetics (i.e. physics problems should be solved in a brief, condensed manner). Table 3 shows the number and percentages of TAs (total N = 43) who consider these different reasons when grading SSE in the quiz vs homework contexts. In the quiz context, nine TAs took the burden of proof of student understanding on themselves, stating reasons such as: 'SSE is brief, but I can still understand what was done' and 'the student obviously knew what he was doing.' In contrast, nine TAs mentioned that SSE contained inadequate evidence in the quiz context, stating reasons such as 'we cannot determine if he has fully understood the points of the problem' and 'it does not show if he/she is actually thinking correctly.' Six TAs noted that they would be lenient in grading the quiz because of the time limitations in a quiz context. Additionally, five TAs mentioned aesthetics as a reason for the grade on SSE in the quiz context, stating that they liked the conciseness of SSE. In the homework context, a larger number of TAs (N = 17) noted that SSE contains inadequate evidence of understanding. Thus, while little difference was found between the solution features mentioned and graded on in the homework and quiz contexts, TAs' consideration of evidence of students' thought processes and consideration of time limitations in a quiz may result in the SSE grade differences in the homework and quiz contexts (i.e. more TAs graded SSE higher in the quiz context as opposed to the homework context).

In the group discussions, many of the groups scored SSE highly (i.e. gave a score of 9/10 or 10/10). It was often the case that all three of the TAs in one group had previously given SSE a score of 10/10 on their individual worksheet in the pre-lesson stage of the GAIQ. As a result, in the group grading activity, all three TAs agreed on a final score of 10 for SSE. Other groups stated that there were disagreements in their group about the grading of SSE, and they could not come to a consensus. Furthermore, they were unable to suggest ways to resolve this conflict when reporting their group grading to the entire class. In summary, the group discussions did not result in acknowledgment of the roots of disagreements and resolution between different points of view.

We found that TAs' grading decisions (i.e. how they scored SSE and SSD) remained relatively constant. Similar to the beginning of the semester, at the end of the semester, the majority of the TAs graded SSE significantly higher than SSD (see figure 7). TAs who are above the diagonal line in figure 7 scored SSE higher than SSD. The gap in average quiz grade between SSE and SSD of the subgroup of 18 TAs became larger over the course of the semester (the average SSE score changes from 7.7 to 8.3 and the average SSD score changes from 7.0 to 6.6 from the beginning to the end of the semester). Thus, the TAs were somewhat more likely to focus on correctness as opposed to explication of reasoning in a quiz context when grading at the end of the semester compared to the beginning.

Zoom In Zoom Out Reset image size

We also found little change in the distribution of solution features mentioned and graded on by TAs. Regarding reasoning TAs used for assigning a specific grade, the TAs' stated reasons for the final grade on SSE remained approximately the same. We do not show reasons for the grade on SSD since TAs mainly focused on physics knowledge and correctness. Table 4 shows that there was a small increase in the percentage of TAs who stated that SSE does not give evidence of understanding in the quiz context after teaching experience and the professional development program (from 21% to 33% of the TAs). However, this did not translate into the TAs grading SSE lower, in fact, their average score on SSE was higher at the end of the semester despite a higher percentage of TAs noting the lack of evidence of understanding in SSE compared to the beginning of the semester.

Lastly, we found that TAs' general beliefs about the purpose of grading did not change significantly (see figure 8). The majority of TAs continued to state that grading is a means for students to learn from their mistakes. However, the percentage of TAs who stated that grading can serve as a formative assessment tool for the instructor decreased by approximately 20%. The number of TAs who stated that grading is a means to give a final grade (i.e. summative assessment tool) increased by approximately 20%. We speculate that their teaching experiences may have partly resulted in this change: while TAs may have initially believed that student solutions provide feedback to the instructor as to what difficulties are common among students, the TAs' grading experiences may have instilled in some TAs the belief that surveying student solutions to determine common difficulties is impractical given the amount of grading required. They may then believe that the purpose of grading is primarily to provide a learning opportunity for students and a means to assign students' grades for the institution. In summary, TAs' grading practices, considerations, and stated purposes for grading did not change significantly after a brief professional development intervention and one semester of teaching experience.

Zoom In Zoom Out Reset image size

Study limitations. The findings of this study are contextualized in a task mimicking a 'real' grading situation as closely as possible. However, since the TAs graded designed student solutions, the results are valid in this context and shed light on TAs' intended practice rather than TAs' actual grading approaches. In actual practice, TAs' grading approaches may become even more focused on correctness as opposed to encouraging desired problem solving approaches due to external factors, such as a large grading workload and a lack of control and involvement in the design of courses, in particular, homework, quizzes, and exams. In addition, even though TAs were told to grade the student solutions as the instructor of the course (i.e. that they are in control of the class and have told their students how they will be graded), it is possible that TAs' actual grading practices (when they are a TA in another instructor's class) conflicted with how they would like to grade and impacted their responses on the grading activities.

In the TA professional development program, we have also given TAs student solutions to a different context-rich physics problem that is isomorphic to the core problem in this study (see figure 1). One student solution was similar to student solution D in that it included many effective problem solving practices and showed reasoning and the other student solution was similar to student solution E in that it was very brief with only three lines of work. Although we do not discuss the details of this research in the present study, we found that TAs graded the analogous student solutions similarly. This suggests that TAs' grading practices are fairly consistent across analogous types of student solutions and that the way the TAs grade SSD and SSE in this study may provide some evidence about their grading practices in general.

The study was designed to portray TAs' practice and considerations, but further study is needed to learn what factors shape these practice and considerations. One possible factor is TAs' prior educational experiences. The study involves a common mixture of TAs [70] who had experienced undergraduate education in different countries (14 from the U.S., 17 from China, 12 from other countries). Prior research has shown that American, Chinese, and other international TAs perform similarly in identifying common student difficulties [71]. However, grading practices and considerations might be more sensitive to institutional cultures in different countries. We did find small differences between international TAs compared to those of American TAs, however, it is not possible to determine whether differences are significant between the groups due to the small numbers of each group⁷. Additional studies are needed to corroborate the results and draw more robust conclusions about TAs' grading approaches.