Multimodal expressions of disciplinary relevant aspects in inquiry-based physics learning

In this study we are interested in how learners of geometrical optics express disciplinary relevant aspects multimodally in an inquiry-based learning with support of a computer simulation. Students’ discussions and work with the simulation were filmed, transcribed, and we analysed some parts of the transcript where the students show signs of disciplinary relevant aspects by a method based in conversation analysis. We found evidence that the students use multiple semiotic resources for expressing disciplinary relevant aspects about the tasks, especially gestures. The letter are by their visual nature well suited for communication in physics, and are used by the students to transduct semiotic information.


Introduction
Physics is a subject that most students consider to be difficult, both at university level and in secondary school, and because of this numerous specific difficulties have been investigated in research [1].Teaching methods centered around student interactive-engagement have been shown to be superior to more traditional forms of education [2].As a result the main focus of method development has been on creating teaching situations where the students confront known difficulties and communicate with peers in order to solve problems and answer questions [3].
Favorable conditions for learning critical concepts in physics education thus occurs in situations where students have to express their understanding to peers in discussions.A large part of learning physics is to learn how to use the language properly, and communication occurs not only through spoken and written language, but includes other important means, such as nonverbal ones [4].

Theory
In this study we focus on the learning of disciplinary relevant aspects of optics with support of a computer simulation.We investigate how students communicate disciplinary relevant information to each other through different modalities, and we employ a semiotic perspective for our analysis.Semiotics deals with the meaning of signs [5], which are to be understood in the most general meaning possible as combining conveyed meaning with form through a semiotic resource, the meaning potential of any material resource apt for the task.Semiotic resources are GIREP-2022 Journal of Physics: Conference Series 2750 (2024) 012004 IOP Publishing doi:10.1088/1742-6596/2750/1/012004 2 used for communicating meaningful expressions, utilizing not only spoken and written words, drawn figures and other permanent material artifacts, but also more transient ones such as gestures, body language and different aspects of the voice, to mention a few [6].The meanings carried by expressed signs in different modes are negotiated socially, and if this is applied to the sciences, it follows that meaning of any signs will depend on consensus in the group of relevance, as well as what scientific discipline or even sub-discipline that is in question.
In general, most attention in studies has been paid to the obvious modes, such as written and spoken words, as well as drawings, partly because these modalities convey their meaning in a clear and undisputable way.More nebulous, and open to interpretation, are the nonverbal modalities gestures and voice, both which can convey a multitude of information which to a much higher degree is dependent on the interpretation of the receiver [7].We are here mainly interested in how both verbal and nonverbal cues are coordinated spatially, temporally, as well as with regards to content, to communicate information that is relevant in disciplinary terms.We focus on disciplinary aspects that are of importance to the problem the students are working on.The voice, excluding the verbal meaning of the spoken words [8,9], can be expected to complement expressions [10,11] in a coordinated manner together with the gestures.
Gestures are often categorised according to their use for transducting semiotic content [12], and for our purposes here two categories suffice; namely deictic and iconic gestures.Deictic gestures are used for drawing attention to specific spatial locations, i.e. mostly ponting and showing where something is or occurs.Iconic gestures are in themselves a representation of something, a way to portray a concept, either abstract or concrete.
In social semiotics the focus is on the agreed meaning-making potentials of signs.Applied to physics, central concepts are taken to have multiple aspects [13,14] that are expressible in multiple ways.Of chief importance are those aspects that the physics community implicitly has agreed to be disciplinary relevant, the ones that must be mastered in order to understand the concept [15].Here we narrow down our focus to the question of how learners express disciplinary relevant aspects in educational situations, since multimodal expressions of disciplinary relevant aspects give clues to students' thinking about the physics at hand in their effort to negotiate an understanding and also to express a disciplinary acceptable solution on their own terms.
The students in this study were novices in physics and as a result their knowledge of physics vocabulary as well as the concepts of physics is restricted.This naturally affects how the students verbally express their understanding of the disciplinary content of physics.The fact that many students at this early stage in their formal education lack a complete understanding of the words of physics language means that the situation in many ways is similar to how students of a foreign language have a limited vocabulary while learning the new language.As a result students in both disciplines can compensate for their missing verbal knowledge by using other semiotic resources in a communication situation.One crucial difference between physics and language learning is that in physics the students have no lack of words in their native language outside of physics.In physics they only have knowledge from possible earlier education or other experiences in the subject at hand.The lack of words in a foreign language might be compensated by the student through the use of other modalities in conversations to express known concepts, assuming enough of a universality of body-language.The students in this study, however lack a full grasp of the disciplinary relevant concepts, as well possibly the words to express the concepts in the language of the physics.To express their thoughts in the discussion they can, and as we will show do, compensate by the use of other modalities such as gestures [16,17,18,19], to a high extent.From the discussion above it follows, that to gain an understanding of the student's expressions and possibly also of the thinking behind them, a multimodal analysis is necessary.

The study
In this study we used software from the Constructing Physics Understanding project (CPU) by Goldberg et al [20].We developed our own inquiry-based tasks for the study, and based the activities on a prediction phase when the students worked individually, which was followed by work in pairs with the simulation to construct answers the tasks at hand.The entire sessions were filmed, including the time students responded to the tasks individually.Before the recorded sessions, the tasks were tried out in a pre-trial with a different, but essentially similar, student population, after which there were minor revisions of the specific formulations of the different tasks.All the tasks were presented to the students in written form, and one of the researchers was present to answer questions while the students worked with the tasks, and the researcher could initiate or encourage a discussion between the students by asking questions to make them express their thoughts about the different concepts in the task.
The results reported here come from one out of a sequence consisting of five learning sessions.The students may or may not have studied optics at elementary school level, but at the time of this study they had not covered it in their secondary level curriculum.The students were allowed to form the pairs they worked in by themselves, and their peer was in all cases a fellow student familiar to them from before the study.The goal was to investigate the students' thinking and understanding of vision, reflection and other basic properties of light by observing the interactions between the students and with the simulation.The experimental setup was constructed with the intent to encourage students to use different modalities in a very explicit manner as they were asked to respond to the question in both written and spoken words.The discussions between the peers, and also with the researcher, were filmed and the position of the camera made it possible study the body language of the students, such as gestures, in addition to their verbal expressions.
Here we only report results from the work with one of the tasks, which is shown in figure 1, and which was chosen to give the students ample opportunities to express certain chosen disciplinary relevant aspects of optics.The task is focused around how human vision works and how basic optics is relevant for an understanding of the functioning of the human eye.The symbols used in the task are fairly straightforward and could be considered to be known by the students, and where in addition also explained to them by the monitoring researcher when needed.We here present material from one pair of students, who we here individually give the labels student A and student S.This pair of students was chosen as they had a disagreement over how the task should be solved, and this lead to their interactions producing a wealth of data on how they express themselves.the important disciplinary aspects of change of direction and continuity, both before and after the change of direction, are exhibited.Finally, for vision to be possible to occur in the eye, it must be understood to function as a receptor, or more technically as a detector, implying that the rays must reach the eye.Regardless of what specific mental explanation model the student chooses for the other parts of the task, the solution must contain an unobstructed path for light all the way from the lamp to the eye.This task gives the students multiple opportunities to express their thinking and reasoning about important concepts in geometric optics.
Learning new concepts involves discerning the relevant aspects [21,22], which here involves specific features of the semiotic resources used.To learn optics requires the student to become able to discern the important aspects of the physics in the task, which in turn means that the task must be constructed in a way which makes it possible to do this in a fruitful way.One of the most fundamental concepts of geometrical optics that is present in this task is that of light-rays.In the learning of light-rays the students have to learn certain aspects of light rays such as linearity, directionality, and continuity.As a part of this, and the learning progression of the student it is vital that the student become able to discern the relevant aspects [23] of the light-source and have a grasp of the role of the eye in vision.

Transcription
The films were transcribed in detail using the framework of conversation analysis [24].The purpose was to achieve a transcript rich in detail, including evidence of expressions using multiple semiotic resources.An example of a transcript is shown in figure 3, which was chosen Figure 3.An extract of the transcript used, together with transcribed detail according to conversation analysis using the Jefferson transcription system.The additional detail allows us to get a deeper and more objective picture of the interaction that goes on between the two students and also with the reseracher.In the transcript pauses(lines 1, 2 & 3), latches(lines 1 & 4), vocal emphasis(line 4) and gestures(line 4) are explicitly coded and timecoordinated, as is shown by the annotations.as a short representative example of a discussion occurring when the students worked with the simulation for the task shown in figure 1.The semiotic resources of gestures, voice, speech of the students give valuable information about students' reasoning, and show us how how the students express subject matter.For this purpose, we chose to use the Jefferson transcription system, a standardized and well-established tool for systematic work, and often used when the aim is to achieve a detailed transcript.In this way we aim to minimize subjectivity in the transcription process and in an objective way show all the details of how the students use different semiotic resources.This also makes it easier to establish how the semiotic resources are coordinated to express more complicated aspects of the subject matter.The main advantage of using the Jefferson transcription system is that it focuses on the conversation itself, and not on analysis of the content of the conversation, thus avoiding the mixing of transcription and interpretation at the transcription stage.It also gives the researcher the opportunity to faithfully produce and present a record of the communication situation with a range of different semiotic resources included.The transcript also visualizes for the researcher how the use of the semiotic resources differs between different parts of the conversation.The details in the conversation are made explicit in the different modalities and how they are temporally coordinated in an explicit way in the transcript.

Analysis
Here we will first present an example of a gesture showing how student express semantic content, before going in to a deeper analysis of the transcript.If we first look closer at the gesture shown in figure 4, we see that it is performed with a pen the represents points, locations where the rays propagate towards the screen.The temporal sequence of the finer gestures in figure 4 also shows how iconic aspects are expressed simultaneously with the deictive aspects of indicating locations, as the directions of how the pen is moved trace out the rays, the invisible objects the student has in mind.Student S stressed the positions verbally with her mentioning them with emphasis as the pen was moved from position to position where the rays are traced out from different locations.Next we turn to an example of a transcript from the session under consideration, which is shown in figure 3, which corresponds to part of the session for the task shown in figure 1.In the discussion there is a disagreement between the two students A and S regarding the best way to solve the task.The main issue concerns whether there occurs a reflection in the screen or not The detailed description of the conversation included in the transcript exhibits pauses in the turnovers, an example of which can be seen in line 1 where the researcher makes a pause for 2.5 seconds.Right at that point the researcher starts a dialogue summarising that there obviously has surfaced a difference between how the two students think about what happens to the light at the screen.In line 1 the researcher starts by rephrasing what student S has expressed.
Further information about the details in the conversation is shown in the transcript, such as for instance, there is a latch between the first utterance and the second utterance shown in line 2.This means that student S is very quick to start responding to the researcher's question, so that there is not really any pause in the turnover between the different speakers.In line 2 student S considers the researchers utterance, and the pause can be interpreted as time taken to consider and think about the optics content in the utterance.After a pause the student approves of the researchers statement.The turnovers themselves also become explicit in this transcription with additional details such as the latching of lines 1 & 2. Since there is a disagreement between the two students, the researcher, as is shown in line 3, turns to student A to obtain that students' answer.In line 4 student A responds and clarifies the answer using two gestures to express her opinion.The content of the provided verbal answer shows that student A understands the important aspect that a diffuse reflection occurs, and the gestures add valuable spatial information about the detailed understanding of how the reflected rays propagate on the screen before and after reflection.
In the discussions with the supervising researcher the students use various semiotic resources to express different aspects of light-rays, for instance, student A says that reflection means "bounce off", on line 4 in the transcript shown in figure 3. Student A expresses the aspect of reflection verbally and combines this with a gesture, which is shown in figure 5.By using the gesture, the concept of reflection is illustrated by the student by having the fingers represent the light rays.The movement of the hand and fingers indicates that the student has a grasp of the aspect that the light-rays must be continuous.
It is also possible to see when a person is emphasizing words in their utterance, as in line 4 where student A puts an emphasis on the words "bounce off".This utterance is temporally synchronised with two different gestures, where the second one is shown in figure 4. In line 5 the researcher just confirms student A:s utterance in line 4. Line 4 and 5 form an adjacency pair in this discussion.
Line 4 is combined with the gesture that show the reflected rays, which is shown in figure 5.The combined gesture and its temporal synchronisation with the emphasised utterance is a good example of the importance to study the multimodal expressions.This is because, in the verbal expression, the meaning of reflection is not explicit without the gesture.The fingers represent the physical rays, and the movement of the fingers show the direction.The student shows explicitly that rays propagate in different direction, spreading from the screen.To express what happens to the light at the screen in detail, there is a change from using the tip of the pen early in the gesture to instead using the fingers as a way to show the diffuse reflection, expressing additionally that the rays do go in different directions from the point of reflection.The student highlights this with the change between the gestures, i.e. that something special occurs at the reflection.The reflection in the screen affects the light, and student A adds a verbal emphasis when saying that reflection means bounces off.The gesture in figure 5 is performed simultaneously with the verbal expression with a slight acceleration of the hand at the words bounces off.This gesture is iconic as the fingers are used to represent the light rays.The spatial direction of the fingers shows the direction of the light-rays on the screen, and there is a spatial coordination of the gesture with the location of the rays on the screen.The verbal expression doesn't include the spatial part of the utterance, but the light-rays by necessity include a direction in the room, which is disciplinary relevant for the solution of the task.
The figures 4 and 5 provide us with example of gesture that students use to complement their verbal utterance.We see here that gestures are a semiotic resource the students use throughout their interactions to express and clarify the semantic content [7] of what they say.

Discussion
We have in this study investigated how learners of introductory geometrical optics express disciplinary relevant aspects multimodally with the use of different semiotic resources.
A series of inquiry-based tasks with disciplinary relevant aspects of optics in focus were developed with the aim to investigate the resulting interactions between the students.All task included computer simulations that the students worked on together in pairs during the collaborative parts of the sessions.The sessions were filmed and conversation analysis was used to get a detailed transcript and to discern different semiotic resources.We found that during the exercises both students use multiple semiotic resources for expressing and clarifying the semantic content.
Gestures is one example of a semiotic resource that is used on multiple occasions by the students to express different aspects of optics, as can be expected for students solving problems collaboratively [16].Figures 4 and 5 show two examples of gestures that occurred during the discussion when the students were trying to find a mutually agreed upon solution to the task.The gestures shown in figure 4 are deictic and the student uses the pen to point out directions or points on the screen.The gestures used by the students were in general found to be temporally synchronized with the verbal expression, as can be seen for example line 4 in the transcript shown in figure 3 where the actions corresponding to figure 5 are transcribed.
The session studied in detail here turned out to be particularly fruitful as a source for multimodal expressions of disciplinary relevant aspects since the two students disagreed on central aspects of the posed problem.A careful analysis of the film and details in the transcript shows how students express disciplinary relevant information through temporal and spatial coordination of their multimodal expressions.This effort to communicate through coordinated multiple semiotic resources can be seen as a purposeful step solving a physical problem [25], for instance by achieving more precision and clarity in situations where the abilities to practise/use the formal language of physics is limited with the novice students.
The importance of a multimodal analysis lies in the window it opens to follow the student thinking during physics learning and is a fruitful method that can be expected to shed light on details through piecing evidence together.Further work is required to follow the chain from multiple resources to expression in further detail to investigate how the expressions are used in problem solving and reasoning about physics.

Figure 1 .
Figure 1.An illustration of one of the a tasks given to the students to work with.To the top left in the illustration there is a light source and beneath it the eye, and to the right there is the screen.The dashed horizontally oriented line shows the normal for the screen.The question posed was to explain what the eye in the illustration sees when the light-source is turned on, and to motivate why they think this occurs.

Figure 2 .
Figure 2. The figure shows a disciplinary correct solution to the task presented in figure 1, i.e an explanation to what the eye in the illustration sees.The solution show the disciplinary relevant aspects of physics, such as the properties of light-rays, diffuse reflection, as well as the role played by the eye.

Figure 2
Figure2shows an adequate solution to the task in figure1, including the main disciplinary relevant aspects.In the task, the light-source to the left is the origin of rays, and the rays spreads linearly in all directions from the light-source.Only rays relevant for the human vision are shown here.The light-rays drawn in a solution must show the aspects of linearity, directionality and continuity.In the screen to the right a diffuse reflection occurs, and at the reflection points

Figure 4 .
Figure 4.The figure shows a gesture made by student S. The gesture shown here is an example of a deictic gesture where the student uses the pen to point out distinct points on the screen, combined with iconic aspects indicating how the rays propagate from each point.The information of the gesture is to point out the location and to support the verbal expression where each location is essential, and is stressed verbally to add emphasis.

Figure 5 .
Figure 5.In this figure a gesture is shown where the fingers serve as a physical representation of the lightrays diffusely reflected from the obstacle.The movement of the hand shows the direction of the light and its hypothetical movement.This an example of an Iconic gestures.The spread of the fingers indicate that rays go in different directions and the fingers represent the physical lightrays.