Development and didactic analysis of smartphone-based experimental exercises for the smart physics lab

This study was pre-piloted and tested with the SPL04 tilting smartphone exercise (see figure 1) during the winter term 2020/2021. A quasi-experimental field study with a CG and IG setup with pre- and post-tests to measure the learning achievement was chosen. The selection and composition of the control and intervention groups as well as the study design proved to be suitable. The results of the pre- and post-tests showed a positive influence on the learning achievement for the experimental exercise. Based on the results of the pilot phase, the pre- and post-test contents were revised in order to improve their validity and reliability. In addition, the scales for measuring interest and motivation via surveys in the beginning and at the end of the intervention were changed from a six-point to a four-point Likert scale for the follow-up study [16].

The presented study took place during the winter term 2021/2022 with first-semester students on the experimental physics 1 course (mechanics). It was designed to be similar to the pilot phase and was carried out for two successive experimental exercises, SPL03 and SPL04, highlighted in red in figure 1. A total of 85 students participated in all pre- and post-tests. The IG consisted of the first semester students enrolled in our physics teacher trainee courses. The CG consisted of the first semester bachelor of science physics and bachelor of science meteorology students.

The timeline of the study is shown in figure 2. At the beginning (τ₀), the demographic data of the two cohorts were collected via a self-report. During the same survey, a questionnaire was used to measure the motivation and interest of the students at the beginning of their study via a four-point Likert scale. The measurement of these two variables was based on existing, valid test instruments [17, 18].

Zoom In Zoom Out Reset image size

The topics investigated during the two intervention phases by the IG were the analysis of circular motions during the interval $[\tau_1,\tau_2]$ with the SPL03 and the dynamics of rigid bodies during the interval $[\tau_3,\tau_4]$ with the SPL04. For each topic, paper–pencil tests were developed in which students in the CG had to apply and link the same physical concepts as part of their weekly homework.

Figures 3(a) and (b) show sketches and measurement data of both interventions. In the IG (task SPL03), students were asked to determine the position of the micro-electromechanical systems (MEMS) accelerometer within their smartphone by rotating it in two different orientations. They were required to measure the centrifugal acceleration dependence on the angular velocity and determine the distance of the MEMS from the axis of rotation. Finally, a coordinate transformation to the frame of reference associated with the edges of the smartphone yielded the required coordinates. In a sample solution (see figure 3(a)), the smartphone was mounted on a door leaf in two different orientations where it was allowed to move on a circular path with a fixed axis of rotation. In the IG, SPL04 students analysed the tilting motion of their smartphone. Using the conservation of energy and Steiner's theorem, students determined the moment of inertia of their smartphone from the maximum of the angular velocity measured with the internal gyroscope just before it met a horizontal soft mat (see figure 3(b)). They were required to compare the experimental results for the moment of inertia with the values calculated under the assumption that the smartphone is a homogenous cuboid.

Zoom In Zoom Out Reset image size

The main difference between the smartphone-based experimental exercise and the pencil–paper test of the CG was that the students in the IG performed the experiment with their own smartphone and used self-generated data to evaluate physical models, whereas for the students in the CG, independent experimentation was omitted and the solution needed to be derived via a calculus-like algorithm in the cookbook principle. The exercise text for both groups can be found in the supplementary material (available online at stacks.iop.org/PhysEd/57/045038/mmedia).

Before the students received their respective exercise sheets, pre-tests $(\tau_1~\mathrm{and}~\tau_3)$ were conducted to measure the prior knowledge level of both groups regarding each topic (see figure 2). After the experimental and the paper–pencil homework were submitted by the students, post-tests $(\tau_2~\mathrm{and}~\tau_4)$ were given to both groups to measure the learning achievement for each of the two subject areas. Each test consisted of nine single-choice questions for intervention 1 (analysis of circular motion) and of 10 single-choice questions for intervention 2 (dynamics of rigid bodies). The concept questions of the pre- and post-tests were derived from validated test instruments [19–21] and adapted from questions from textbooks [22–24], respectively. Our own developments of concept questions were validated during the pre-pilot phase.

At the end of the study $(\tau_5)$, a final survey was conducted to measure changes in student motivation and interest again on a four-point Likert scale. In addition, questions were asked about the cognitive load of the respective exercises via a six-point Likert scale [25]. The cognitive load test instrument used allowed us to distinguish between three types of cognitive load [25]. The intrinsic cognitive load results from the exercise difficulty, its complexity, and its requirements in terms of time and content. The extraneous cognitive load depends on the didactic design of the task. It describes the load on the working memory that is irrelevant to learning and influences the structuring of the knowledge transfer and the presentation of the learning content. If it is too high, there are too few cognitive resources available to achieve the learning content. With the germane cognitive load, the free cognitive resources are designated, which can be raised for real knowledge construction in the working memory.

For measurement of the learning achievement between pre- and post-tests, the Hake index g was determined [26]. It is calculated from the percentages of correct answers in the pre-test p_pre and the post-test $p_{\mathrm{post}}$. The Hake index describes the ratio between the reached learning achievement $p_{\mathrm{post}} - p_{\mathrm{pre}}$ and the maximum possible learning achievement $1-p_\mathrm{pre}$. For evaluation of the data collected via the surveys and the concept tests, covariance analyses including the factor group membership or t-tests were utilized.

A summary of demographic data for the IG and CG is presented in table 1. With more than 70%, the majority of the members in both groups are less than 22 years old. Thus, both groups consist mainly of students participating for the first time in a university course of experimental physics. With respect to their secondary school education in mathematics and physics, i.e. the participation in advanced courses and their averaged A-level degrees received, there are also no significant differences between the IG and the CG. This is indicated by the p values in table 1, which were obtained via a t-test comparing both groups and which are all found to be p > 0.05. In summary, the CG and IG are similar with respect to composition and to performance of their members in mathematics and physics from secondary school education.

Table 1. Demographic data for the intervention and control group. The last four lines refer to the courses taken and A-level degrees achieved.

	IG	CG	Significance p
Number of participants	40	45	—
Female students	15	25	—
$\leqslant$age 21 years	71%	75%	—
Advanced course mathematics	80%	73%	0.08
Mathematics A-level degree	$11.03 \pm 4.11$	$11.13\pm 3.72$	0.56
Advanced course physics	55%	54%	0.41
Physics A-level degree	$11.50 \pm 3.98$	$10.25 \pm 4.46$	0.11

Table 2 provides an overview of the number of items for each of these variables and their reliability. The reliability (Cronbach's α) of the scales is always $\alpha \geqslant 0.70$. Thus, the items and the scales applied in this survey can be considered acceptable [27]. By analysing the variables of motivation and curiosity prior to the first intervention (τ₀), no systematic group differences were found (see τ₅ table 2). In both groups, motivation and interest in studying physics remained at approximately the same intermediate level during the post-intervention survey at time τ₅. There are several approaches to explain this. The students in the IG have been working with the smartphone-based tasks since the beginning of the semester. Thus, the novelty effect, which often increases curiosity and motivation, partially expires by the time of the intervention. In addition, our experimental exercises are at a higher academic level compared to similar studies [7, 8]. Thus, the difficulty level and the pressure situation of the required exam admission might have limited a considerable increase in motivation and curiosity.

Table 2. Development of motivation and curiosity over the course of the study. Average $\overline{x}$ and standard deviation σ from the four-point Likert scale survey as well as the t-values and significances p of the statistical analysis of the data for both groups. The applied scale ranges from 1 (high) to 4 (small).

			IG		CG
	Time	Items	$\overline{x}$	σ	$\overline{x}$	σ	t	p	α
Curiosity	τ0	7	2.15	0.81	2.11	0.84	0.24	0.42	0.77
	τ5	8	1.97	0.68	1.82	0.69	0.97	0.16	0.75
Motivation	τ0	6	2.09	0.72	1.99	0.68	0.63	0.26	0.73
	τ5	7	2.25	0.74	2.09	0.73	0.97	0.17	0.70

Table 3 shows the comparison of average and standard deviations of the Likert-scale data for the three types of cognitive load for both groups after successful completion of all tasks in this study (τ₅). The reliability of the scale of the seven test items used is with α = 0.78 also acceptable [27]. It can be seen that the intrinsic and the germane cognitive load do not differ for the two groups. However, the extrinsic cognitive load differs significantly for both groups (see p < 0.05 in table 3, line 4). The CG value of 3.1 is lower than that of the IG of 3.61.

Table 3. Comparison of the three types of cognitive load. Average $\overline{x}$ and standard deviation σ from the six-point Likert scale survey, as well as the t-value and significance p of the statistical analysis of the data of both groups. The applied scale ranges from 1 (high) to 6 (small).

		IG		CG
	Items	$\overline{x}$	σ	$\overline{x}$	σ	t	p
Intrinsic load	2	3.06	0.94	3.18	1.03	0.55	0.29
Extrinsic load	3	3.61	1.33	3.12	1.13	1.80	0.04
Germane load	2	2.03	0.56	2.01	0.73	0.13	0.45

The comparison of intrinsic and germane cognitive loads for both groups after the two interventions shows that the smartphone-based exercises are as demanding for the IG as the traditional paper–pencil tests are for the CG. This means that the task-based measurement and analysis process does not impose such additional loads on the IG learners. For the IG, the extrinsic cognitive load is lower than for the CG with the paper–pencil tests. Obviously, the experimental exercises allow a more elementary access to the physical concepts of the respective topics than the traditional pencil–paper tests. Thus, learners benefit from the use of smartphone-based exercises and can be more conducive to learning.

Figure 4 compares the proportions of correct answers for both groups in the pre- and post-tests for both interventions. For the analysis on circular motion topic during $[\tau_1,\tau_3]$, 64% correct answers were given in the pre-test of the IG. At 54%, the figure is lower for the CG. In the post-test, the probability of a correct answer increases to 71% for the IG and to 59% for the CG, respectively. Obviously, both groups were able to increase their performance. However, the CG achieved a slightly greater increase in learning than the IG.

Zoom In Zoom Out Reset image size

The proportions of correct answers for both groups in the pre- and post-test for the intervention the dynamics of rigid bodies topic during $[\tau_3,\tau_4]$ shows that the IG was able to increase its proportion of correct answers from 72% in the pre-test to 76% in the post-test. The CG was also able to improve its results, but the increase here was only from 61% to 64%.

To investigate the learning achievement, the Hake index g_i was calculated for each participant in the study from data from the pre-test and the post-test. Table 4 presents the averaged learning achievement $\overline{g}$ for both groups and topics. For the IG, the learning achievement is larger than for the CG in both interventions. The calculated learning achievements were additionally examined by means of covariance analysis (p < 0.05). The only factor taken into account was group membership. In addition, Cohen's d was calculated as an effect size measure. It was found that for the dynamics of ridged bodies topic the learning achievement of the IG working with the smartphone-based exercises is significantly higher compared to the achievement of the CG solving a traditional paper–pencil test (see table 4). A medium effect size of d = 0.42 was calculated. For the analysis of rotational motion topic, there is, however, no evidence for a significant increase of a higher learning achievement for the IG compared to the CG. The effect size for this intervention is with a Cohen's d of d = 0.20 considered to be small.

Table 4. Comparison of the learning achievement $\overline{g}$ for both groups and both exercises. The significance p and the effect size Cohen's d of a covariance analysis are also given.

	IG	CG
	$\overline{g}$	$\overline{g}$	d	p
SPL 03: analysis of rotational motion	0.19	0.11	0.20	0.37
SPL 04: dynamics of rigid bodies	0.28	0.08	0.42	0.04

These data agree with Mazella and Testa's finding [10] that smartphone-based exercises are an attractive alternative to the traditional paper–pencil tests. The main difference between the compared exercises is that the students in the IG use self-generated experimental data and evaluate them accordingly, while the students in the CG do not have to experiment themselves. The close theory–experiment interplay supports the knowledge transfer for the IG. The learning results show a positive trend in favour of the presented smartphone-based exercises. For physics courses at universities, they confirm the results of Hillmayr et al [9] and Hochberg et al [7] that smartphone-based exercises have a positive effect on physics learning.

The effect size of our smartphone-based exercises strongly depends on the topic and the task itself. The analysis of rotational motion topic (first intervention) is already discussed during advanced courses in secondary school. Thus, in contrast to the dynamics of rigid bodies topic (second intervention) it is not completely new for the learners on our courses. The experimental problems for the IG and the paper–pencil test for the CG are perhaps too similar in the first intervention and the IG learners were not triggered enough. Thus, careful task and exercise design also remain a challenging task for smartphone-based experimental exercises.