The measurement of statistical reasoning in verbal-numerical and graphical forms: a pilot study

Numerous subjects have trouble in understanding various conceptions connected to statistical problems. Research reports how students' ability to solve problems (including statistical problems) can be influenced by exhibiting proofs. In this work we aim to contrive an original and easy instrument able to assess statistical reasoning on uncertainty and on association, regarding two different forms of proof presentation: pictorial-graphical and verbal–numerical. We have conceived eleven pairs of simple problems in the verbal–numerical and pictorial–graphical form and we have presented the proofs to 47 undergraduate students. The purpose of our work was to evaluate the goodness and reliability of these problems in the assessment of statistical reasoning. Each subject solved each pair of proofs in the verbal-numerical and in the pictorial–graphical form, in different problem presentation orders. Data analyses have highlighted that six out of the eleven pairs of problems appear to be useful and adequate to estimate statistical reasoning on uncertainty and that there is no effect due to the order of presentation in the verbal–numerical and pictorial–graphical form.


Introduction
In recent research we observe a strong interest in statistical reasoning and in judgment making in uncertainty situations [1,2]. Generally the concept of statistical reasoning is mentioned as the manner in which individuals reflect on statistical concepts and form conclusions on statistical data [3]. Garfield [2] suggests that six kinds of reasoning (on data, on representations of data, on association, on statistical measures, on samples, and on uncertainty) can be recognised in statistical reasoning. Each type of reasoning can be elicited by specific activities related to data comprehension, analysis, and communication. Specifically, the reasoning of association is important in statistics because this process is used to identify statistical relationships between two or more variables [2]. Reasoning on uncertainty is defined as the ability to apprehend and apply concepts of probability, casualness and chance, such as the capability to make conclusions with uncertainty, and with an awareness that not all results are obtained in the same way [3,1]. But many studies show that individuals make mistakes when they are reasoning on association and on probabilities [4]. Subjects appear not to be familiar with statistical association and probabilistic information [5,6]. For example, when individuals try to use uncertainty information, they tend to incorrectly estimate the probability of events; this occurs for both expert and non expert individuals [7]. Referring to these aspects, the use of several visual aids could appear as an important method for clarifying and improving reasoning on uncertainty and on association [8,9]. Green [10] states that graphs could elicit an in-depth understanding of data complexity. On the other hand, Knauff and Johnson-Laird [11] affirm that visual aids could impede probabilistic reasoning in some problems, because they could overload the cognitive system with additional information.

Aim and method
This work represents the middle stage in our research which investigates the use of imagery in statistical reasoning, in order to detect the effect of graphical facilitation [12]. In this phase of the research we aim to explore statistical reasoning on uncertainty in association with appropriate proofs, in relation to two different forms of proof presentation: verbal-numerical and pictorial-graphical. We appoint a measure instrument of these articulated dimensions, which presents several pairs of the analogous proof; in this way we can compare, in the same individual, the statistical reasoning developed in comparable problems presented in two forms. We intend to analyse the reasoning in subjects without statistical skills, in order to evaluate the real contribution of the presentation method in the development of statistical reasoning. Then, we investigate reasoning on uncertainty and on association, with reference to basic mathematical abilities learnt at high school. Therefore we decided to exclude from our assessment those types of statistical reasoning which require a specific statistical knowledge.
In this pilot study we apply specific statistical analyses in order to detect the adequacy of proofs in this investigation. Moreover, in our work we aim at evaluating the potential effect of the order of presentation between proofs in verbal-numerical and pictorial-graphical form.

Participants, instrument and procedure
Participants consisted of 47 undergraduates at the University of Cagliari (Italy), attending the first year of the degree course in Psychology. The average age of the participants was 22.2 years (SD = 5.7). The group consisted of 32 females (68.1%) and 15 males (31.9%). Data collection took place in December 2012. The sample involved students who voluntarily participated in the research and the sampling was non probabilistic. We submitted for resolution eleven pairs of analogous items in paper format, to a large group in a classroom.
Each item was given to the same student to be solved using two modalities: verbal-numerical (form A) and pictorial-graphical (form B). Each problem necessitated an open response, where the subjects were required to explain the reasoning applied in the resolution. We analysed the students' answers as given to each problem and then in each pair of problems and we coded the correct and the incorrect answers (1/0). In the Appendix we show some exemplificative pairs of items in two dimensions.
Double forms of problems were presented in two separate sessions (one week apart), in different orders of subgroup students. We analysed the students' open answers and then we coded the responses as correct or incorrect. We computed the descriptive statistics the Bivariate Chi Square, the McNemar test, the Discrimination index, the Difficulty index, and the Alpha Reliability.

Results
In Figure 1 we show the percentages of correct responses in the two forms in each pair of items of two dimensions: uncertainty (pairs 1,2,3,4,5,6,7) and association (pairs 8,9,10,11). By this figure we can observe that our participants frequently had a better performance in the resolution of problems in the verbal-numerical form of each pair of proofs.
Next we applied a Bivariate Chi Square for each problem, in order to detect the effect of the order of presentation (verbal-numerical / pictorial-graphical; pictorial-graphical / verbal-numerical) in the correct/incorrect answers of each item (see Table 1). The sequence of the order of proof presentation did not have an effect on the number of correct responses; indeed we did not find significant effects by the application of these inferential statistics.  In assessing the dimension of reasoning on uncertainty for the verbal-numerical form we obtained an Alpha Reliability =0.556 (St. Alpha =0.553) and for the pictorial-graphical form an Alpha Reliability =0.656 (St. Alpha=0.666). Then, in evaluating the dimension of reasoning on association for the verbal-numerical form we had an Alpha Reliability = 0.357 (St. Alpha = 0.320) and for the pictorial-graphical form an Alpha Reliability = 0.416 (St. Alpha= 0.432). We also calculated the indices of Difficulty and Discrimination in order to evaluate the goodness of each proof (see Table 2 and Table 3). The reliability of the two dimensions was low; particularly in the dimension of reasoning by association some statistics appear inept in order to pursue our aim to construct an evaluation instrument of statistical reasoning.

Discussion
This research presents a pilot study in order to develop a measurement instrument to assess undergraduate students' statistical reasoning on uncertainty and on association, in relation to methods of proof presentation. By the construction of paired items in two forms, we could compare the reasoning applied to problem resolution, with regard to the specific problem structure.
With evaluation of the statistical analysis, we could presume that the dimension of reasoning on association did not have good indices; therefore these pairs of items could be discarded by our questionnaire [13]. In relation to the reasoning on uncertainty, we could assume that one item (FB 4) did not have a good Discrimination Index, therefore it is not appropriate for this measurement [13]. We could also assume that the pair corresponding to this item could be discarded by the instrument, using the remaining six couples for the assessment of reasoning on uncertainty. Moreover, in order to improve the agility and the promptness of this instrument, we could prepare a new version characterized by closed response options for these six pairs of items; then each problem could be followed by an open question, inquiring as to the reasoning implemented in the proof solution. In this way we could reduce the solution time of the proofs and simultaneously investigate the cognitive resolution process applied.
Furthermore we have highlighted that the proof disposition could not affect the problem resolution. Nevertheless, in order to avoid and control this contingent effect, it could be useful to balance the two forms of proofs by a casual disposition of problems in the random sub-samples of students.
The results of this work could be useful regarding the intention to develop statistical teaching and to project appropriate learning experiences in order to improve statistical reasoning.

Appendix
Some exemplificative pairs of items in verbal-numerical format and pictorial-graphical format

Reasoning on uncertainty -VERBAL-NUMERICAL FORMAT
A factory produces electronic games, but not all work well. Of every 100 game products: 20 may have an electrical problem; 80 can function correctly. The company has developed control systems to identify faulty games; however, these systems do not work properly. In reality, half of the games with electrical problems continue in the production line where they are considered as functional.
If you randomly extract a game that has been addressed to the shops and evaluated free of defects, what is the probability that it is defective?

Reasoning on uncertainty -PICTORIAL-GRAPHICAL FORMAT
A factory that produces personal computers has problems in the production process. Some of the computers are defective (problems with the video card). Such problems are not always identified by the quality control and consequently some defective computers are sent forward in the production line. The graphic below shows this process. What is the probability that a computer addressed to the shops, evaluated free of defects, is defective?

Reasoning on association -PICTORIAL-GRAPHICAL FORMAT
A and B are two variables in the relationship between them. On the basis of the graphical representation below, can you identify the type of mathematical relationship between the two variables?