Profiling context-based mathematics tasks developed by novice PISA-like task designers

Designing context-based mathematics task has been becoming one of current research topics of final research projects of many student teachers in Indonesia. This study aims to characterize context-based problem developed by student teachers in their final research project within their studies. Data were collected from 124 items of mathematics tasks designed by fourteen student teachers from seven universities in Indonesia submitted to the authors within a developmental research. The characteristics of tasks were measured in terms of three dimensions of problem generation (plausibility, sufficiency of information, and complexity), task profile (content, context, and process), level of context use, task format and language issue. Results point out that 88.2% of the items were plausible, 70.5% has sufficient information, in which most of them have lower-level cognitive demands (procedures without connections). Analysis of task profile shows that the items were distributed into the context of personal (26.37%), occupational (25.27%), societal (36.26%), and scientific (12.09%); the content of quantity (35.56%), change and relationship (27.78%), space and shape (20.00%), and uncertainty and data (16.67%); the process category of formulate (18.89%), employ (48.89%), and interpret (32.22%). The items were identified to have problem with camouflage context and have less number of open-constructed items. Regarding language issue, the tasks designed mostly meet the problem of having too much information, language ambiguity, the use of unfamiliar terms, and the use of unspecific unit of contexts.


Introduction
The emergence of mathematical literacy has been widely admitted as one of interesting issues in mathematics education discussed deeply by many scholars and policy makers both theoretically and practically. It is due to the need for individuals to have such ability to encounter today's complex problem. The existence of mathematical literacy has been empirically maintained by OECD (Organization for Economic Co-operation and Development) through its global survey report from one of its assessment programs, i.e. PISA (Programme for International Student Assessment) to assess students' mathematical literacy internationally once in every three years [1]. See the reports in years 2003,2006,2009,2012 at http://www.oecd.org/pisa/. To assess, PISA employs instrument in the form of mathematics task which has some unique characteristics which essentially follow the nature of mathematical literacy described in its framework [1]. The increased attention on mathematical literacy implied on the increased emphasis on tasks promoting connections and applications of mathematics to the real world had been called by researchers in many countries since 1990s [2][3]. In Indonesia, particularly, this emphasis has attracted many researchers and policy makers to develop curriculum and its supplementary materials which support the existence as well as the sustainability of this type of task in school mathematics learning. Some effort has been done, for example, by providing teacher professional learning concerning on the quality improvement of context-based task designed by teachers [4], providing mathematics books compiling more number of context-based problems (see for example [5]), supporting teachers in bringing contextbased task into their daily teaching [6][7] and encouraging research topics on developing context-based task like in PISA task by student teachers as their final projects [8]. The latter emphasis, specifically, becomes a step stone for novice PISA-like tasks like student teachers to take parts as future mathematics teachers or mathematics educators which could encourage students develop their mathematical literacy.
As novice PISA-like task designers, student teacher finds many challenges. The challenges might not only about making problem plausible to solve and has sufficient information, but also making problem authentic in which the real world included can be modeled mathematically satisfying the PISA mathematics framework [9][10]. Zulkardi and Kohar [10] added that the challenges also around the use of accessible language and the level of cognitive demand which meets the criteria of higher-order thinking skill task. Furthermore, PISA task developers are required to design context-based task which is rich and interesting for 15-year-olds around the world, neither too hard nor too easy, do not has any seemingly artificial questions, and as much as possible equally accessible and equitable for students of different gender, culture, religion, living conditions [11].
It has been asserted that mathematical tasks, including context-based task, play a critical role in the teaching and learning of mathematics [12][13]. Furthermore, they argued that teachers' goals, mathematical knowledge, and knowledge of students can influence how the teachers set up the tasks included in intended curriculum materials, such as textbooks. This indicates that students' mathematics learning can be positively influenced by tasks teachers pose through their content, pedagogical, and contextual knowledge. Therefore, the issue of competency of teachers as well as student teacher as future teacher to design context-based tasks needs to be addressed more actively in current research trend in teacher education.
This study aims to report the profile of context-based task like in PISA task designed by Indonesian student teachers. The profile included the task characteristics based on PISA task profile as mentioned in PISA mathematics framework [1] and the quality of tasks regarding authenticity of context, plausibility of the task, and the complexity of tasks.

The Nature of Context-based Task in PISA Framework
Context can be defined as the aspects of the real world in which a mathematical problem is embedded [14]. In a wider perspective, context may also be referred to not only real-world settings but also fantasy situations or even to the formal world of mathematics [15]. This interpretation shows that contexts are not restricted to real-world settings. The crucial thing is that contexts create situations for learners experienced as real and related to their commonsense understanding. The 'context' meant in this study follow the latter interpretation since the task created by student teachers in this study can be either in realworld setting or non-real world setting although they were actually expected to design context-based task following the most current framework of PISA task [1], in which authentic real world context is required. Thus, context-based task in this study is defined as mathematics task which employ situations either found in real world or non-real world setting which make sense with learner understanding.
The issue of authenticity of the context-based task in PISA is a major concern of item writing. It depicts the basis for finding situations and tasks from various countries based on authentic stimuli and then compiling a series of questions that someone wants to answer, or based on information in the stimulus [16]. The level of authenticity can be then derived from the extent to which the level use of context is embedded in the task designed. In this regard, Salgado [17] reconceptualised context level by drawing on the idea of de Lange's work, categorizing context as zero-order, first order, and second- order of context. Zero-order use of context provides for direct actions or inferences to be made from the instructions given in a mathematics problem, but the context is not used to interpret mathematical results or arguments. First-order use of context allows for relevant information, variables or relationships to be identified and/or selected for the mathematical formulation of a problem, with the context also providing for the adequateness of mathematical results to be determined. Meanwhile, second-order use of context allows for relevant variables, relationships or assumptions to be defined and/or retrieved for the mathematical formulation of a problem, with the context being used to judge the adequateness of mathematical results and arguments in relation to the original problem. Sawatzki [18] asserts that the higher level of context used in a task, the more aspects of real world and potential mathematics required to be involved during solving such a task.
PISA provides some profile for each of items released for the purpose of international survey as the characteristics of PISA tasks. The profile encompasses domains of context, content, and process. Regarding context category, PISA covers four types of context: personal, occupational, societal, and scientific. Regarding content category, PISA mentions four types as well: quantity, change and relationship, space and shape, and uncertainty and data. In relation to process category, a PISA task can be either 'formulated', 'employed', or 'interpreted'. The item format of PISA task is also restricted into three types, i.e. selected responses/multiple choice, closed-constructed item, and open-constructed item. The examples of how this profile is given, the readers may visit PISA released items 2012 [19].

The Quality of Context-based Task
Every problem posed by a learner/teacher can be assed through three dimensions: plausibility, sufficiency of information, and complexity [20]. An implausible problem is one that contains an invalid assumption. Therefore, is not plausible to solve even with more information. If a posed problem was plausible, the designer of the problem would then determine whether there was sufficient information to solve the problem or not. In this case, task with extraneous information were coded as having sufficient information since they are solvable. Complexity of task, on the other hand, refers to the linguistic complexitywhether the word problem involved propositions with an assignment, a relational and/or a conditional structure [21], the semantic complexity-combine, change, compare, and equalize structure for addition and subtraction word problems [21].
Complexity can also be assessed through the level of cognitive demand of a task. In this regard, Smith and Stein [22] gives the categorization into four, as follows: (1) Lower-level cognitive demands (memorisation). Routine exercises that involve the memorisation of formulas, algorithms, or procedures, and without connection to the underlying concepts or meaning. (2) Lower-level cognitive demands (procedures without connections). Tasks that are algorithmic and focus solely on describing the procedure that was used. Little ambiguity exists about what needs to be done and how; and there is no connection to the concepts or meaning that underlie the procedure being used. (3) Higher-level cognitive demands (procedures with connections). Tasks that focus on the use of broad general procedures for developing deeper understating of concepts and ideas. These tasks are usually represented multiple ways and require a degree of cognitive effort to complete the task successfully. (4) Higher-level cognitive demands (doing mathematics). Tasks that require complex and nonalgorithmic thinking to explore and understand the nature of mathematical concepts processes and relationships. These tasks may involve some level of anxiety for the student because of the unpredictable nature of the solution process required. In this study, the tasks designed by student teachers were assessed by following Smith and Stein' s idea of cognitive demand level. Because the tasks are context-based, the quality is not only assessed through plausibility, sufficiency of information, level of cognitive demand, language structure, but also the extent to which the task designed meets the criteria of PISA framework: level of context authenticity and the appropriateness of the context-based task posed with PISA task item profile.

Research method
This is an explorative descriptive research which analyze the characteristics of context-based task designed by Indonesian student teacher within a developmental study in their final projects of their undergradute or graduate studies.

Participants
As many as 14 student teachers who were completing their final research project on developing contextbased task using PISA framework from five different universities in Indonesia took part in this study. The participants were asked to submit their prototype to the authors of this paper.

Data Collection
Data were collected from the items submitted by the participants to the authors. Thus, there were a total of 124 items. This is not assess who is better in designing PISA context-based task. Rather, it is to draw on patterns of all the items based on the criteria the authors determined in this study. Within their development research, they designed around 8-15 items of tasks. No matter what analytical framework of developing those items, all the student teachers admitted to have the stage of expert reviews, in which the prototype comprising those tasks and their supplementary instruments were submitted to some experts, which in this case is lecturers of mathematics education or PISA tasks designers having experiences in developing PISA context-based task, including the authors of this article. Table 1 shows the number of tasks designed by the student teachers

Data Analysis
Data were analysed by counting the frequency of items regarding the categories of each domain of task profile. Table 2 shows the categorization of context-based task examined for each item. Item profile refers to four domains: context, content, process, and item format, while the quality of task refers to domains: plausibility, cognitive demand, and level of context use. To examine the reliability of the coding, the authors applied Cohen's Kappa for each of the codings from the two raters (the second and the third author). To compute the Cohen's Kappa ( ), a statistics software, namely SPSS 22 were employed, which resulted score ranging from 0.782 to 0.924 for all categories of task profile. See table 3.  , 1977). To report our analysis, the coding of the second author was then selected to be further analysed. To discuss more deeply the results of counting the frequency based on table 1, the items of context-based task were also qualitatively evaluated based on the unique findings from the items, such as regarding the issues of language use, the presentation of task, the context use, level of difficulty, and other relevant issues. Table 4 summarises the profile of context-based task designed by student teachers.  Table 4 points out that the student teachers tended to select personal (26.37%) and societal (36.26%) context for their posed problem. With regard to content domain, they tended to select space and shape more frequently than other content categories, while they also selected problem with concern on 'employ' process (48.89%), which accentuate mathematical facts and/or procedures within solution process. Furthermore, closed-constructed item was chosen as the most frequent item format (61.36%) for their posed problem.

PISA profile for context-based tasks
The following table shows the examples of context-based task with different categories for each domain posed by the teachers.  Table 5. Examples of profiling context-based task posed by student teacher

Examples
Item profile PULSE RATE Pulse is the number of times your heart beats per minute. Normal heart rate varies from person to person. Pulse rate has a maximum limit. Maximum pulse rate is pulse which can occur when doing maximum activity. Many researchers formulate maximum pulses. Here are two researchers who formulated a maximum pulse.

DOUBLE DISCOUNT
A shop gives a double discount for every purchase of a shirt. The first discount of 40% is given to the initial price, while the second discount of 25% is given to the price after being deducted by the first discount. If the initial price of the shirt is Rp. 500,000, then the final price of the shirt is ...

Source: S-14
 Personal  Quantity  Employ  Closed constructed item Assessing the quality of context-based task designed by the student teachers, table 6 indicates that basically they were indicated to pose plausible problem (88.2%), although few tasks (11.80%) were found to have insufficient information. Regarding cognitive demand, most of the context-based tasks were identified to have lower-level cognitive demands (50.00%), in which single procedures are required, while only few tasks which had higher level cognitive demands (4.35%). In relation to the use of context, the tasks posed by the student teachers were mostly identified as tasks with zero-order use of context task and first order-use of context. This result indicate that the student teacher found problem with creating authentic context. Table 7 examplifies context-based tasks found from student teacher which have been grouped based on the level of context use. Table 7. Example of context-based tasks regarding their quality posed by student teachers

Tasks
Item profile

Fish Cultivation
Mr Rudi is a catfish farmer who has two large ponds. During the harvest season, the highest number of catfish is produced from the area of the pond.
If the fish pond is illustrated in the two figures below. a) b) Which of the two ponds has the larger area? Give your explanation! source: s-4  Plausible problem with sufficient information  zero-order use of context  Lower-level cognitive demands (procedures without connections)

PRAMBANAN VISITORS
The line chart below shows the number of visitors of prambanan temple in a day whom were noted per hour from 8.00 a.m to 4.00 a.m presenting the number of visitors entering and leaving the temple region. In ancient times, the barrel was one of the reservoirs of water for ablution before there is an electric water pump that drains water from the well to the faucets ablution of a mosque. The following is a picture of the ablution barrel.
Which one of the following graphs shows the change in height of the ablution barrel model when the water outlet is opened and the water is full.

Source:S-8
 Plausible problem with sufficient information  Higher-level cognitive demands (doing mathematics)  Second-order use of context The findings indicate that student teacher has problem concerning on the authenticity and structure of language. These findings indicate that the authenticity of the context/situation of the problem is an important consideration for PISA-like task designers. To increase the authenticity of the context, the experience of authors in changing the context of 'camouflage' to a 'more authentic' context suggests that the student teachers change the problem situation which seems to only test students' math practice skills with situations where someone really has to solve the problem. Other points of findings related to the profile and quality of tasks are described by reporting unique findings from the responses given to particular task designers. For example, the following table describes the points of the authors' review results written through email for S-4's context-based task. Table 8. Comments related to the profile of PISA-like tasks designed by student teacher

Review results
Main issue 1. In general, many interesting contexts are presented in this prototype. The context is also fairly balanced in terms of categories. It's just that there isn't yet an "occupational" context. It can be considered to design questions with this type. Likewise, from the content side is also quite balanced. Context is very possible to be developed into a variety of questions again with different PISA content. 2. Problems raised by PISA are authentic problems in everyday life that need to be resolved. For this, PISA uses a type of context that is not camouflage, the context needed to do mathematical / modeling, and the context used to solve everyday problems so that the context is not separated from the mathematics content being tested (Read use of context orders (OECD, Jan De Lange) In this prototype, I found a number of questions in the context of camouflage, for example, why did Selin have to use walking speed in the Easy Health Walk category when leaving home to find out the time needed by Selin to get to school? 3. General information that is under the context title should be made in sentences / information that are as effective as possible so that students can read it easily. It would be better if only the relevant information was presented to be used in solving the problems in the context unit. 4. Each problem solving in each question unit in the same context in the PISA question is not permitted to be related to each other. That is, to answer the problem there is unit 2, it is not necessary to refer to the results of the answers to the unit problem 1. All that is permitted is to refer to the general information that appears at the beginning before going to the problem units. I found that there was an answer to unit 2 in the context of Basal Metabolic Figures that are directly related to unit 1 in the same context. 5. It is necessary to add questions that really belong to the "Formulate" category where students themselves determine what mathematical models should be used, or in other words, questions from authentic real contexts with many forms of different ways / answers, problem solving strategies which are diverse, and require higher cognitive demand and quite complex mathematical reasoning and argumentation.

Potential rich context
Authentic context, instead of camouflage context Clear presentation of information Independent item within one unit of context More higher-order thinking items Point 1 discusses the distribution of task designed based on the PISA domain categories. The problem that student teachers tended to design tasks which only for particular categories of context, content, or formulae indicates that student teachers might have few experiences on designing mathematical problem with those categories. In this study, we found, for example, there were only few items with scientific context and content of uncertainty and data. The causes of this finding may be investigated from how 10 student teachers deal with such two categories. In our point of view, the limited number of scientific context-based task is that the nature of scientific knowledge itself, in which it is not enough to simply consider theories of the nature of explanation and the difficulties of searching for authentic data which comes from trusted source which keep the authenticity of the situation of he information in the task. Also, it needs proper understanding explanation which requires grasping the relation between explanation and understanding, as well as how explanations can lead to scientific knowledge [24]. Another finding related to the distribution of categories of PISA domain, we also found that the student teachers seems gave irrelevant profile, primarily regarding content categories. This is found from the tasks posed by S-3, in which some of her items should be categorised as quantity, instead of change and relationship. In this regard, PISA stress that while categorisation by content category is important for item development and selection, it is also important to note that some specific content topics may materialise in more than one content category [1]. This is because the nature of PISA task which relies on contextual setting, in which more than one branches of mathematics are employed.
Regarding point 2, the use of camouflage context seems become crucial issue for the student teachers like many other findings on teacher or preservice teachers (e.g. Siswono, Jurnal te edu, Kohar n zu). Simultaneously with the S-4's result of review in relation to this matter, we summarise the types of situations of the context-based taske posed which cause the context of those task are camouflage. First, some tasks were identified to have irrelevant situations which impossible to occur in real world. For example, one of the tasks posed by S-5 used the situations of a car route in which the race track is round. Incorrectly, S-5 provides the information about the decrease of a car speed constantly without considering the significant decrease of the car speed when it arrive at a turning point of the track. Second, some tasks put the context as ' a wrap' of the problem in the task, in which if the context is changed with another situation or story, it does not influence the problem solution of the task. For example, the task with zero-order use presented in table 7 indicates that if the context of 'fish cultivation' is changed with other situation: area of farm, for example, the solution process of finding the larger area can still be found with a standard mathematical procedure.

Conclusion
To conclude, the student teachers in this study hs tried to design context-based task which meets PISA criteria. However, the authors found that they found problem with context authenticity of the tasks they designed. This is indicated by the few number of tasks with second-order use of context and have less number of open-constructed items. Regarding language issue, the tasks designed mostly meet the problem of having too much information, language ambiguity, the use of unfamiliar terms, and the use of unspecific unit of contexts.