Developing and Validating the Assessment Instruments to Measure Students’ Analytical Thinking Ability and Chemical Literacy on Colligative Properties

The purpose of the study was to develop and validate the assessment instruments utilized to measure students’ analytical thinking ability and chemical literacy on colligative properties. This study was research and development (R&D) implementing the ADDIE model. However, this study was limited to the validation of products. The assessment instrument was developed based on indicators of analytical thinking ability and chemical literacy and contained 15 items of questions as the result. Based on the validation result conducted by 3 validators, it was found that the average score of validation was 94% for the material; 91% for the language, and 91% for the construction. Moreover, the validation of the assessment of analytical thinking ability and chemical literacy was 93.33%. Thus, this assessment instrument was valid and feasible to be used in order to measure the students’ analytical thinking ability and chemical literacy on the colligative properties.


Introduction
Education is crucial to determine the advancement of a nation in order to adapt to the rapid changes and progress in science and technology [1]. The goals of education are to train knowledgeable and skilful students to be independent, effective, and creative [2]. This is in line with the goal of education proposed by [3], that is to teach students to think and solve problems in order that the students can think, question, research, and formulate answers to a problem. Good education not only prepares students to continue learning to a higher level but also solves challenges and problems faced by students in daily life or at work.
A curriculum, as a renewal instrument in science education, is the main plan for organizing teaching and learning [4]. In Indonesia, the 2013 curriculum was revised for answering the tasks and moves in the development paradigm. The curriculum is a plan about the subjects the students have to study in education in Indonesia (Kurniasih, Imas & Sani, 2014). This curriculum is designed to prepare young generations to have the capability to live with faith, to be productive, creative, innovative, and effective, and to contribute to the society, nation, state, and world civilization [6]. (Vanada, 2014) states that in dealing with the 21st century challenges students need to improve their collaborative, creative, analytical, and practical thinking skills included in higher-order thinking skills (HOTS). Furthermore, high-level thinking skills nowadays are not only the development of facts and concepts, but a student also have to possess the ability to solve problems in daily life [8].
According to [9], the knowledge and technology are changed rapidly which influence the society. Thus, understanding the truth of science and the association of knowledge, technology, and society are required. The ability to apply knowledge in everyday life is known as scientific literacy. [10] assert that scientific literacy becomes one of the goals in science education. Scientific literacy requires standards of content, pedagogy, and assessment. PISA (Program for International Students Assessment) explains scientific literacy as "scientific knowledge and utilizing the knowledge to classify problems, to obtain knowledge, to clarify scientific phenomena, and to demonstrate scientific issues by using scientific confirmation such as laws, principles, and concepts. Those with genuine scientific literacy will understand the main features of science as a system of human knowledge and scientific inquiry. They also realize the way science and technology create the material and intellectual and cultural environment, as well as willing to contribute to science-related events as reflective citizens [11].
Based on a test conducted by PISA, it was revealed that the scientific literacy of the Indonesian students was very unsatisfying. The literacy ability of Indonesian students was still below the average and the ranks were also still low. The data showed that Indonesia was ranked 64 of 72 member countries of the Organization for Economic Cooperation and Development (OECD) in 2015. This result indicated that the scientific literacy achieved by the students in Indonesian is below the average. Moreover, in 2018, the results of PISA showed a decline in the ranking of scores, in which Indonesia was at the rank of 70 from 78 member countries, meaning that this rank declined from the previous rank in 2015 [12]. These facts emphasize that indeed the Indonesian students have low scientific literacy abilities. Thus, it is crucial to develop their higher cognitive abilities (Eka and Endang, 2018). According to Santhitiwanich, Pasiphol, and Tangdhanakanond (2013), teachers need a certain test instrument to evaluate and assess students' higher order thinking skills (analytical) as well as their scientific literacy.
In addition, [13] argue that a key of the success of one's intelligence is analytical thinking ability which is a part of higher order thinking skills [14]. Meanwhile, analytical thinking ability is one of the abilities the students should optimize in learning in the 21st century. It is the basic ability to reach the level of thinking such as evaluating (C5) and creating (C6). This thinking ability makes students be able to solve the issues such as situations, problems, ideas, and so on into parts related one another and to the entire purpose [15] [16].
[17] say that assessing the learning outcomes still focus on low-level thinking skills (LOTS). Increasing the education quality should be in line with the employment of assessments appropriately assessing the final results of a learning process. Besides, educators' ability to develop test instruments certainly affects the students' learning outcomes. By utilizing an assessment instrument that meets the criteria, the students' learning outcomes will obviously be evaluated properly and it can be used as an evaluation material for further learning programs. Achieving the goals of chemistry learning requires an assessment instrument which not only covers memorization and understanding but also requires assessments that train higher-order thinking skills. Some items usually have an emphasis on HOTS such as thinking critically and analytically, and solving problems. Chemistry teachers need to be motivated to support the students to do assignments and tasks involving HOTS that are self-developed [18].
Based on the interviews conducted with several chemistry teachers participated in MGMP forum in Bengkalis, it was found that not all teachers developed their own assessment instruments. The instruments used by teachers were mostly adopted from various sources. The teachers never developed an instrument to evaluate the students' analytical thinking skills and chemical literacy especially on colligative properties. They rarely considered the level of thinking of students as desired on basic competencies. It was found that from the analysis of the daily assessment the material about colligative properties, the questions in some items were at C1 -C3 levels. In fact, the basic competencies in the 2017-revised 2013 curriculum state that learning colligative properties requires the learning of HOTS or at the level of analysis (C4) and colligative properties contains many basic concepts, facts, phenomena, and mathematical calculations. Assessing the cognitive aspects of learning outcomes can be conducted better through an instrument of assessment that can assess analytical thinking skills and chemical literacy so that the basic competencies set out in the 2013 curriculum can be achieved properly. The development of indicators of learning achievement on colligative properties should be adjusted to 3 the elements of analytical thinking ability and chemical literacy. As a result, a valid item indicator will be produced and can measure both aspects of the ability.According to [19], the ability to think analytically is related to someone's chemical literacy, meaning that a person having chemical literacy is capable to ask, to seek information, and to analyse the causal relationship. Based on the explanation above, developing an integrated instrument to assess students' analytical thinking ability and chemical literacy was necessary. In this study, researchers developed and validated the instruments utilized to assess students' analytical thinking skills and chemical literacy on colligative nature.

Design and Model
This study was research and development (R&D) intended to produce an instruments for assessing cognitive aspects in order to assess analytical thinking skills and chemical literacy on colligative properties. The development model applied in this study was ADDIE model which consists of the stage of Analysis, Design, Development, Implementation, and Evaluation. This research is relevant to research conducted by Prastiwi and Laksono (2018). However, this study was limited to the analysis, design, development, and validation stages from expert judgement. For more details, the development method can be seen in Figure 1. Stage 2 Design, the researcher designed the method of assessment instruments. The design was started from defining an integrated indicator (the integration of the basic competency indicators, analytical thinking ability, and chemical literacy), which was used as a recommendation in developing the instruments. Then, the instrument was equipped with a scoring technique; therefore, the design of the scoring technique was defined based on the form of the instrument to be developed. Stage 3 Development, at this stage, the researchers developed the product that had been designed in the previous stage based on several aspects, such as good assessment criteria, the adjustment of instruments with learning material, and the range of material taught. The instrument was developed to be an essay writing test. These instruments including the assessment instrument of analytical thinking skills and chemical literacy were validated by experts based on the aspects of readability/ language, material/ content, and construction. The validator provided advice and inputs on the instruments in order to improve a valid instrument. As a result, a valid instrument could be used to assess the students' analytical thinking ability and chemical literacy at class XII on colligative properties

Techniques of Data Analysis
The techniques of qualitative data analysis were used to analyze the indicators of colligative properties, analytical thinking ability, and chemical literacy based on the results of validation from 3 experts. In this study, the data analysis was explained descriptively. The analysis stage began with the validation by experts regarding the aspects of readability, construction, and suitability of the content/ material. The validation was conducted by giving a questionnaire to the experts. Then, the validators were also asked to provide suggestions and inputs to revise the deficiencies in the instrument. The results of validation were analyzed using Aiken's V. If the value of Aiken's V was lower than the minimum value of Aiken's V, the item was considered invalid. The items that were declared valid were revised based on the notes provided by the validators. The next step was testing the instrument to the high school students. The formula is presented below = Note: V = Validity Aiken S = rlo lo = the lowest validity rating score c = the highest validity rating score r = score given by validator n = number of validators The validation of the analytical thinking skills and chemical literacy assessment was measured by utilizing Guttman scale which consisted of two choices: "yes" or "no". This was performed by giving checklist on the choice by expert validators. If the expert validators chose "yes", he got a score of 1 and if he chose "no", the score was 0. In addition, there was also a space for the expert validators to write inputs/ suggestions on this validation sheet. The Guttman scale is displayed in Table 2. After that, the collected data were analyzed by using descriptive statistical analysis techniques. The data were processed through numbers in the form of descriptive percentages. The calculation to get a percentage from the expert validators' ratings used the formula: The decision making about the validity of analytical thinking ability and chemical literacy instrument on the colligative properties was that if the percentage was ≥ 50%, it would be classified as good. On the contrary, if the percentage was ≤ 50%, it would be classified as not good (Riduwan, 2012).

Analysis phase
Stages of developing the instrument started with an initial analysis and the needs of teachers and students. After the learning process, the teacher conducted an assessment of students by using an instrument adopted from various sources so that it did not consider the demand of the competency in the 2013 curriculum. Analysis of the daily test questions on the colligative nature of the material used by teachers from several schools in Bengkalis showed that the questions used were less training high-order thinking. The questions used at the cognitive level of C4 are only 7.5%, C3 is 27.5%, C2 is 32.5% and C1 is 32.5%. As a result, the students' thinking process could not be assessed because the teacher did not analyze basic competencies that must be achieved. One of the basic competency included in colligative properties was analyze (C4), which means that the assessment of learning outcomes should use instruments that asked for high-level thinking / HOTS. Meanwhile, the materials about colligative properties was very contextual with the phenomena of everyday life. Thus, it was necessary to use instruments in accordance with basic competencies in the curriculum content at a high level of thinking and integrated with chemical literacy.

The Analysis of Basic Competence and Material Domain
The analysis of competencies and material domain was carried out by selecting basic competencies (KD) that would be developed for the assessment instruments.

The Analysis of Analytical Thinking Ability and Chemical Literacy Indicators
In this study, indicators of analytical thinking ability were differentiating, connecting, and organizing according to Ramirez & Ganaden (2008). 1) Differentiating means to distinguish relevant/ important parts from irrelevant/ unimportant parts; 2) Connecting refers to determining the standpoint or intention underlying the existence of an element; and 3) Organizing refers to determining how an element is in accordance with its structure. In the other hand, the indicators of chemical literacy used in the development of the instrument were based on [10], which consisted of 3 aspects of literacy abilities, namely: 1) Describing phenomena through chemical concepts; 2) Utilizing th chemical understanding to solve problems; and 3) Investigating the approaches and advantages of chemical applications (Endang and Sadhu, 2018). The analysis of analytical thinking ability and chemical literacy indicators produced an integrated indicator that is displayed Table 3.

Indicators of Chemical Literacy
Integrated Assessment Indicators (Analytical thinking ability + Chemical Literacy)

Differentiating
Explaining the phenomena by using chemical concepts 1. Linking the phenomena by using chemical concepts 2. Distinguishing phenomena by using chemical understanding to solve problems 3. Organizing chemical problems by using chemical understanding 4. Linking the phenomena by using strategy analysis and the benefits of chemical applications.

Organizing
Applying the chemical understanding to solve problems Connecting Investigating approaches and advantages of the chemical applications

Design Stage
At this stage, the questions were developed based on the basic competencies on colligative properties, indicators of competency achievement, and the analytical thinking ability and chemical literacy indicators. In this study, there are 15 indicators developed for the assessment instrument. Those indicators are described in Table 4.

Table 4. Blueprints of Assessment Instrument on Colligative Properties
No Assessment Blueprints 1 Able to connect the nature of colligative properties to the number of particles by using the concept of colligative properties 2 Able to distinguish the vapor pressure phenomenon by using the concept of colligative 3 Able to connect mole fraction and the decrease of vapor pressure by using chemical understanding to solve the problems 4 Able to distinguish the phenomenon of the increase in boiling point and the decrease of the solutions' freezing point by using chemical understanding to solve problems 5 Able to relate the phenomena of osmotic pressure and substance concentration by analyzing the benefit strategies of applying colligative properties 6 Able to organize the decrease in saturated vapor pressure, increase in boiling point, the decrease of freezing point, and osmotic pressure by using chemical understanding to solve problems 7 Able to connect the increase in boiling points and levels of substances by analyzing the strategies and benefits of applying colligative properties

No
Assessment Blueprints 8 Able to distinguish the meaning of colligative properties of electrolyte and non-electrolyte solutions by implementing the concept of colligative properties 9 Able to connect the number of particles of electrolyte solutions based on the phenomenon of osmotic pressure by analyzing the strategies and benefits of applying colligative properties 10 Able to organize the colligative properties of electrolyte solutions by using chemical understanding to solve problems 11 Able to organize chemical problems by using chemical understanding 12 Able to organize chemical problems by using the understanding of colligative properties of electrolytes 13 Able to distinguish the phenomenon of boiling point of electrolytes by using chemical understanding to solve problems 14 Able to connect the phenomenon of using chemical concepts by using the concept of colligative properties 15 Able to relate the phenomenon by using strategy analysis and the benefits of chemical applications The instrument to assess the ability to think analytically and chemical literacy was designed in form of essay questions. This instrument was specifically designed to measure the students' analytical thinking skills and chemical literacy. As a result, the ability level of the students could be measured. To do so required a scoring guideline based on the indicators of the assessment instrument. The design of guidelines for scoring assessment instruments are displayed in Table 5. Students can answer the questions by using concepts / theoretical studies correctly and clearly 3 Students' answers are approaching concepts / theoretical studies 2 Students' answers are less related to the concept / theoretical study 1 Students' answers are not in accordance with the concept / study theory 0 Cannot answer the question at all

Development Stage
Based on the problem analysis and needs analysis results, it was revealed that developing an instrument to assess the students' analytical thinking skills and chemical literacy on colligative properties was required. The first step in drafting an assessment instrument was to compile an instrument matrix as a basis for developing the question grids in accordance with the KD. The question grids contained basic competencies, indicators of competency achievement (IPK), levels of knowledge, analytical thinking skills and chemical literacy indicators, item indicators and item numbers, answer keys, and scoring guidelines. The instrument was an essay since it could assess the students' ability according to the indicators. This is in line with Khan and Badr (2011) who state that there are various forms of questions that can be used to assess students' mastery of material; one of them is an essay (Ekha octaria et al, 2017). 15 items were based on C4 (analyzing) which is categorized as HOTS problem. Examples of items developed are shown in Table 6. grams/mol). Desalinization is a way to eliminate the excess salt levels in sea water in order to produce fresh water with reverse osmosis. If the volume of treated sea water is 500 liters at a temperature of 35 o C (the density of sea water = 1 kg/L) and a reverse osmosis pressure is 32 atm, prove that fresh water can be obtained in that desalination process! Source: http://putramandirigroup.blogspot.com/2012/08/carakerja-reverse-osmosis.html The item above presented a reading/ text stimulus of chemical literacy with topics related to the phenomena of everyday life. Each item had different rating score, depending on the assessment indicators to be measured.

Results of Expert Validation
After developing the assessment instruments, the questions were validated by experts/ validators. The instrument validation was performed by 3 lecturers who were expert in materials. The assessment instruments contained 15 items of essay and the validation consisted of the aspects of readability/ language, construction, and materials/ content. The validation experts/ validator filled the rubric of the assessment instrument with a score ranged from 1 to 5. Material aspects consisted of 5 indicators of validation assessment; the language aspects consisted of 3 assessment indicators, and construction aspects consisted of 5 assessment indicators. The results of the initial validation of 15 items were that several items needed to be revised in terms of language, image, and construction. Then, the questions were re-validated by the experts in order to obtain a valid instrument which was in accordance with the assessment indicators. The results of the item validation by the experts were then analyzed by using Aiken (V). The analysis of the Aiken values of each item after validation of aspects of language, material and construction are displayed in Table 7. Based on the results of Aiken's (V) value in Table 7, it was found that the 15 items were declared to be valid and feasible to being tested at the next stage. The validity of the developed assessment instruments was obtained based on suggestions and inputs from the validators so that the instrument was feasible to be used with revisions/ improvements. The results of expert validation on the aspects of readability, material, and overall construction of the assessment instruments are shown in Figure 2:  Figure 2 displays that the average score of validation in the aspect of content/ material suitability was 94%. This result indicates that the instrument was in accordance with the material aspects. The assessed content/ material suitability aspects included the suitability with basic competencies, the suitability of indicators with the instruments, the suitability of the instruments with Bloom's taxonomic levels, the suitability of the material with the instruments, and the content of the material in questions was appropriate with the school levels. After that, the result of expert validation on the language aspects of the instruments was 91%. The aspects of language assessed by the expert included the presentation of cases and the writing in the assessment instruments, the presentation of the language and the writing in the assessment instruments, and the suitability of the language used with the thinking level of students'. The result of expert validation on the construction aspects of the assessment instruments was 91%, as shown in Figure 2. This result indicates that there is a suitability of the presentation of tables, pictures, graphs, or others on the assessment instruments, the readability of the assessment instruments for analytical thinking skills and chemical literacy, the suitability of the breadth of questions with the development of high school students, the appropriateness of the assessment instruments with the answers, and the completeness of the instruments (questions grids, questions scripts, answers, and scoring guidelines).

Results of Validation for Assessing Analytical Thinking and Chemical Literacy
The validation of assessment instruments on the suitability of analytical thinking skills and chemical literacy was analyzed descriptively by calculating the percentage of the answers from the validators. The results of validation of the assessment of analytical thinking skills and chemical literacy by experts/ validators are shown in Table 8. 10 use to measure analytical thinking and chemical literacy of students on colligative properties. This instrument was declared feasible because the percentage was ≥ 50% with a very good category. The higher the value of the measurement analysis of an instrument, the better the quality of the instruments produced. Hence, the assessment instruments of analytical thinking skills and chemical literacy which was developed and validated could be used to determine the characteristics of other instruments at the next stage of the study.

4.
Conclusion Findings and discussion described above summarized that the whole items of the instrument got very high validity and were declared valid or feasible. The assessment instruments developed in this study got the average index of material aspect validity for 94%, language aspects for 91%, and construction aspects for 91%. Based on the validation results, the percentage of the validation score of the assessment for analytical thinking ability and chemical literacy was 93.33% and the instrument was categorized as very good. This assessment instrument was declared feasible to be used in the subsequent tests and could assess the students' analytical thinking skills and chemical literacy on colligative properties.