ABSTRACT

This research study investigates the cognitive levels of the questions used for assessment and evaluation purpose in three national school board secondary school leaving examinations in Mathematics and Science subjects, conducted in Indian schools from 2011 to 2020. The research used Bloom’s Taxonomy to identify the cognitive levels of 3071 board examination questions. The study addresses the gap in the current literature on the non-availability of a comparative study of national school board assessment practices in India. The study provides a comparative analysis of three English medium Indian school boards (ICSE, CBSE, and NIOS) by analyzing what areas/topics the board exams test and which ones they ignore. Based on this analysis, certain trends were analysed which are present in boards/subjects in the 10th standard national level exams in India. This study will help school education stakeholders to get an insight into the cognitive level trend/patterns of assessment questions.

Keywords

1. INTRODUCTION

A learning process can be segmented into three sections [7]. First, the learning goals, which are also referred to as instructional objectives / Learning Objectives (LO). They specify the learning outcomes that mention the skills and knowledge which is to be imparted to the learner. Second, the selection of learning materials that meet the requirement to achieve the learning goals. Third, assessment questions that determine whether the learner is learning the necessary skills and knowledge to solve the problem. In the educational domain, the Learning Objective refers to the statement(s) of a course curriculum, specifically describing the skills and knowledge that the student must gain after the course is completed. If complemented by a similar cognitive level of instructional techniques and tests, the learning outcomes help the teachers determine whether the students are achieving the expected skills and knowledge.

The Learning Objective also determines the performance results by determining the conditions under which performance will occur. It also defines the requirements (specific skills, competencies, and attitudes) that the learners will follow. “Learning outcomes are precise statements of what faculty expects students to know and to be able to do in some measurable way as a result of completing a program, course, unit, or lesson" as stated in [3]. It was also observed that “in addition to guiding, teaching, learning, and assessment strategy, effective learning outcomes facilitate student orientation to the subject and communicate expectations" as reflected in this research [6]. For active learning to take place, ‘there must be a constructive alignment of the curriculum, which should ensure that in an education program, the learning objectives, teaching and learning methodologies, and assessment techniques should complement each other’.

Student’s ability to think for activities during instruction and examinations are essential for improving their intellectual abilities, performance execution, and professional growth. As a result, exams should be designed to encourage students to express their thoughts on the exam questions, develop creative answers, and connect the exam answers to their own experiences and real-life situations [5]. Also, good questions not only promote effective learning and assessment, but they must also be consistent with curriculum and instruction, as assessment has a significant impact on both learning and teaching. The phenomenon is known as the ‘washback effect’, and it refers to how testing affects teaching and learning [17].

As a result, writing high-quality exams that include both higher-level questions (HLQs) and lower-level questions (LLQs) is critical in assisting students in achieving the desired learning outcomes and evaluating their level of proficiency in a specific course. The HLQs help students dig deeper into the learning materials while also encouraging critical thinking and creativity. A work by [2] stated that, schools must emphasise higher-order skills in order to develop critical thinking.

Therefore, the present study seeks to examine and compare to what extent the questions of the tenth grade school board examinations in Mathematics and Science subjects prepared by three national level Indian school boards include both higher and lower-order thinking levels. In the first place, it is essential to assess how well students master the information of the educational materials within the six levels of the Bloom’s Taxonomy. Additionally, it is essential to analyze whether the exam questions of the given examinations in the Mathematics and Science subjects are based on both higher and lower-order thinking levels.

2. LITERATURE REVIEW

2.1 Mapping Bloom’s Taxonomy Cognitive Levels to Thinking orders

Bloom developed a taxonomy in this context that is used to develop assessments that take into account each of the six levels of hierarchy in the cognitive domain [14]. Knowledge (recalling details), comprehension (description in someone else’s words), and application (using existing knowledge to produce results) are examples of lower-order thinking domains. Higher-order thinking domains include analysis (discovering connections between facts and concepts), synthesis (creating new original work), and evaluation (judging and demonstrating one’s position) [12]. Researchers used Bloom’s Taxonomy’s two cognitive categories to analyse and determine the levels of questions asked in exams, and they established two types: lower-level questions (LLQs) and higher-level questions (HLQs) [16]. The LLQs are designed to test students’ recall of fundamental and universal concepts and processes. The HLQs, on the other hand, are more advanced and difficult because they require students to engage in deeper and analytical thinking processes.

2.2 School Board Examination System in India

The work by [8] gives a comprehensive overview of Indian school education system. The research article extensively covers the Indian school education system, which is one of the largest education system in the world. With both public and private schools, the Indian school system can be divided into four main categories – pre primary (consisting of pre-school, lower and higher kindergarten), primary school (standard one to five), middle school (standard six to eight), secondary school (standard nine and ten) and high school (standard eleven and twelve / or pre university standard). The public schools are majorly either central government schools (such as Kendriya vidyalayas, navodaya vidyalayas, Sainik schools etc.) or state government schools of respective states. The private schools are usually run by individuals, trusts or societies and may or may not receive fund from the government. Apart from these two major categories, some other semi government type schools run by local government bodies also exist (e.g. Municipality schools). The central government schools are usually affiliated to the Central Board for Secondary Education (CBSE) supervised by the National Council of Educational Research and Training (NCERT) under the Ministry of Education. The Council of Indian School Certificate Examinations (CISCE) is a semi-private, non governmental education board in India. It conducts the Indian Certificate for Secondary Education (ICSE) examination (for tenth standard) and Indian School Certificate (ISC) examination (for twelfth standard) in India. These two are the major all India based school examination boards. Apart from the all India based school examination boards, the state government affiliated school examination boards constitute a major part of Indian school examination system. All Indian school boards for the sake of collaborating and exchange of information with each other forms an umbrella body called Council of Boards of School Education in India (COBSE), a voluntary association of all the boards of school education in India. There are more than 50 members with associate members from Nepal, Mauritius, Bhutan, Pakistan and United Kingdom. Other than these Indian school boards, foreign school boards such as International Baccalaureate Organization (IBO) and Cambridge International Examinations (CIE) are emerging as newer school boards in urban areas. These schools boards offer global school level examinations all over the world and follows universal curriculum. It must be noted that all school boards in India are autonomous having their own syllabus, curriculum, method of assessment and evaluation.

An elaborate research work on the quality of school education in India has been done by Institute for Studies in Industrial Development as mentioned in [9] for Quality Council of India, New Delhi. Firstly, they have clearly defined the distinction between syllabus and curriculum. Curriculum is being defined as – “In formal education, a curriculum (plural curricula) is the set of courses, and their content, offered at a school or university”, while syllabus is defined as “A syllabus is an outline and summary of topics to be covered in a course”. They have identified that in CBSE, the advantage is that the curriculum is same all over the country and the continuity of education is not a problem if someone needs to change a school. They have also inferred that ICSE syllabus is tougher than that of the CBSE and state based school boards. Their research work showed that the school boards are giving high importance to evaluation and examination system which includes some additional forms of evaluation such as (a) project work, (b) reading and writing skills, (c) participation in co-curricular activities, (d) attitude and behaviours, etc. However major emphasis was given on written examination by schools. IBO puts more emphasis on project based and practical work compared to the Indian school boards as it follows a global curriculum all over the world. IBO assessment focuses on what skills the students have learnt or what level of understanding can the students demonstrate. British Council in India, in their report [13] on the Indian school education system provides an overall picture into this large and evolving school education system of India. They remarked that “the present education system in India is guided by different objectives and goals but is based around the policies of yester years.” They claim that two important policies of the Government of India—the Sarva Shiksha Abhiyan (SSA) in 2001 and the Right of Children to Free and Compulsory Education (RTE) Act, 2009 have made education priorities rise among common people of India and have been responsible for improvements in educational performance. However, this report does not mention about the challenges faced by the Indian school education system today. Also National Institute of Open Schooling (NIOS) is not covered in these reports.

3. NEED AND SIGNIFICANCE OF THE STUDY

The objective of this research is to find the trends and patterns of cognitive level in secondary school board examination questions. To the best of our knowledge, there is no previous research on a comparative analysis of the cognitive level of questions from three different school boards (with respect to India) and in multiple subjects. This study tried to address this gap through a comprehensive research and provide an accessible and open dataset as a solution to the problem and for future similar comparative research.

4. RESEARCH QUESTION

This study investigates the cognitive levels of the questions used in the English medium school leaving examinations for three national school boards in India administered nationwide from 2011 to 2020. It put forwards the following research question: To what extent do Indian school board school leaving examination questions cover the lower and higher-order cognitive levels of Bloom’s Taxonomy?

5. DATASET DETAILS AND DESCRIPTION

5.1 Why are these data useful?

Every year, millions of students appear for the 10th board (secondary exam) in India from various national and regional state education school boards. In 2021, 21,50,608 students appeared in the 10th Board exam in Central Board for Secondary Education Examination (CBSE) in India [1]. However, not much is studied about what Cognitive areas the board exams test, which sections of learning they stress, and which ones they ignore. This research tries to provide a comprehensive comparative analysis of three English medium national school boards – Indian Certificate of Secondary Education (ICSE), the Central Board of Secondary Education (CBSE), and the National Institute of Open Schooling (NIOS). The Indian Certificate of Secondary Education popularly known as ICSE is an examination conducted by the Council for the Indian School Certificate Examination (CISCE), a private board of school education in India for Class 10. The CISCE board is headquartered in New Delhi. The Central Board of Secondary Education (CBSE) is a national-level board of education in India for public and private schools, controlled and managed by the Government of India. There are more than 27,000 schools in India and 240 schools in 28 foreign countries affiliated with the CBSE. The National Institute of Open Schooling (NIOS), formerly National Open School was established by the Ministry of Human Resource Development of the Government of India in 1989 to provide education to all segments of society with the motive to increase literacy and aimed forward for flexible learning. The NIOS is a national board that administers examinations for Secondary and Senior Secondary examinations similar to the CBSE and the ICSE. NIOS enrolls about 350,000 students annually which makes it one of the largest open schooling systems in the world.

5.2 Who can benefit from these data?

This data will help school education stakeholders to get an insight into the cognitive level trend/patterns of assessment questions.This data can be extremely useful and can be further re-used for creating intelligent tools as mentioned in similar research articles like [10] and [9]. The following table, Table 1 shows the details of the dataset.

6. METHODOLOGY

The data was collected from the physical and digital copies of the previous question papers. The text data of digital copies were cleaned and curated in online spreadsheet systems (Google Sheets). Once compiled, all the questions were segregated into individual sheets year and subject-wise¹ . Three annotators having sufficient domain knowledge about Cognitive levels using Bloom’s Taxonomy action verbs [11] annotated each question individually to identify the most appropriate cognitive level. The inter-annotator agreement for the appropriate cognitive level was in the substantial agreement range (using Fleiss’s kappa) [4]. For data analysis and visualization, Tableau [15] was used.

7. RESULTS

7.1 General Findings

Table 2 and Table 3 shows the top five Bloom’s Taxonomy action verbs used for Science and Mathematics across all the three boards. Figure 1 shows the various types of questions, based on their cognitive level and their frequencies. Figure 2 shows various types of questions and their frequencies against each board. Figure 3 shows Cognitive level with respect to individual boards.

7.2 Patterns and Observations

7.2.1 General Question patterns

Figure 4 (Mathematics), 5 (Physics), 6 (Chemistry), and 7 (Biology) shows the comparison of the number of questions asked on behalf of different Cognitive levels of Bloom’s Taxonomy against each Board. Figure 8 shows an overall comparison of Boards, Question type and Cognitive levels. From this analysis, it can be inferred which type of assessment was most preferable for a specified cognitive level. As it is observed, Application-level questions are most frequently asked in all three types of questions (MCQ, Long, Short).

For the MCQ type of questions, there was no “create” and “evaluate” type of questions asked. The reason can be that MCQ questions are known as single marks questions, and asking for creation and evaluation will be much more in terms of time and effort required. In long and short type questions, all Cognitive levels of Bloom’s Taxonomy questions can be found. It was also observed that lower cognitive levels of Bloom’s Taxonomy questions are asked more for the long answer types questions.

7.2.2 Board vs Subject vs Cognitive level

The highest number of application-level questions are asked in Mathematics subject in all three boards. Physics also had the highest frequency of Application level questions among all three boards similar to Mathematics. For Biology subject most number of Knowledge level questions were asked for CBSE and NIOS boards. At the same time, ICSE emphasises more on the understanding-type questions, as they had the highest tally among all other cognitive levels. Similarly for Chemistry, Knowledge level questions frequency was highest among all three board subjects. For all three boards for 9 years, there were no create level cognitive-type questions asked in Chemistry.

7.3 Discussions

7.3.1 Analysis of the three boards for educational and assessment practices

According to the study findings, lower order cognitive level exam questions outnumber higher order cognitive level exam questions by a wide margin. Such an imbalance in questions based on the six cognitive domains in national examinations may have a negative impact on instructional quality and student learning. CBSE has two categories of papers in almost every year, Delhi and Outside Delhi. These papers have some differences in number of questions in each cognitive level. In ICSE science, all questions have multiple subparts (upto 5 or 6) each from different chapters as well as cognitive levels. These can be very confusing for the student and can cause ambiguities in their analysis. Maths papers throughout the three boards have an abundance of application level questions. Most of these have the somewhat ambiguous Bloom’s Taxonomy Action Verb ‘Find’. In NIOS papers (Mathematics), the questions for visually impaired students are usually of lower cognitive level than the equivalent for the other students. NIOS Maths asks students to name and define terms in the form of MCQs (approximately 5%) that comes under memorization level and is absent in CBSE. CBSE papers before 2018 only test concepts from the second half of the syllabus, as was the case in Continuous And Comprehensive Evaluation (CCE) pattern.

7.3.2 The impact of secondary school national board exams on teaching and learning quality

The questions indicated that students are required to spend more instructional time preparing for the exam or studying past exams that are heavily lower cognitive thinking (LOTS) driven and derived from curriculum books. As a result, students will fail to master complex reasoning skills as required by the curriculum. Students are more likely to face challenges in secondary and tertiary education, as well as in their personal and professional lives, if primary education does not include critical thinking instruction and long-term assessment.

8. LIMITATIONS OF THE STUDY AND FUTURE WORK

The study, however, was limited to the English medium national school board examinations. With 30 or more state boards in regional languages, data collection and annotation would have been a real challenge. However, if it had been implemented, it would have provided a more in-depth understanding of the national education system by reflecting on assessment approaches.

9. CONCLUSION

All the Indian school boards are similar in terms of making students remember the facts, understand the concepts and apply them to solve problems. Thus the questions for evaluation and assessment also focus on these three major aspects. The CBSE syllabus is comparatively “more easy” on students in its approach as it has been designed for a specific year and is divided into various segments. Every segment is given a specific number of periods so that it can be completely and thoroughly taught in one year. It emphasises on understanding of concepts and processes with their application. The ICSE system stresses more in terms of aptitude development and thoroughness by almost equally focusing the syllabus on remembering facts, understanding of concepts and application of the processes learnt. NIOS being an open schooling option, do provide some relaxation in terms of cognitive rigour when compared to the other two national boards. In conclusion we can say, if the syllabus is written considering the Bloom’s Taxonomy and knowledge dimension, it will be lot easier to analyze and evaluate that whether the learning objectives have been successfully satisfied on completion of the curriculum.

10. REFERENCES

1. Cbse 10th results 2021. https://www.ndtv.com/education/cbse-10th-result-2021-9904-pass-result-under-process-for-over-16000-students. [Accessed 14-Jan-2023].
2. Rigor/Relevance Framework: A Guide to Focusing Resources to Increase Student Performance — leadered.com. https://leadered.com/papers/rigor-relevance-framework-a-guide-to-focusing-resources-to-increase-student-performance/. [Accessed 20-Jan-2023].
3. L. W. Anderson. Objectives, evaluation, and the improvement of education. Studies in educational evaluation, 31(2-3):102–113, 2005.
4. R. Artstein. Inter-annotator agreement. In Handbook of linguistic annotation, pages 297–313. Springer, 2017.
5. I. R. Assaly and O. M. Smadi. Using bloom’s taxonomy to evaluate the cognitive levels of master class textbook’s questions. English Language Teaching, 8(5):100–110, 2015.
6. S. Chadwick. Curriculum development in orthodontic specialist registrar training: Can orthodontics achieve constructive alignment? Journal of orthodontics, 31(3):267–274, 2004.
7. S. Das. Cognitive Level Analysis in a Learning Cycle. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pages 449–451, Mumbai, July 2018. IEEE.
8. S. Das and A. Basu. A sample study on the distribution of indian school curricula over bloom’s cognitive domain categories. In EDULEARN16 Proceedings, 8th International Conference on Education and New Learning Technologies, pages 4811–4817. IATED, 4-6 July, 2016 2016.
9. S. Das and A. Basu. A sample study on the distribution of indian school curricula over bloom’s cognitive domain categories. In EDULEARN16 Proceedings, pages 4811–4817. IATED, 2016.
10. S. Das, S. K. Das Mandal, and A. Basu. Cognitive complexity analysis of learning-related texts: A case study on school textbooks. In International Conference in Methodologies and intelligent Systems for Techhnology Enhanced Learning, pages 74–84. Springer, 2020.
11. S. Das, S. K. Das Mandal, and A. Basu. Classification of action verbs of bloom’s taxonomy cognitive domain: An empirical study. Journal of Education, 202(4):554–566, 2022.
12. N. M. Freahat and O. M. Smadi. Lower-order and higher-order reading questions in secondary and university level efl textbooks in jordan. Theory & Practice in Language Studies, 4(9), 2014.
13. B. C. India. Indian school education system-an overview. retrieved march 20, 2017, 2014.
14. D. R. Krathwohl. A revision of bloom’s taxonomy: An overview. Theory into practice, 41(4):212–218, 2002.
15. A. Ohmann and M. Floyd. Creating Data Stories with Tableau Public. Packt Publishing Ltd, 2015.
16. T. V. Ramirez. On pedagogy of personality assessment: Application of bloom’s taxonomy of educational objectives. Journal of personality assessment, 99(2):146–152, 2017.
17. W. Sundayana, P. Meekaeo, P. Purnawarman, and D. Sukyadi. Washback of english national exams at ninth-grade level in thailand and indonesia. Indonesian Journal of applied linguistics, 8(1):167–176, 2018.

© 2023 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.