Assessment Matters! How good are Canadian 15-year-olds at solving problems? Further results from PISA Introduction

Please download to get full document.

View again

of 19
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report
Category:

Word Search

Published:

Views: 0 | Pages: 19

Extension: PDF | Download: 0

Share
Related documents
Description
Assessment Matters! 6 How good are Canadian 15-year-olds at solving problems? Further results from PISA 2012 Introduction The skills and knowledge of a population are crucial to the well-being of both
Transcript
Assessment Matters! 6 How good are Canadian 15-year-olds at solving problems? Further results from PISA 2012 Introduction The skills and knowledge of a population are crucial to the well-being of both individuals and society. For individuals, high skill levels contribute to economic security and personal fulfillment; for society, they promote productivity and economic growth. In Canada as in most countries education systems recognize that there is more to skills than simply competence in core subject areas. Skills also include crosscurricular abilities: creativity, interpersonal abilities, project management, and entrepreneurship, to name but a few. Successful learners will not only master these soft skills while in school, but will also carry them forward into the workplace and apply them as required. These skills can broadly be categorized as problemsolving skills. They encompass multiple sources of knowledge, which are brought together and applied using various types of reasoning (deductive, inductive, analogical, combinatorial). Their advantage lies in allowing individuals to process multiple sources of information in unfamiliar situations, where the solution to a problem is not readily apparent. Developing such skills, in addition to cognitive abilities, is increasingly a goal of today s education systems. In our rapidly changing world, individuals are faced more and more with unfamiliar situations where rote learning fails to serve them, and which demand problem solving. In response, governments are investing heavily in education that includes these skills. To evaluate this investment, the Organisation for Economic Co-operation and Development (OECD) measures the problem-solving skills of youth as part of the Programme for International Student Assessment (PISA). The overall aim of PISA is to assess the extent to which 15-year-olds in participating countries and economies have acquired the essential knowledge and skills to succeed in the future. Organization of report This report examines the most recent results of PISA in the area of problem solving. It begins by describing the concept of problem-solving skills and PISA. It then presents the performance of Canadian youth in problem solving in a global context, explains results in terms of proficiency levels, and examines differences in problemsolving skills between provinces and among certain populations. The report finishes with a discussion of the relationship between problem-solving skills and skills in mathematics, reading, and science, and then a summary of findings. How is problem solving assessed in PISA? Problem solving was first tested in PISA in 2003, when it was added to the core domains of mathematics, reading, and science. Selected students were asked to test their problem-solving skills using a paper-based assessment, and Canadian students performed significantly above the OECD average. Only four of the 40 participating countries achieved a higher score than Canada. 1 In 2012, testing was enriched by the introduction of a computer-based assessment. Problem-solving skills in this latest iteration of PISA were defined as follows: an individual s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one s potential as a constructive and reflective citizen. 2 In total, 470, year-olds from 65 countries and economies participated in PISA in 2012, of whom approximately 21,000 were from Canada. 3 Out of these countries, only 44 participated in the problem-solving option of PISA Although 65 countries and economies participated in the PISA 2012 assessments of mathematics, reading, and science, the number participating in the problemsolving assessment was lower, at 44. The 28 participating OECD countries were: Australia, Austria, Belgium, Canada, Chile, Czech Republic, Denmark, Estonia, Finland, France, Germany, Hungary, Ireland, Israel, Italy, Japan, Korea, the Netherlands, Norway, Poland, Portugal, Slovak Republic, Slovenia, Spain, Sweden, Turkey, United Kingdom, and United States. The 16 non-oecd countries and economies comprised: Brazil, Bulgaria, Chinese Taipei, Colombia, Croatia, Cyprus, Hong Kong-China, Macao-China, Malaysia, Montenegro, Russian Federation, Serbia, Shanghai-China, Singapore, United Arab Emirates, and Uruguay. Canadian students performed well in problem solving Overall, Canadian 15-year-olds demonstrated high levels of problem-solving skills. They attained an average score of 526, which is well above the OECD average of 500, and were outperformed by students from only two other OECD countries Korea and Japan. Five non- OECD members also outperformed Canada (Singapore, Macao-China, Hong Kong-China, Shanghai-China, and Chinese Taipei). 1 Bussière, P., Cartwright, F. and Knighton, T. (2004). Measuring up: Canadian results for the OECD PISA study. The performance of Canada s Youth in mathematics, reading, science and problem solving. Ottawa: Ministry of Industry. 2 OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Paris: Author. 3 Brochu, P., M-A. Deussing, K. Houme and M. Chuy. (2013). Measuring up: Canadian results for the OECD PISA study. The performance of Canada s Youth in mathematics, reading, science and problem solving First Results for Canadians Aged 15. Toronto: Council of Ministers of Education, Canada. Assessment Matters! No. 6, CMEC 2 Figure 1 Estimated average scores and confidence intervals for countries and provinces in problem solving Above the Canadian average At the Canadian average Below the Canadian average 95% Confidence interval Estimated average score Countries and provinces Average S.E. Singapore 562 (1.2) Korea 561 (4.3) Japan 552 (3.1) Macao-China 540 (1.0) Hong Kong-China 540 (3.9) Shanghai-China 536 (3.3) British Columbia 535 (3.5) Chinese Taipei 534 (2.9) Alberta 531 (5.1) Ontario 528 (5.7) Canada 526 (2.4) Quebec 525 (4.5) Australia 523 (1.9) Finland 523 (2.3) United Kingdom 517 (4.2) New Brunswick 515 (3.1) Estonia 515 (2.5) Saskatchewan 515 (2.8) Nova Scotia 512 (5.7) France 511 (3.4) Netherlands 511 (4.4) Italy 510 (4.0) Czech Republic 509 (3.1) Germany 509 (3.6) United States 508 (3.9) Belgium 508 (2.5) Austria 506 (3.6) Newfoundland and Labrador 504 (7.3) Manitoba 504 (3.6) Norway 503 (3.3) Ireland 498 (3.2) Denmark 497 (2.9) Portugal 494 (3.6) Prince Edward Island 493 (2.6) Sweden 491 (2.9) Russian Federation 489 (3.4) Slovak Republic 483 (3.6) Poland 481 (4.4) Spain 477 (4.1) Slovenia 476 (1.5) Serbia 473 (3.1) Croatia 466 (3.9) Hungary 459 (4.0) Turkey 454 (4.0) Israel 454 (5.5) Chile 448 (3.7) Cyprus 445 (1.4) Brazil 428 (4.7) Malaysia 422 (3.5) United Arab Emirates 411 (2.8) Montenegro 407 (1.2) Uruguay 403 (3.5) Bulgaria 402 (5.1) Colombia 399 (3.5) Canada showed marked variation in problem-solving skills at the provincial level. 4 Students in British Columbia registered a score of 535, which was above the Canadian average, while those in Alberta, Ontario, and Quebec scored at the Canadian average. Students in the remaining provinces scored below the Canadian average, with Prince Edward Island s score (493) the only one below the OECD average of 500. Appendix A, Table A.1 presents multiple comparisons between Canadian provinces and all other countries and economies that participated in the problem-solving component. 4 No data were collected in the three territories and in First Nation schools. Assessment Matters! No. 6, CMEC 3 Figure 2 Estimated average scores in problem solving Canada and the provinces Canada (526) OECD (500) BC AB SK MB ON QC NB NS PE NL Boys in Canada held a small advantage over girls in problem-solving skills The difference between the sexes in problem solving was small, with boys outscoring girls by five points. Furthermore, this difference was attributed to a single province Ontario, where boys outscored girls by nine points in problem solving. For purposes of comparison, in reading, girls outperformed boys in Canada by 35 points, while in mathematics boys scored 10 points higher than girls. In science there was no difference between the two. Table 1 Estimated average scores and gender differences in student performance by province Females Males Gender difference (Female - Male) Canada and provinces average standard average standard difference standard Canada 523 (2.5) 528 (2.8) -5 (2.2)* Newfoundland and Labrador 512 (5.4) 496 (10.6) 16 (8.3) Prince Edward Island 494 (3.5) 492 (3.3) 2 (4.5) Nova Scotia 512 (8.0) 512 (5.5) 1 (7.4) New Brunswick 520 (4.0) 511 (4.7) 9 (6.1) Quebec 523 (4.7) 526 (5.5) -4 (4.8) Ontario 523 (5.2) 533 (6.8) -9 (4.1)* Manitoba 503 (5.1) 504 (4.5) -1 (6.3) Saskatchewan 520 (3.9) 510 (3.7) 10 (5.1) Alberta 529 (5.6) 533 (5.1) -5 (3.7) British Columbia 530 (5.1) 540 (4.0) -9 (5.9) OECD average 497 (0.7) 503 (0.9) -7 (0.8)* * Statistically significant differences Assessment Matters! No. 6, CMEC 4 Overall in Canada, no significant differences were observed between problem-solving skills of students in French- and English-language school systems. However, differences were observed in Ontario and New Brunswick, where students from the English-language systems outperformed their francophone counterparts by 38 and 28 points, respectively. Table 2 Estimated average scores and differences in student performance by language of school system, by province 5 Francophone school system Anglophone school system Difference between the anglophone and francophone school systems Canada and provinces average standard average standard difference standard Canada 522 (4.5) 527 (3.0) 5 (5.8) Nova Scotia 524 (6.6) 512 (5.9) -12 (9.3) New Brunswick 494 (3.8) 522 (3.9) 28 (5.4)* Quebec 525 (5.0) 519 (5.0) -6 (6.8) Ontario 491 (3.9) 529 (5.9) 38 (6.7)* Manitoba 494 (6.8) 504 (3.7) 10 (7.7) Alberta 517 (14.2) 531 (5.1) 14 (15.3) British Columbia 531 (8.5) 535 (3.5) 4 (9.6) * Statistically significant differences Internationally, Canadian 15-year-old students performed relatively well, placing third among OECD countries and eighth among all participating countries/ economies. Some significant differences were observed between provinces. For example, the results for British Columbia were comparable to those of the highestperforming countries, while Prince Edward Island scored below the OECD average. A very small gender difference in problem-solving skills was observed. This was in contrast to the gender differences in reading and mathematics as measured by PISA. When it comes to performance by language of the school system, only two provinces saw significant differences, and these were sizable. What students can do in problem solving To give concrete meaning to scores, PISA has defined ranges of scores that correspond to specific sets of skills. These are referred to as proficiency levels. For problem solving, six proficiency levels were identified and are defined below. 6 5 Due to low target population sizes, estimates for Saskatchewan, Newfoundland and Labrador, and Prince Edward Island were not statistically reliable and therefore not reported. 6 Adapted from OECD (2014). PISA 2012 Results: Creative problem solving: Students skills in tackling real-life problems. Volume V. Paris: Author, p. 57. Assessment Matters! No. 6, CMEC 5 Summary description of the six levels of proficiency in problem solving Level Score range 6 Equal to or higher than 683 points to less than 683 points to less than 618 points to less than 553 points Percentage of students able to perform tasks at this level or above (Canadian / OECD average) What students can typically do 5.1% / 2.5% At Level 6, students can develop complete, coherent mental models of diverse problem scenarios, enabling them to solve complex problems efficiently. They can explore a scenario in a highly strategic manner to understand all information pertaining to the problem. The information may be presented in different formats, requiring interpretation and integration of related parts. When confronted with very complex devices, such as home appliances that work in an unusual or unexpected manner, they quickly learn how to control the devices to achieve a goal in an optimal way. 17.5% / 11.4% At Level 5, students can systematically explore a complex problem scenario to gain an understanding of how relevant information is structured. When faced with unfamiliar, moderately complex devices, such as vending machines or home appliances, they respond quickly to feedback in order to control the device. 40.5% / 31.0% At Level 4, students can explore a moderately complex problem scenario in a focused way. They grasp the links among the components of the scenario that are required to solve the problem. They can control moderately complex digital devices, such as unfamiliar vending machines or home appliances, but they do not always do so efficiently. 66.3% / 56.6% At Level 3, students can handle information presented in several different formats. They can explore a problem scenario and infer simple relationships among its components. They can control simple digital devices, but have trouble with more complex devices to less than 488 points to less than 423 points 85.3% / 78.6% At Level 2, students can explore an unfamiliar problem scenario and understand a small part of it. They try, but only partially succeed, to understand and control digital devices with unfamiliar controls, such as home appliances and vending machines. 94.9% / 91.8% At Level 1, students can explore a problem scenario only in a limited way, but tend to do so only when they have encountered very similar situations before. Based on their observations of familiar scenarios, these students are only able to partially describe the behaviour of a simple, everyday device. Level 2 is deemed to be the minimum proficiency level to function adequately in society. In Canada, 85.3% of 15-year-olds reached at least this level in problem solving. This compares well with the OECD average of 78.6%, but is below other OECD countries such as Korea and Japan (93.1% and 92.9%, respectively). At the other end of the scale, 17.5% of Canadians tested scored at Level 5 or above, compared to the OECD average of 11.4%. While these results are strong, Canada did place significantly lower than certain OECD countries, such as Korea (27.6%) and Japan (22.3%). Assessment Matters! No. 6, CMEC 6 Table 3 Country, economy, and province Percentage of students at each proficiency level for provinces, countries, and economies Proficiency levels Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Korea 2.1 (0.3) 4.8 (0.6) 12.9 (0.9) 23.7 (1.0) 28.8 (0.9) 20.0 (1.2) 7.6 (0.9) Japan 1.8 (0.4) 5.3 (0.6) 14.6 (0.9) 26.9 (1.1) 29.2 (1.0) 16.9 (1.0) 5.3 (0.7) Macao-China 1.6 (0.2) 6.0 (0.4) 17.5 (0.6) 29.5 (0.8) 28.9 (0.9) 13.8 (0.6) 2.8 (0.3) Singapore 2.0 (0.2) 6.0 (0.4) 13.8 (0.6) 21.9 (0.7) 27.0 (1.0) 19.7 (0.7) 9.6 (0.4) Hong Kong-China 3.3 (0.5) 7.1 (0.7) 16.3 (1.0) 27.4 (1.4) 26.5 (1.0) 14.2 (1.1) 5.1 (0.6) Shanghai-China 3.1 (0.5) 7.5 (0.6) 17.5 (0.8) 27.4 (1.1) 26.2 (1.0) 14.1 (0.9) 4.1 (0.6) Chinese Taipei 3.4 (0.6) 8.2 (0.6) 17.8 (0.8) 26.3 (1.0) 25.9 (1.0) 14.6 (0.7) 3.8 (0.4) British Columbia 3.1 (0.7) 9.4 (1.0) 18.2 (1.3) 26.1 (1.4) 24.0 (1.4) 13.8 (1.3) 5.3 (0.7) Alberta 4.6 (0.6) 9.6 (1.0) 16.8 (1.4) 26.2 (1.6) 23.9 (1.6) 13.6 (1.2) 5.3 (0.8) Finland 4.5 (0.4) 9.9 (0.5) 20.0 (0.9) 27.1 (1.1) 23.5 (0.8) 11.4 (0.6) 3.6 (0.5) Ontario 5.1 (0.7) 9.4 (1.0) 19.4 (1.1) 24.9 (1.2) 22.5 (1.3) 12.6 (1.0) 6.0 (1.0) Quebec 5.8 (0.8) 8.9 (0.7) 18.0 (1.0) 26.5 (1.2) 23.4 (0.9) 12.6 (1.1) 4.7 (0.8) Canada 5.1 (0.4) 9.6 (0.4) 19.0 (0.6) 25.8 (0.7) 22.9 (0.6) 12.4 (0.6) 5.1 (0.4) Estonia 4.0 (0.5) 11.1 (0.8) 21.8 (0.7) 29.2 (1.0) 22.2 (0.8) 9.5 (0.7) 2.2 (0.3) Australia 5.0 (0.3) 10.5 (0.5) 19.4 (0.5) 25.8 (0.7) 22.6 (0.5) 12.3 (0.5) 4.4 (0.3) New Brunswick 5.4 (0.7) 10.3 (1.2) 20.8 (1.6) 28.0 (2.4) 23.4 (1.7) 9.3 (1.2) 2.8 (0.6) Nova Scotia 5.1 (1.4) 10.8 (1.6) 22.6 (3.2) 27.3 (2.8) 22.6 (2.4) 9.2 (1.1) 2.5 (0.8) Saskatchewan 5.2 (0.7) 11.1 (1.0) 21.1 (1.6) 28.0 (1.6) 20.7 (1.3) 10.9 (1.1) 2.9 (0.6) United Kingdom 5.5 (0.8) 10.8 (0.8) 20.2 (1.3) 26.5 (0.9) 22.7 (1.1) 10.9 (0.8) 3.3 (0.6) Italy 5.2 (0.7) 11.2 (1.1) 22.5 (1.0) 28.0 (1.1) 22.3 (1.1) 8.9 (0.9) 1.8 (0.3) France 6.6 (0.9) 9.8 (0.7) 20.5 (1.0) 28.4 (1.1) 22.6 (0.9) 9.9 (0.7) 2.1 (0.3) United States 5.7 (0.8) 12.5 (0.9) 22.8 (1.0) 27.0 (1.0) 20.4 (0.9) 8.9 (0.7) 2.7 (0.5) Czech Republic 6.5 (0.7) 11.9 (0.9) 20.7 (1.0) 27.2 (0.9) 21.8 (0.9) 9.5 (0.7) 2.4 (0.3) Austria 6.5 (0.9) 11.9 (0.8) 21.8 (1.1) 26.9 (1.2) 21.9 (1.0) 9.0 (0.8) 2.0 (0.4) Netherlands 7.4 (1.0) 11.2 (1.0) 19.9 (1.2) 26.0 (1.3) 22.0 (1.2) 10.9 (1.0) 2.7 (0.5) Newfoundland 7.6 (2.1) 11.3 (1.6) 21.6 (1.5) 26.9 (1.7) 21.0 (1.6) 9.3 (1.1) 2.3 (0.6) and Labrador Germany 7.5 (0.8) 11.8 (0.9) 20.3 (0.9) 25.6 (1.0) 22.0 (1.0) 10.1 (1.0) 2.7 (0.4) Ireland 7.0 (0.8) 13.3 (0.9) 23.8 (0.8) 27.8 (0.9) 18.8 (0.8) 7.3 (0.6) 2.1 (0.3) Denmark 7.3 (0.7) 13.1 (0.7) 24.1 (0.8) 27.8 (0.9) 19.0 (1.1) 7.2 (0.7) 1.6 (0.3) Manitoba 7.3 (1.0) 13.2 (1.2) 21.6 (1.1) 24.8 (1.6) 21.2 (1.4) 9.2 (1.2) 2.7 (0.5) Portugal 6.5 (0.6) 14.1 (1.0) 25.5 (0.9) 28.1 (1.0) 18.4 (0.9) 6.2 (0.6) 1.2 (0.3) Belgium 9.2 (0.6) 11.6 (0.6) 18.3 (0.7) 24.5 (0.6) 22.0 (0.7) 11.4 (0.7) 3.0 (0.3) Prince Edward Island 7.0 (0.7) 14.2 (1.2) 25.7 (1.5) 28.2 (2.1) 17.7 (1.2) 5.6 (0.9) 1.6 (0.5) Norway 8.1 (0.7) 13.2 (0.7) 21.5 (0.9) 24.7 (0.8) 19.4 (0.8) 9.7 (0.7) 3.4 (0.4) Russian Federation 6.8 (0.7) 15.4 (1.1) 27.0 (0.9) 27.9 (1.2) 15.7 (0.9) 5.9 (0.7) 1.4 (0.3) Sweden 8.8 (0.7) 14.6 (0.8) 23.9 (0.9) 26.3 (0.8) 17.6 (0.7) 7.0 (0.5) 1.8 (0.3) Poland 10.0 (1.1) 15.7 (1.0) 25.7 (0.9) 26.0 (1.0) 15.7 (1.0) 5.8 (0.7) 1.1 (0.2) Slovak Republic 10.7 (1.1) 15.4 (1.1) 24.3 (1.0) 25.6 (1.3) 16.2 (1.2) 6.3 (0.6) 1.6 (0.5) Spain 13.1 (1.2) 15.3 (0.8) 23.6 (0.9) 24.2 (1.0) 15.9 (0.8) 6.2 (0.6) 1.6 (0.3) Slovenia 11.4 (0.6) 17.1 (1.0) 25.4 (1.2) 23.7 (0.8) 15.8 (0.8) 5.8 (0.5) 0.9 (0.2) Serbia 10.3 (1.0) 18.3 (0.8) 26.7 (1.4) 25.8 (1.1) 14.3 (0.8) 4.1 (0.4) 0.6 (0.2) Croatia 12.0 (1.0) 20.2 (1.0) 26.8 (1.2) 22.9 (1.1) 13.2 (1.1) 4.0 (0.6) 0.8 (0.2) Hungary 17.2 (1.3) 17.8 (0.9) 23.9 (1.2) 22.4 (0.9) 13.0 (1.0) 4.6 (0.7) 1.0 (0.2) Turkey 11.0 (1.1) 24.8 (1.3) 31.4 (1.4) 21.2 (1.2) 9.4 (1.1) 2.0 (0.5) 0.2 (0.1) Chile 15.1 (1.3) 23.1 (1.1) 28.6 (1.0) 22.2 (1.0) 8.8 (0.7) 1.9 (0.3) 0.2 (0.1) Israel 21.9 (1.4) 17.0 (0.9) 20.1 (0.8) 18.5 (0.9) 13.7 (0.9) 6.7 (0.8) 2.1 (0.4) Brazil 23.5 (1.6) 25.5 (1.4) 26.1 (1.3) 16.8 (1.4) 6.3 (0.8) 1.4 (0.3) 0.4 (0.1) Malaysia 22.7 (1.5) 27.8 (1.2) 27.8 (1.2) 15.7 (0.9) 5.2 (0.6) 0.8 (0.2) 0.1 (0.0) United Arab Emirates 30.3 (1.2) 24.6 (0.8) 22.0 (0.7) 14.2 (0.6) 6.4 (0.4) 2.1 (0.2) 0.4 (0.1) Bulgaria 33.3 (1.9) 23.3 (1.1) 22.1 (1.0) 14.1 (0.8) 5.6 (0.7) 1.4 (0.3) 0.2 (0.1) Montenegro 30.0 (0.8) 26.8 (0.8) 23.9 (1.0) 13.8 (0.7) 4.6 (0.4) 0.7 (0.2) 0.1 (0.1) Uruguay 32.4 (1.6) 25.6 (1.0) 22.4 (1.0) 13.2 (0.7) 5.3 (0.5) 1.1 (0.2) 0.1 (0.1) Colombia 33.2 (1.7) 28.3 (1.1) 22.2 (0.9) 11.3 (0.8) 3.9 (0.5) 0.9 (0.2) 0.2 (0.1) OECD average 8.2 (0.2) 13.2 (0.2) 22.0 (0.2) 25.6 (0.2) 19.6 (0.2) 8.9 (0.1) 2.5 (0.1) Note: Countries, economies, and provinces have been sorted by the total percentage of students who attained Level 2 or higher. Assessment Matters! No. 6, CMEC 7 At the provincial level, the proportions of students reaching at least Level 2 varied from 78.8% in Prince Edward Island to 87.5% in British Columbia. At Level 5 or above, percentages ranged from 7.3% in Prince Edward Island to 19.1% in British Columbia. Figure 3 Distribution of students by proficiency level on the problem-solving scale, Canada, provinces, and OECD NL PE NS NB QC ON MB SK AB BC Canada OECD Average Percentage Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 and above In Canada as a whole, slightly more
Recommended
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x