• Did the Ministry of Education try to rig the PISA 2015 sample schools in order to artificially boost Malaysia’s scores?

    Media Statement by Dr. Ong Kian Ming, MP for Serdang, on the 8th of December, 2016

    Did the Ministry of Education try to rig the PISA 2015 sample schools in order to artificially boost Malaysia’s scores?

    When the Program for International Student Assessment (PISA) 2015 results were released on the 6th of December, 2016, officials from the Ministry of Education (MOE) took pride in the fact that Malaysia’s PISA’s scores for Mathematics, Reading and Science had improved from 421, 398 and 420 respectively in 2012 to 446, 431 and 443 respectively in 2015.[1] No doubt, Ministers, Deputy Ministers and politicians from the Barisan Nasional (BN) will use the latest PISA scores as ‘proof’ that Malaysia is on the ‘right track’ when it comes to the standard of education in the country. What they would have conveniently left out is the fact that Malaysia does not feature anywhere in the 2015 PISA rankings for Mathematics, Reading and Science.

    The official reason stated in the PISA report for Malaysia’s non-inclusion is:

    “In Malaysia, the PISA assessment was conducted in accordance with the operational standards and guidelines of the OECD. However, the weighted response rate among the initially sample Malaysian schools (51%) falls well short of the standard PISA response rate of 85%. Therefore, the results may not be comparable to those of other countries or to results for Malaysia from previous years.”[2]

    Why was it that only 51% of the schools initially chosen for the PISA test participated in the test in 2015? Was it because the Ministry of Education wanted to over-represent students from better performing schools and leave out students from low performing schools? This 51% participation rate raises many suspicions since Malaysia’s participation rate was 99.3% and 100% in PISA 2009 (151 out of 152 schools participated)[3] and PISA 2012 respectively.[4] It is hard to imagine any school principal not allowing his or her school to participate in the PISA 2015 test if the Ministry of Education had already chosen that school to be in the original sample.

    One suspects that the Ministry of Education over-sampled the high performing schools in the PISA 2015 sample and excluded some of the lower performing schools from the sample. For example, according to the website of the Negeri Sembilan Department of Education[5], the 14 schools listed as the PISA 2015 sample schools include all 7 (100%) of Negeri Sembilan’s secondary-level High Performing Schools or Sekolah Berprestasi Tinggi (SBT), and all 8 (100%) of its Fully Residential Schools or Sekolah Berasrama Penuh (SBP). The average student from a SBTs or SBPs will clearly outperform an average student from a regular secondary school.

    The evidence of a biased sample in favour of high performing schools can also been seen in PISA 2015’s own data on Malaysia.[6] Out of a total sample of 8861 students, 2661 or 30% were from fully residential schools (See Table 1 below). This is clearly an over sampling of students from fully residential schools since they only comprise less than 3.0% of the 15-year-old cohort in 2015.

    It is highly likely that those overseeing PISA 2015 saw that the Ministry of Education in Malaysia was trying to rig the sample size in order to artificially boost its scores. Is this why Malaysia was ultimately excluded from PISA 2015 rankings? The Minister of Education should explain so that we are not fooled into thinking that all is well and good in our education system as ‘evidenced’ by the latest PISA scores.

    Dr. Ong Kian Ming
    Member of Parliament for Serdang

    [1] http://www.thestar.com.my/news/nation/2016/12/06/malaysia-sees-improvement-in-pisa-scores/

    [2] From PISA 2014 Results (Volume 1), p. 340 (https://www.oecd.org/publications/pisa-2015-results-volume-i-9789264266490-en.htm):

    [3] See http://research.acer.edu.au/cgi/viewcontent.cgi?article=1000&context=pisa, p. 103

    [4] See https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf, p. 181

    [5] http://www.spa.jpnns.gov.my/v4/viewpage.php?page_id=1

    [6] The document “Codebooks for the additional files for Albania, Argentina, Kazakhstan and Malaysia” (https://www.oecd.org/pisa/data/2015database/Codebook_CM2.xlsx, see the variable named “STRATUM”) states that students in Malaysia’s sample came from the following types of schools

Leave a reply.