The New PISA Rankings are coming to town!

De Finnish education expert Pasi Sahlberg (twitter) has mentioned that Finland is going down on the PISA rankings of mathematics achievement of 15 year old students. PISA, of which the 2012 data will be released on Tuesday December 3rd, is an important international assessment of student skills collected every three years. Education policy in many western and non-western societies is affected by student performance in PISA. The decline of the Finnish position is remarkable, given that Finland is typically ranked among the top positions in international skill assessments, including grade 4, grade 8, fifteen-year-olds, and, recently in the PIAAC data, also among adults. Delegates of many countries have gone to excursions to Finland to try to see why Finland has been doing so greatly. The next destination of education experts may be Estonia, that is said to have moved up. Also the Netherlands have gone up, rumours say, at times when the government has increased the emphasis on the core competencies of mathematics and language. Let me guess what our minister will say on Tuesday!

The problem with understanding how well an educational system performs is that rankings do not say much. Explanations for the Finnish miracle have included teacher autonomy, teacher professionalization, a low number of migrant children, and the introduction of an egalitarian comprehensive schooling system until the age of 16 (as opposed to a tracked system as we see in, for instance, Germany and the Netherlands). However, to understand whether such factors contribute to student achievement we should better isolate such factors and try to research them, rather than to go to one country and see which compendium of factors could have contributed to the miracle. The knowledge that emerges out of in-depth studies of one whole educational system has a high teleological character, similar to many historical accounts of great events, that come up with a whole array of factors that have changed history. It was the only possible outcome, we may tend to think. The problem with such forms of knowledge is that they do not emerge from a falsificationist research endeavour, which can easily lead to ‘explanations’ that are, in fact, not very sound.

As said, it would be much better to isolate factors that may contribute to student learning, and see whether they contribute to student learning. Basically two sorts of research can be carried out: one focusing on a specific education policy reform, and one concentrating on a broad spectrum of countries, thereby identifying institutional characteristics that may benefit learning, or reducing social inequalities in achievement.

In the GINI research project (Growing Inequalities’ Impacts) we have analyzed cross-national variation in student achievement, and in inequality therein by social class origin. In the forthcoming GINI volume at Oxford University Press some of this evidence can be found (joint work with Daniele Checchi and some other education economists). It is useful to show some descriptive graphs of achievement outcomes, including the mean performance, the performance at the ‘bottom’, the performance at the ‘top’, and the country-specific relationship between social origin and student outcomes. I focus on early tracking, the vocational orientation of the educational system, standardization of input (curriculum standardization, textbooks used), and standardization of output (the existence of central exams). The educational institutional variables have been constructed in joint work with Thijs Bol (twitter).

The first question is whether early selection is good for performance. Whereas Sahlberg claims that the postponement of the moment of selection in Finland has contributed to higher achievement, typically proponents of early tracking argue that students of diverse teaching needs can all be taught more effectively in homogeneous (i.e. tracked) classes. Early tracking should then be associated to higher levels of student performance. Looking at the PISA 2006 data we don’t find any evidence for such a relationship (Figure 1).


Figure 1: early tracking and student achievement at the mean and the 75th percentile



So if the gains are not achieved by early tracking, are there downsides to the early separation of students? Let’s have a look at figure 2, which plots two types of inequality by the level of tracking: inequality in terms of dispersions (within-country standard deviations in math), and inequality by social origin (the regression slope of social background on math test score). Figure 2 shows clearly that both types of inequalities tend to be comparatively high in more strongly tracked systems. Even though it is perhaps not desirable to eliminate completely, it is striking that early selection may aggrevate inequalities in learning among non-adults.

Figure 2: tracking and two types of inequality in student performance

The following graph (figure 3) examines standardization of ‘input’, which, if anything, doesn’t help to improve learning.

Figure 3: student performance across levels of standardization of input in an educational system


Figure 4 shows whether centralized exams are related to student performance (again using PISA 2006). The average performance, the performance at the bottom and at the top, are not related to whether a system imposes centralized exams onto schools. However, the dispersion in student learning is significantly lower in countries with centralized exams than in countries without centralized exams. I have never quite understood the criticism on central testing, and here is why: centralized testing reduces inequalities between schools, and through that, between social classes. A great gain.

Figure 4: centralized exams and student achievement



Figure 5 shows, lastly, the association between the vocational orientation of the upper secondary system (in the Netherlands: the share of MBO across all upper secondary levels including HAVO/VWO, quantified in a factor score with data from two sources) and adult literacy (calculated on the IALS data of the mid 1990s, which are a bit old; whenever I have time similar graphs can be made with PIAAC). A strong vocational orientation of the system, which is typically found in Germany, the Czech Republic, and Germany, is positively related to adult literacy. Thus, while early tracking (in lower secondary education) magnifies inequalities, a strong vocational specificity is not harmful to learning, quite the contrary. Early selection comes at a cost, but it should not necessarily lead to an abolishment of vocational education at the upper secondary and tertiary levels.

Figure 5: The vocational orientation of the system and adult literacy



I would recommend policy makers to look more analytically to the case of individual countries. Do not go to delve into one specific highlight, as a country’s position may change from time to time. Economists have developed great designs to study the small-scale impacts of reforms, and sociologists have studied the impact of institutional environments. Both these approaches help us to understand which policy measures are related to student outcomes, and which aren’t.