Recalculating Proficiency in Schools: COVID-19 Related Learning Losses and Dropping Out

Traditional ways of measuring educational proficiency, based on assessment results and the percentage of children enrolled in schooling, may not be as effective a measurement while COVID-19 impacts upon attendance rates.

Analysts working with SDG indicators on learning proficiency, in particular Indicator 4.1.1, would in many cases be familiar with a formula such as the following:

The percentage of enrolled learners who are proficient, S, is typically known from assessment results. The percentage of children of the age of the grade in question who are not enrolled, N, is typically known from household data. is what planners are ultimately interested in: the percentage of the relevant age cohort in the population who are proficient. While there is some ambiguity over whether Indicator 4.1.1 refers to or Pis clearly important and must be monitored.

The above formula assumes that non-enrolled children do not reach minimum proficiency levels, following the approach taken in UIS (2020) and UIS (2017). The relationships can be illustrated as in the following graph, which is based loosely on levels of learning proficiency and the out-of-school problem in developing countries. Here is a function of and N, according to the above formula. 

In the context of COVID-19, sudden declines in enrolment, meaning rises, and declines in proficiency in the population due to disruptions in schools, meaning drops, are possible. In examining the effects on the three statistics Pand S, it is important to remember that and influence each other. Assuming that children who drop out of the schooling system are the academically weakest learners, an assumption that is likely to hold for a number of reasons, one can expect S either to rise or decline, depending on the magnitudes of the two effects: dropping out of school and learning losses. If one has estimates of and N, then the new is calculated according to the following transformation of the previous formula:

To illustrate the ambiguity, we can imagine learning losses resulting in a drop in from the 35% seen in the graph to 33%. If out-of-school increases from 5% to 9%, declines from 37% to 36%. However, if displays a larger increase, from 5% to 13%, then displays an increase from 37% to 38%. Even with learning losses in the population of a specific age, if enough learners drop out, and we assume that these learners are those who struggled most academically, then assessment systems may in fact detect an increase in the percentage of proficient learners.

Clearly, planners need to be fully aware of these relationships. Above all, the first formula above should not be used to conclude that more out-of-school children on its own produces a decline in P. If was 37% (as in the graph) and then out-of-school increases from 5% to 13%, one cannot conclude that drops to 32%. This would ignore the fact that if changes, automatically changes too. 

Projections For Learning Proficiency Can Inform Post COVID-19 Educational Strategies

The UIS will soon be releasing its projections of the impact of COVID-19 on the learning proficiency of children and youth, building upon previous UIS projections published in early 2020, before the pandemic. Projections of this kind are vital and inform the planning conducted by national governments, global bodies, and development assistance organizations. 

The data suggest that the impact of the pandemic on learning occurs through two distinct channels:

  1. Disruption in schooling has denied learners their usual classroom experience with teachers and other children. Disruptions have been particularly serious for younger children, who are most in need of contact teaching, and often receive meals through the school. These needs are felt especially in developing countries.
  2. The economic shock of the pandemic has put education budgets under pressure, and worsened household poverty. These factors also impact negatively on learning. 

The findings suggest that the effects of school disruptions on learning proficiency could be felt for many years, even after the pandemic is declared over. In future years, however, as children who did not directly experience the disruptions enter the education system, the longer-term budgetary and poverty effects will be of greatest significance. It is important for planners both to mitigate the effects of the disruptions, and to plan for the educational well-being of future age cohorts of children not affected directly by the disruptions. 

Current focus

The following diagram outlines the variety of factors influencing any schooling system’s ability to minimize learning losses: 

This illustrates how future priorities depend to a large degree on the strengths and weaknesses of the schooling system that existed before the pandemic. Systems with effective support and accountability structures, and which had been experiencing improvements in learning proficiency before 2020, are likely to be the most resilient to the shock of the pandemic. 

Good national assessment programmes have become more important than ever. Countries with such programmes will soon have an idea of the magnitude of the pandemic-related learning losses, information which is vital in planning the recovery process. Initial studies of the actual (not projected) magnitudes of these losses have started to emerge. Countries without good national assessment programmes, need to invest in this now. This will enable them to gauge how proficient children are relative to children in other, similar countries, and to gauge whether improvement occurs.The following map is based on UIS.Stat data and points to clear regional and school calendar differences: 

Note: ‘A’ means school year starts in January to April, ‘B’ that it starts in July to October.

Categories of school year

The data show that having a school year starting in January to April (‘A’ in the map) is associated with 11% more of the school year having been disrupted. Being in Latin America and the Caribbean is associated with a further 11% increase. 

To illustrate the effects, in grade 3 reading the pandemic-induced disruptions lead to an immediate decline in global learning proficiency, from 59% proficient in 2019, to 50% in 2020. This takes into account the fact that if a learner loses 50% of the school year due to disruptions, the learner loses more than 50% of a year’s learning. Given a global learner-weighted average of 50.3% of the year lost, counting the situation up to the end of September 2020, around a year’s worth of learning lost globally is the estimated result. Further, the data suggest that the more disadvantaged learners experience both greater school disruption and poverty effects.

Beyond 2020

Two key factors influence the future trajectory:

  1. The 2020 disruptions experienced by learners entering grade 3 in future years – linked to the extent of the disruption to pre-schooling. 
  2. The ability of schooling systems to catch up to where they should be in terms of fundamental skills, particularly reading and mathematics. 

Analysts have warned of the risk of a scenario which is even worse than zero catching up. This would be a scenario where beyond 2020 children fall even further behind the pre-pandemic trajectory because teachers fail to adapt their teaching – essentially, continuing to teach as if the pandemic had not happened. Such a scenario should be avoided, through appropriate messaging and support to teachers, with a focus on the learning outcomes of learners in each school, and in the system as a whole.

Disruptions to pre-schooling would affect, roughly, those children taking grade 3 in 2023 and during a number of subsequent years. Evidence suggests institutions offering education to children below the primary level were disrupted as severely as primary schools. While there is little data on the quality of pre-schooling across the world, UIS.Stat provides statistics on national coverage of the population at this level. 

In general, countries with more pre-schooling have displayed better grade 3 learning proficiency, though the correlation is relatively weak. Better performing countries are thus more likely to experience learning losses relating to the disruption of pre-schooling. By having greater pre-school coverage, these countries have more to lose.The following graph illustrates the estimated impact of the pandemic on global grade 3 reading proficiency. 

Global projections for Grade 3 reading

The model predicts a relatively large impact on cognitive development for those children who were in utero during 2020. This rests on evidence that in utero impacts of socio-economic shocks can in fact be even larger than impacts felt by infants who are already born. 

In the model, a return to the original trajectory occurs in 2030, for grade 3, or possibly earlier, depending in part on the success of catch-up programmes. This return can be expected to occur later for the upper primary or lower secondary levels, given those levels will have to deal for more years with children who in some way experienced a 2020 disruption. While a return to the pre-pandemic learning trajectory when children born after 2020 enter grade 3 may seem optimistic, it is not that unrealistic. But such an outcome depends on fundamental elements of the schooling system being preserved, and improved upon. 

Maintaining and enhancing the quality of the teaching profession is especially important, given that it is the capacity and motivation of teachers which lies at the core of improving learning proficiency. Teachers, like many other workers, are likely to see their real income decline as a result of the pandemic. This process must be well-managed and must be seen as fair by teachers. The entry of new and young teachers into the profession, as older teachers retire, is a key mechanism through which the quality of teaching is raised. The pandemic should not disrupt innovations in teacher training. Moreover, there should not be an interruption in the hiring of young teachers joining the profession.  

The regional picture

The next graph illustrates differences by world region. The Latin America and Caribbean region is indeed expected to see the largest 2020 losses, amounting a population-weighted loss of 1.21 years of learning, compared to a global average of 1.01. The second-highest 2020 loss is that of Central and Southern Asia, at 1.18 years of learning. The longer duration of proficiency deficits in Latin America and the Caribbean beyond 2020 is in part driven by the exceptionally large primary school disruptions in 2020, meaning catching up takes longer, but also by the fact that this region’s pre-primary coverage has been relatively good, meaning disruptions at this level affect more learners entering primary schooling. Eastern and South-eastern Asia also displays high levels of pre-school coverage, but here the smaller magnitude of the 2020 disruptions to primary schooling assists in bringing about a return to the pre-pandemic trajectory by 2025. Sub-Saharan Africa’s pre-school coverage is less than half of that in Latin America and the Caribbean, which translates to a relatively early return to the pre-pandemic trajectory. It is particularly important for sub-Saharan Africa’s schooling systems to recover, and to improve, given how low learning proficiency has historically been in this region.

Projections for Grade 3 reading with 10% catching up by world region

Closing remarks

The United Nations Secretary-General, António Guterres, echoed the concerns of people and organizations around the world when he recently referred to the impact of the COVID-19 pandemic on schooling as a ‘generational catastrophe’. Children and youth are falling behind in their learning, and this is expected to have an impact lasting decades. This said, the data suggest that there are actions that can be taken now to limit this impact and national governments, global bodies, and development assistance organizations have a role to play in implementing this through evidence-based policy choices and support. Equally, schools and teachers themselves have to adapt to the ‘new normal’ and focus on ensuring that current and future generations are taught in ways that minimize losses – essential if the SDG 4 2030 target is to be achieved.

Nurturing Education at Home in the Midst of a Health Crisis: How SDG Indicator 4.2.3 can Help Guide Where More Support is Needed

By Silvia Montoya, Director, UNESCO Institute for Statistics

In the midst of this global health crisis that threatens lives and containment measures that threaten our ways of living, we are faced with the stark reality that the world we return to will be forever altered. The far-reaching consequences of the COVID-19 pandemic may also jeopardize the hard won gains made in improving global education. 

The country statistics data collected and housed at the UIS has proved crucial in this time of crisis to help us estimate the global scope of the pandemic’s impact on education due to massive school closures. Given the importance of education as the foundation for all development, we must strive to safeguard learning at all ages. Thus, as a response, UNESCO has launched the Global Education Coalition in an attempt to support learning in the home as this becomes the new normal.

Continue reading