header EEA ROIndependent Evaluation of the Food Coupons Program

Executive Summary The program

The program Fiecare Copil în Grădiniţă (Every Child in Preschool, FCG) is designed to increase early education enrollment and attendance, and by so doing to correct the root causes of school abandonment. The program (1) organizes communities to prioritize early education and (2) incentivizes impoverished parents to send their 3- to 5-year-old children to kindergarten. Since its inception in June 2010, each year 1300-1500 children in 25 rural and semi-rural municipalities have participated in the program.

FCG offers food coupons, worth 50 RON per child per month, to eligible families with children of kindergarten age in the localities where it operates. Initial eligibility depends on certain poverty criteria, and each month families receive food coupons if they comply with the program conditions. The conditions are that (1) the child must attend kindergarten regularly, and (2) at least one parent must participate in kindergarten activities at least once per month, as well as be present for the monthly parents’ meeting. In each municipality where it operates, FCG helps set up a local action group made up of local decision makers, teaching staff, and social services personnel, which handles various aspects of the implementation of the FCG program. In addition, FCG offers teacher trainings, requires public officials to intensify parent counseling, works with teaching staff to ensure accurate attendance monitoring, and supplies kindergartens with educational materials.


This evaluation

In 2015, the implementing organization Asociaţia OvidiuRo (OvR) decided to commission an impact evaluation of the FCG program, in order to (a) be able to convincingly demonstrate the assumed impact of the program to policy makers and potential donors, as well as to (b) identify ways to improve the implementation of the program. The guiding research question of the impact evaluation is:

What is the impact of the FCG program on children’s enrollment, attendance and performance in school, after they graduate from preschool and the FCG program?

The evaluation has two research goals. The primary goal is an assessment of FCG’s impact on enrollment and attendance of beneficiary children in primary school. An assessment of the program’s impact on enrollment and attendance in kindergarten supports the credibility of these findings. Secondly, the evaluation identifies contextual factors and processes that are likely to affect the impact of the program. This assessment also allows us to give a qualitative answer to the question how the program brings about changes in enrollment and attendance of poor children.


Evaluation approach, methods and limitations

The impact evaluation is based on two bodies of evidence: (i) 172 semi-structured interviews with local education stakeholders in 16 rural and semi-urban municipalities in Romania, as well as (ii) primary and secondary quantitative data sources on educational outcomes and socio-economic indicators, at the municipality and individual level, in 13 program and 26 comparison municipalities. There are three main data sources. First, we directly collected attendance data: study staff performed in total 3236 surprise visits to record attendance in 541 kindergarten and lower primary school classes in the 39 municipalities. Secondly, we collected enrollment data across time from the National Institute of Statistics (INS) and the Ministry of Education (MinEd). Thirdly, we directly collected enrollment and attendance data for the 2014-2015 academic year from school ledgers in the same schools where surprise visits took place.

Overall, we present results from data on approximately 10,000 children each year, from program and comparison communities. However, our study covers only a relatively small number (39) of study municipalities, and the comparison communities offer only an approximation of what would have happened in the program communities, absent FCG. Also, the focus of the study is on municipalities

that enrolled in the first two years of the program, while the implementation of the program may have evolved in the meantime. These study limitations should be taken into account when interpreting the results.


Main findings

1. FCG has a substantial, positive impact on enrollment

The FCG program has large and significant impacts on preschool and lower primary school enrollment. We arrive at this conclusion based on the comparison of INS enrollment data for program and control communities before and after the beginning of FCG implementation, as well as on the comparison of enrollment data from the Ministry of Education for program and control communities for the academic year 2014-2015, which we cross-checked against school ledgers.

Using INS data, we find that children between 3 and 6 years old in FCG municipalities were 10 percentage points more likely to be enrolled in the educational system (preschool or clasa pregătitoare).* Based on the average number of children in the respective age bracket per community, this means that about 300 children in program municipalities enrolled in the educational system, who would have not done so in absence of the program. This represents approximately 30% of the total number of FCG beneficiaries. In other words, about a third of the children who are FCG beneficiaries would not have been enrolled at all without the FCG program.

Based on MinEd data, we found that in the 2014-2015 academic year, 6-year-olds in FCG municipalities are significantly less likely, by 25% to 30% percent, to not be enrolled in school. Concretely, in comparison municipalities we estimate that 12.8% of 6-year-olds are not enrolled in school at the beginning of the school year, while in program communities this number is only 8%. These results are very robust when we control for municipality socio-economic differences between the FCG and the comparison groups, such as total population or ethnic composition.

2. FCG has a positive impact on attendance

It is not a priori clear how the FCG program’s impact on enrollment translates into higher attendance. First, students may enroll but never or rarely show up to class. Secondly, the FCG program likely induces students from more difficult situations (the “difficult cases”) to enroll, and these students may have lower than average attendance. This may drive the average attendance rate down, even though more students are attending. To counter the latter problem, we compute attendance relative to the total number of children in a given age group.

The results indicate that FCG had a significant impact on attendance in both preschool and lower primary school, although this effect is not as large as the effect on enrollment. This conclusion is based on a comparison of attendance rates in program and control municipalities for the second half of the 2014-2015 academic year. Attendance rates were calculated by averaging attendance taken in repeated surprise visits of all kindergarten and lower primary school classes in the 39 study municipalities.

In program municipalities, an average of 41.9% of all children aged 3 to 5 were attending preschool on a given day. This compares to a rate of 38.3% for control municipalities. These numbers do not take into account excused absences due to illness, teacher training (cerc pedagogic) or early dismissal. This is one reason why the attendance rates reported here are lower than the numbers previously reported by OvR.

The difference of 3.6 percentage points in overall attendance is driven by the attendance of poor children, i.e. children eligible for the FCG in program and control municipalities. Poor children (in this definition) make up roughly 30% of all children in program municipalities. Hence, we assume that the difference in attendance rates among eligible, poor children is 3.3x larger than the difference in overall attendance rates between program and control municipalities.

We therefore estimate that the FCG program increased preschool attendance among poor children by ca. 11.8 percentage points. As the average attendance for all students is roughly 40% (41.9% for program and 38.3% for control municipalities), the FCG program is likely to have increased preschool attendance among its target group by at least a third (11.8/40.0 = 29.5%). These are conservative estimates.

For lower primary school classes (clasa pregătitoare and first grade), average attendance rates in program municipalities were 60.0% and 68%, respectively. This compares to average attendance rates for control municipalities of 58.9% and 64.6%. The impact of the FCG program on overall attendance rates is therefore estimated at 1.2% for clasa pregătitoare and 3.5% for first grade.

Taking into account the average ratio of non-poor over poor children in program municipalities, we arrive at an impact of the FCG program on attendance rates among poor children of 3.5% for clasa pregătitoare and 10.5% for first grade.

Note that the result for clasa pregătitoare is smaller than for enrollment, and the one for first grade is significant and large. These results indicate that the impact of FCG on attendance for early primary school cannot be identified precisely, and these results should be interpreted with care. Overall, they are compatible with a positive and either modest or large effect of FCG on attendance of children who have finished the FCG program and continued to school.

3. Official kindergarten and school attendance records are likely to be inflated

We have found that official attendance data as recorded in school ledgers and compiled by the Ministry of Education are not a reliable source of information for the calculation of attendance rates in the poor rural and semi-urban municipalities we have studied. In the 39 study municipalities, official attendance records report absence rates up to 50% lower than indicated by our calculations based on more than 3000 surprise visits.

In our more than 170 qualitative interviews with local stakeholders in 16 municipalities, we have identified processes that might provide an explanation for these stark differences. We believe that official attendance records are inflated because teachers have a strong interest in high attendance rates. This might lead them to at least occasionally manipulate attendance data, for two reasons: To secure their jobs, and out of sincere concern for the well-being of poor children.

Teachers’ job security depends on attendance rates because the number of teachers in a given kindergarten or school depends on the number of children enrolled and attending. A concern for the well-being of poor children might drive the inflation of official attendance records because social security regulations in Romania link several types of state support to school attendance, such as family assistance (alocația pentru susținerea familiei).

4. Socio-economic context of municipalities influences FCG program’s impact

Based on our interviews with local stakeholders, we have identified several ways in which the socio- economic context of municipalities is likely to affect the FCG program’s implementation and impact. In the following, we will concentrate on three of these context factors: The proximity of municipalities to urban centers, local school policy and the extent of work migration among parents in a given municipality.

We believe that semi-urban municipalities may have decidedly different incentive structures to cooperate with the FCG program than rural municipalities. The proximity of an urban job market and higher property prices in the city provide semi-urban municipalities with the opportunity to carve out a development strategy: Attract young families with jobs in the city, and reinvest their tax money.

Commuting parents, however, can take their children along on the way to their urban workplace and leave them in a kindergarten or school in the city. That is, in semi-urban municipalities, local institutions of public education compete with city kindergartens and schools for the children of these parents. This means that due to the influx of young urban parents, local poor children are neither necessary for local educational institutions to secure the jobs of teaching staff, nor are they wanted there, as they make local kindergartens or schools less attractive to better-off, commuting parents.

Hence, an FCG program that promises to increase attendance among poor children may run counter to the development logic of semi-urban municipalities.

Local school policy more generally might affect FCG implementation and impact as well. Among the schools we visited during field work, we identified two ideal types of institutional self-understandings: The school as an “educational center”, and the school as a “social institution”. According to the former self-understanding, a school has the duty to offer the best quality education possible to those in the community that value education. It is the duty of parents to understand this value. This implies concentrating efforts on those pupils with good future prospects. According to the latter self- understanding, a school has the primary duty to provide some level of education to all children in the municipality, whether they come from families that value education or not. This implies that outreach is an important activity of the school. Given the same amount of resources, the differences in self- understanding may translate in different school policies vis-à-vis poor children in the community: Ignorance versus outreach. The FCG program may find it harder to mobilize local action groups for program implementation in municipalities where schools adopted the “educational center” approach to public education.

Work migration is a significant social phenomenon in all rural municipalities that we have studied. Poor parents with low levels of education who work abroad temporarily or semi-permanently leave their children in most cases with their partner or – if, for example, both parents work abroad – other relatives in their home towns. This has several consequences for the education of these children. First, single parents and relatives may be overwhelmed with the task to take care of these (additional) children which often negatively affects regular school attendance. Secondly, due to the larger earnings of parents working abroad, the economic situation of the household typically improves which helps to overcome the “hidden costs” associated with school visits (proper clothes and shoes, daily break time snacks) as a barrier to school attendance. Thirdly, from our interviews with parents we have learned that their experiences as unskilled workers abroad typically increase their appreciation for basic skills such as reading and writing. Owning a driver’s license or understanding a contract makes, for example, a difference in income in their work environment abroad. As a consequence, they might be more likely to increase attention to and investments in the primary school education of their children. Temporary work migration is therefore an important context factor for the FCG program.


Recommendations

1. Upscale the program

The FCG program has a significant impact on both enrollment and attendance in preschool and lower primary school. With 50 RON per child per month, it offers a cost-effective way to incentivize poor parents to send their children regularly to preschool. This behavior change seems to translate to an increased willingness to attend school well beyond the end of the program. Moreover, the FCG program seems to work in diverse settings: Large and small, rural and semi-urban municipalities with Roma communities that make up a significant proportion of the local population and those that do not. We therefore strongly recommend the upscaling of the program to the national level.

2. Consult with local stakeholders broadly

Our more than 170 interviews with a diverse set of local stakeholders have shown that they play important roles for the implementation and the impact of the program. Whether mayors, social workers, teachers, or parents: They all have possibilities to undermine the program logics in important ways, or – through their personal support – improve the program’s impact significantly.

Currently, all of these local stakeholders are regularly included in program implementation or consulted by FCG monitoring teams. If this were no longer the case, the sometimes divergent incentives of local stakeholders may translate in actions that could harm the success of the
program. We therefore recommend to keep up a broad basis for local stakeholder implication and consultation.

3. Monitor attendance independently

Our research has shown that local stakeholders often do not record preschool and school attendance reliably. We believe that this is partially because they have personal and social interests in inflating attendance records. For a conditional cash transfer program, however, reliable monitoring of conditions is vital. We therefore strongly recommend to continue implementing a mechanism for the monitoring of local attendance record keeping by an independent third party.


See full report here. The report of the Impact Study was presented by researcher Wolfgang Stuppert and OvidiuRo Cofounder, Maria Gheorghiu at the Galway conference, organised by EuroChild – Child Rights in Practice and Research. You can watch the presentation below, starting at 5:57:00:

The Every Child in Preschool project and results have been published on EPIC (European Platform for Investing in Children), a unique evidence-based platform designed to encourage knowledge sharing and collaboration among policymakers and practitioners. On the EPIC website’s ‘Practices that work’ section, practices are listed that have been formally evaluated and have demonstrated their effectiveness through rigorous research (‘Evidence-Based Practices’) and practices that are promising and are shared to support learning across the EU community of policymakers and providers.

* Keep in mind that enrollment rates are computed for all children, and in both FCG and comparison municipalities many children would have been enrolled regardless of the program.