Maria-Josep Cascant i Sempere

One of the most exciting innovations in adult literacy over the last 15 years has been the development and spread of the Reflect ap proach, which won the UN Literacy Prize in 2003, 2005 and 2007 and is now used in over 500 organisations in 70 countries. Reflect has been successful in linking the literacy acquisition process with individual and community empowerment – strengthening the capacity of millions of people to secure their basic rights. Reflect programmes operate in hugely diverse contexts, and approaches to documentation and evaluation have been equally diverse – making it difficult to consolidate evidence and learning. In a project coordinated by the South Africa Reflect Network (SARN), Reflect practitioners have come together internationally to develop a new evaluation framework for Reflect, a flexible tool that can be adapted to local contexts but will allow learn ing to be shared effectively across organisations and programmes. As part of this project, for six weeks during June and July 2008, 69 Reflect practitioners from 36 countries joined an online network to debate the evaluation of Reflect and discuss ideas for a future framework. The author is Reflect Evaluation Framework Coordinator from SARN.

Evaluating Literacy: The Process of Creating an Evaluation Framework for Reflect

 

Why an Evaluation Framework?

The importance of evaluating literacy and non-formal education has been long stated by various institutions. UNESCO, through its Edu cation for All (EFA) Global Monitoring Report, declared in 2001 that:

"Current EFA monitoring systems are mainly relevant to formal educa tion" and "the important role played by NFE programmes in attaining EFA goals is often underestimated." In view of the upcoming global 2009 CONFINTEA meeting in Brazil, the need to gather new evidence about the effectiveness of literacy programmes and the links between literacy and other development goals is even more pressing. If we cannot prove the high development returns provided by investing in literacy, demands made at CONFINTEA will go unheard no matter how loud the cry.

In this context, the International Adult Literacy Benchmarks will sup port adult literacy practitioners, governments and donors in tracking and aiming for effective policy and implementation. Benchmark 4 states that: "It is important to invest in ongoing feedback and evalu ation mechanisms." This Benchmark is not only essential in terms of demonstrating literacy dividends but also in terms of learning what does and what doesn't work in policy. Transversal to all the Bench marks, it is only through the monitoring and evaluation systems high lighted in Benchmark 4 that the other 11 benchmarks can be "filled up" to provide orientation. The Reflect Evaluation Framework project aims to contribute to these evaluation efforts.

The Reflect approach to adult literacy and empowerment is widely considered to be a highly effective force for social change and for the spread of democratic values from the ground up. There are many ex amples of programmes that report significant development benefits ranging from improved health, women's empowerment, diversified and enhanced livelihoods, to active citizenship, HIV prevention and girls' education, suggesting that an empowering adult learning proc ess may provide an invisible glue for attaining the Millennium Devel opment Goals. However, in practice, there is a desperate shortage of effective evaluation evidence. Reflect programmes operate in hugely diverse contexts, and approaches to documentation and evaluation have been equally diverse – making it difficult to consolidate evidence and learning. In the face of a tendency towards donor-led evaluations there are not always the space, time and resources for learning and sharing amongst Reflect practitioners and for the development of participatory evaluation to be properly developed and coherently implemented.

Literacy course in Bangladesh

Literacy course in Bangladesh, Source: Liba Taylor/ActionAid 


Towards a Framework for Evaluation

The most recent attempt to consolidate learning from evaluations of Reflect was seven years ago in 2001, when a consultant, Abby Rid dell, carried out a review of 13 external evaluations of Reflect from 11 countries in order to: 1) synthesise and summarise the 13 evaluations, identifying trends in literacy and empowerment outcomes; 2) identify and classify trends in approaches to evaluation and the indicators used; and 3) draw out some key hypotheses to frame further research/ evaluations. However, the absence of a uniting methodology for the evaluations presented serious challenges for drawing out wider conclusions. A number of evaluations of Reflect projects have been carried out in the intervening seven years, both at local, regional and national levels. However, there is still no uniting methodology for evaluation and the projects themselves vary greatly in their focus and approach. In order to feed into the process of developing a new evaluation framework, a wide selection of these evaluations has been analysed in order to draw conclusions regarding good practice in evaluation.

Parallel to this process, in April 2007 ActionAid developed a draft evaluation framework that was piloted in South Africa in May. In Oc tober, a workshop was held to sensitize UK-based PhD and Masters students to the initiative and critique the framework, and in Novem ber 2007, a second workshop took place in Cape Town to share the framework with over 30 Reflect practitioners from across Africa and South Asia. To further develop and disseminate the framework, an interim Evaluation Framework Team was formed from November 2007 to April 2008, when SARN appointed the Reflect Evaluation Framework Coordinator.

On-line debates: "À vos claviers!" [ "To your keyboards!" ] During May and June 2008, on-line debates about evaluating Reflect in four languages (English, French, Portuguese and Spanish) were conceived. Contacts were made worldwide to invite Reflect practi tioners to participate. Discussions started at the beginning of June 2008 and will continue for six weeks. Topics covered include: evalu ation, literacy, empowerment, power and the literate environment, project/capacity building indicators, advocacy and communication.

At the time of writing a total of 36 countries and 72 practitioners
are participating, with new members joining all the time.
* Anglophone group: 14 countries and 34 practitioners
* Francophone group: 13 countries and 23 practitioners
* uso/Spanish-speaking group: 9 countries and 15 practi tioners (Spanish-speakers: 5 countries
   and 9 practitioners; Portuguese-speakers: 4 countries and 6 practitioners)

    Participant countries are: Angola, Bangladesh, Basque Country, Benin, Brazil, Burkina Faso, Burundi, Cameroon, DRC, Ethiopia, France, Guatemala, Guinea Bissau, Guinea Conakry, India, Kenya, Liberia,
    Madagascar, Malawi, Mali, Mozambique, Nicaragua, Niger, Peru, The Philippines, Rwanda, El Salvador,
    Senegal, South Africa, Sudan, Togo, Uganda, UK, Vietnam, Zambia and Zimbabwe

    Languages: Participants are pleased to be able to contribute in their familiar, national languages during the process . It is fulfilling for those involved to see four different linguistic communities working simultaneously towards the same goal. Given the smaller size of the Spanish and Lusophone groups, a decision was made to merge these two language groups into a larger bilingual one. In this group, the facilitator makes presentations in both languages with participants contributing in either Portuguese or Spanish. The written closeness of the languages helps the group in understanding the contributions.

    Group Diversity: The groups are made up of both senior and junior practitioners of the Reflect approach, as well as some former practition ers now working in other areas of development. This has allowed for diversity and the sharing of experiences. Particularly noteworthy is the involvement of practitioners from a variety of INGOs, including Action-Aid, GOAL, Intermon OXFAM and PLAN International. So is the union of the different Reflect regional networks in this project (PAMOJA, Latin America Reflect Network, Asian Reflect Network). Not to mention the vast representation of national and local associations (AAEA/Angola, CEREBA/DRC, CHIKUKWA/Zimbabwe, CIAZO/El Salvador, J&D/Mali, MIDE La Chuspa/Peru, PAF/Zambia, RESODERC/Togo …).

    Quality of Participation: Discussions are open and space is left for participants to propose their own topics and ideas – which did not take long: during the first week, a participant from Vietnam proposed the inclusion of the topic "social change" ; the topic of "advocacy" was proposed from South Africa, "capacitation" from Togo and "systems of communication" from Burkina Faso.

    On the Need for the Project: Participants have welcomed the project. The wide interest in the project originates in the fact that the project covers at least two perceived gaps: first, a gap in evaluation tools and culture; and second, the need to strengthen information exchange and networking amongst Reflect practitioners globally, in partnership with existing national and regional networks. This second point is in itself an evaluation issue, in terms of learning from each other both good and bad practices (formative evaluation).

    Evaluation: As for the outcomes of the discussion, it is still too early to speak as we are just in our second week as I write; but confidence exists that the whole debate will have significant implications for net working, for literacy-for-empowerment visibility and for the develop ment of participatory evaluation in general, beyond Reflect. So far, we have kept on typing up ideas in our keyboards – or as a Francophone participant exhorted in her intervention to fellow group members: "À vos claviers!"

    What Next?

    Once the online discussions come to an end in early August, a sum mary document will be shared in the four languages with a broader audience working in adult literacy, who will be invited to contribute. This includes Reflect practitioners who were not available to take part in the on-line discussions (including practitioners from countries not yet represented – Belgium, Ghana, Lesotho and Sierra Leone). It will also involve other adult literacy practitioners interested in contributing to, learning from and advocating for this evaluation project. In Africa, this will involve members of ANCEFA, PAALAE and FEMNET which, together with PAMOJA (Africa Reflect Network), make up the "African Platform for Adult Education". Along with the report analysing existing evaluations of Reflect, the on-line discussion summary will feed into a document "Evaluating Reflect and Adult Literacy", which will be ready for advocacy during the African Regional CONFINTEA meeting to be held in Nairobi in November 2008.