IOB has evaluated the centralised and decentralised instruments of the Matra programme of the ministry of Foreign Affairs. The evaluation examined the period from 2017 through 2023 and focused on the instruments’ relevance, coherence and effectiveness.
Background
The Matra programme is an integral part of the government of the Netherlands' budget, Chapter V Foreign Affairs, Article 2 for Security and Stability.
As part of the obligations under the Government Accounts Act (Comptabiliteitswet, article 3.1), the government of the Netherlands is required to evaluate the effectiveness and efficiency of public spending every five to seven years.
Moreover, this evaluation aims to facilitate learning by providing the Europe Department (DEU) with information on the relevance, coherence and effectiveness of (parts of) the Matra programme. Based on the specific knowledge needs of DEU, IOB has provided insights that can contribute to the programming cycle from 2024 onwards and to the new policy framework for 2025-2028.
This infographic shows a visual representation of the two evaluated Matra instruments and their associated theory of change.
The Centralised instrument is shown to contribute to a strengthened knowledge base, and strengthened cooperation between the Netherlands and the Matra programme country.
The Decentralised instrument is shown to lead to strengthened civil society and strengthened local governance.
Both instruments are shown to, in theory, contribute to
- strengthened rule of law, governance and democratization - on one side of the infographic
- strengthened bilateral relations - on the other side
Research process
The research process consisted of four steps:
- The theory of change of the Matra programme was reconstructed based on an internal document analysis and three focus group discussions.
- An evaluation matrix was created linking research questions to a set of indicators.
- Data was collected on the theory of change and the indicators of the evaluation matrix. This data included:
- a total of 91 in-depth interviews with staff of the MFA, including embassy staff, project implementers, foreign government officials, beneficiaries, other donors active in the same field and external experts
- field visits to case study countries Albania and Armenia, during which a second round of interviews took place and more detailed information was collected
- a document review of project proposals, mid-term and final evaluations of selected projects
- an in-depth analysis of external reports and databases on rule of law development and democratisation to provide additional background on the case studies
- The collected data was used to answer the research questions and to test the assumptions of the reconstructed theory of change.
Conclusions
IOB draws out two main conclusions based on this evaluation and provides specific conclusions on the relevance, coherence and effectiveness of the evaluated Matra instruments.
Main conclusions
A notable finding in reconstructing the theory of change for the Matra programme is that many MFA respondents identified the enhancement of bilateral relations as the ‘real’ raison d’être of the programme. This is not surprising, given that the focus of the MFA’s Europe Department is inherently political – i.e. policy coordination and development at the bilateral and European level – rather than programmatic. This evaluation finds that the extent to which the evaluated instruments contribute to macro-level changes in societal transformation processes remains unclear. The instruments do however contribute to the strengthening of bilateral relations. This underscores the political nature of the Matra programme.
A paradox thus emerges between programmatic effectiveness and political utility. The political nature of Matra involves keeping the programme’s objectives deliberately broad or ambiguous so the programme can be utilized by aligning it with political interests and evolving agendas. This can run counter to requirements for effectiveness such as formulating and achieving tangible, measurable outcomes based on predefined objectives.
Despite previous IOB recommendations to anchor the use of Matra instruments in a theory of change, there is still no Matra-wide intervention logic. As a result, there is no overarching level of monitoring and evaluation, which means that information on the achievement of goals is largely absent or anecdotal. Such a situation leaves virtually no room for strategic, feedback-based management of the various instruments and their coordination. While potentially valuable, the outcomes of one instrument are not used to reinforce the projects of another.
Conclusions on relevance
Many Matra programme countries have become more reform-oriented in recent years and have turned to Western countries for assistance. The relevance of the Matra programme as a whole is linked to where a programme country is in terms of credible accession or association prospects and, consequently, whether it is open to cooperation on rule of law and democratisation issues. The fact that Matra’s thematic priorities partly reflect the Copenhagen criteria links them to the countries’ accession or association aspirations and creates the potential to drive reform efforts in the broad sense.
However, requests for assistance from the authorities in the programme countries are not always clearly defined or translated into concrete action plans further down the political chain, which can lead to a lack of clear articulation of needs. Together with finding 1.1., this has the potential to further reduce ownership.
A changing (geo)political context can change the relevance of the centralised instrument, while this is less likely to impact the decentralised instrument. When a context changes from permissive to less permissive, and direct government-to-government cooperation – with the centralised instrument – is no longer possible, the decentralised instrument still remains relevant as it is also linked to the needs of civil society.
In general terms, the centralised projects reviewed are relevant. However, the centralised projects analysed lacked local ownership, due to structural issues in the operation of the centralised instrument. Over-reliance on a limited number of project implementers leads to supply-driven project design. In addition, the context and needs analysis of centralised projects was often insufficiently informed by local knowledge. This is exacerbated by the fact that during the evaluation period there was no formal requirement to involve a local implementer in the design or implementation of centralised projects, nor were embassies adequately consulted or integrated in the design and implementation of centralised projects.
In general, decentralised projects, such as those managed by Dutch embassies, are more responsive to local needs and take a more demand-driven approach to project design. They are flexible in nature and can quickly respond to emerging needs identified by embassies. In terms of challenges, the decentralised instrument is not always well connected to grassroots civil society in contrast to more established civil society organisations.
Conclusions on coherence
Even though the centralised and decentralised Matra instruments have similar overarching goals, there is a lack of coordination between the instruments. As a result of the lack of information sharing, centralised and decentralised projects remain in silos, hindering the ability to potentially align the approaches and find complementarities between the two instruments. This is a missed opportunity, given the significant potential for synergy between their different approaches. Furthermore, the disconnect between the two instruments hinders mutual learning within the Matra programme. In addition, the programme fails to facilitate the exchange of best practices between the two instruments, particularly with regard to cross-cutting issues.
The overlap observed between Matra and other Dutch funds was limited. The decentralised Human Rights Fund and decentralised Matra instrument showed significant overlap in terms of thematic priorities. The overlap thus does not lead to a duplication of efforts, but rather to a process in which both funds are used for the same purpose. In this sense, the decentralised Matra and the decentralised Human Rights Fund are not seen as similar, but rather as partially adjacent funds, which allows embassies to enjoy a certain degree of freedom to exchange project proposals between the funds.
The activities of the centralised and decentralised Matra instrument are broadly linked to Dutch foreign policy goals in the regions where they are implemented. They also signal a willingness to continue engagement through non-ODA assistance.
Given the political and non-programmatic nature of the Europe Department, the resources made available for project implementation are scarce. This, together with the outsourcing of the project management of the centralised instrument, means that opportunities for strategic or political guidance are missed.
Matra is a small fund in a country context where there are many and larger donors present. Combined centralised and decentralised Matra spending between 2017 and 2022 accounted for 0.31% of all donor spending on similar programmes in Matra programme countries. The added value of the programme lies in its niche-seeking behaviour, i.e. the fact that Matra projects focus on a small sub-theme within the broader funding efforts of larger donors such as the EU.
In the case of centralised projects, avoiding overlap is prevented by the Dutch implementing organisations themselves. Due to their limited presence in the programme countries, they are not always successful in avoiding the duplication of existing efforts. However, if overlaps are identified during project implementation, centralised projects are flexible enough to change course. Accordingly, the evaluation did not find much evidence of overlap in the projects examined.
In the decentralised process, overlap is avoided at the embassy level. In the absence of functioning formal donor coordination, the main way of avoiding duplication of existing efforts is through bilateral in-country consultations with like-minded donors, which has been a successful strategy for avoiding duplication among the projects reviewed. The niche of decentralised projects stems from their specific focus on civil society, the short turnaround time between proposal and implementation, and the willingness to take risks and fund projects that other donors might shy away from.
More generally, both centralised and decentralised projects have at times acted as incubators for activities that have subsequently been scaled up by larger donors.
Conclusions on effectiveness
Almost all projects achieved their planned outputs and contributed to the anticipated short-term outcomes. However, most projects fall short in demonstrating how outputs contribute to outcomes. This makes it difficult to assess the progress or the effectiveness of centralised and decentralised projects. In addition, the lack of monitoring, evaluation and learning frameworks for decentralised projects and occasionally centralised projects, means that the discussion of changes or impacts that may have occurred as a result of the outputs is missing. As a result, the instruments operate without guidance or strategic direction.
While the centralised instrument facilitates the transfer of Dutch skills through training and capacity development, the decentralised instrument does not transfer ‘Dutch added value’. Rather, the decentralised instrument transfers ownership of the projects – and, indirectly, of the supposed societal transformation process to which Matra contributes – to local civil society organisations. Given the success at the project level of the decentralised instrument, including Dutch expertise in Matra projects is not always required to obtain results. Considering finding 3.3. about the scarcity of human resources in Dutch (semi)-governmental organisations, this warrants a discussion about the necessity of including their expertise in projects.
The rationale behind the regional approach is to increase cooperation and learning between countries. Regional Matra projects appear to lack direct added value and are sometimes perceived as contrived. There is a lack of genuine cross-border initiatives and an over-reliance on the role of regional conferences in promoting cooperation between countries. It remains unclear how and to what extent the limited number of initiatives intended to create regional added value actually do so. To further complicate matters, the grouping of countries for a project does not always appear to be productive. A Matra project does not need a regional approach to facilitate the exchange of information between programme countries.
Attributing macro-level changes in the societal transformation process to the Matra programme is not possible due to the discrepancy between the overarching objective at the Matra programme level and the objectives (and outputs) set at the project level, leading to a misalignment between the programme’s objectives on paper and the projects’ effectiveness in achieving these objectives in practice. In the absence of a clear logical framework to complete the missing links between projects and the overall programme, the contribution of projects to higher level objectives cannot be determined.
Dutch implementing organisations of centralised projects rather than Dutch (semi)governmental institutions tend to establish and maintain relationships with the (semi)governmental institutions in programme countries, which raises the question of ownership of the relationship. This can be explained by the limited organisational capacity that (semi)governmental institutions can bring to Matra projects. Moreover, the added value for these organisations is sometimes unclear, which is problematic at a time of staff shortages.
Even though the strengthening of bilateral relations is not explicitly defined in terms of practical implementation and assessment criteria, several positive effects have been observed that can be classified under this objective. First, Matra makes a positive contribution to the visibility and reputation of the Netherlands in the programme countries. Second, Matra supports embassies in establishing strong diplomatic networks in the areas of rule of law and good governance in the programme countries. Third, the Matra programme strengthens the Netherlands’ information position in the programme countries. However, these achievements do not fully reflect the potential outcomes of the programme, had it clearly defined its criteria for successfully strengthening bilateral relations.
Image: © Shutterstock / Oscar Gonzalez Fuentes
Recommendations
IOB presents a number of recommendations linked to the conclusions presented above.
- Allow for sufficient scope for the political realities in which Matra operates, while maintaining a sustained focus on programmatic activities.
- Define more clearly the level of ambition of the Matra programme, including through a Matra-wide intervention logic, while maintaining the overall flexibility of the programme and its ability to adapt to political interests.
- Establish a Matra-wide intervention logic that leaves sufficient scope for the political realities described in finding 1, while providing a framework for strategic thinking and course correction of the programme
- Link the different instruments through information sharing.
- Identify opportunities where projects from one instrument could reinforce activities from another and stimulate cross-instrument learning.
- Improve support to programme countries in translating larger reform ambitions into a specific needs agenda that links to their accession or association aspirations, such as the Copenhagen criteria or other international standards.
Regarding the centralised instrument:
- Involve Dutch embassies in the thematic choices of the centralised instrument (as included in the subsidy frameworks).
- Include a more diverse group of implementers for Matra centralised projects.
- Ensure sufficient local ownership of Matra projects and allow local authorities to define priorities and programming themselves as much as possible.
- Involve local implementing partners in design and implementation.
- Increase engagement with local authorities prior to project design and ensure that there is sufficient buy-in and available institutional capacity to carry out the proposed project.
Regarding the decentralised instrument:
- Facilitate and ensure access to the decentralised instrument for smaller civil society organisations.
- In order to increase relevance and effectiveness (follow-up, upscaling, synergies, learning), the different Matra instruments should be better aligned. This could also promote mutual learning and the exchange of best practices between the two instruments.
- Consider organising an annual (online) conference where best practices and mutual learning opportunities can be shared and explored.
- Arrange regular touch-base meetings between the RVO/DEU, embassy staff and Dutch implementers.
- Encourage embassies to regularly share overviews of decentralised projects with colleagues in The Hague, including policy officers at the ministry and the RVO. The RVO and Dutch implementing organisations should also keep the embassy informed of developments in centralised projects.
- Ensure that the differences between the various funds are clear to embassies and the implementing organisations, while making clear when combining funds is allowed and even encouraged.
- Leave room for flexibility and input from implementers in Matra, as this is seen as a major advantage of the programme compared to other Dutch funds.
- Provide sufficient resources for programme management and implementation and establish mechanisms for strategic and political guidance.
- Before starting a project, consider the specific niches and potential incubator functions that Matra projects, both at the centralised and decentralised level, could fulfil.
With regard to Matra’s contribution to societal transformation (objective I):
Project level:
- Instruct project implementers to demonstrate a clear link between their projects (and outputs) and the overarching objectives of the Matra programme through a logical framework.
- Instruct project implementers to provide a monitoring, evaluation and learning (MEL) framework system that connects to a broader ToC for the Matra programme.
- Clarify for each instrument what ‘Dutch added value’ is and how it should contribute to societal transformation.
- Identify areas where ‘Dutch added value’ is needed and consider its (optional) use across instruments.
- Regional centralised and decentralised projects should be initiated by the programme countries themselves, once they see the added value of a regional approach.
- Articulate the added value of a regional approach for each target country involved.
- Encourage mutual learning regardless of the regional focus of a project. Mutual learning should not be confined to regional projects or to the implementation of identical projects in different countries. Instead, it involves learning from both the differences and similarities between programme countries, while sharing best practices.
Programme level:
- Specify and explain the practical implementation and assessment criteria for the Matra objectives.
- Formulate clear country-specific objectives to which all stakeholders should contribute, and move away from the scattered approach by undertaking a multitude of different initiatives simultaneously.
With regard to the strengthening of bilateral relations (objective II):
- Steer more proactively on the achievement of this objective for the centralised instrument. In coordination with the Ministry of Justice and Security, conduct a scoping exercise on the available capacities and thematic and regional priorities of the involved(semi)governmental organisations before committing their roles in centralised Matra projects.
- Instruct project implementers to clearly demonstrate the availability and benefits of involving Dutch semi-governmental institutions in the project inception phase.
- Translate the objective of strengthening of bilateral relations between the Netherlands and the Matra programme countries into country-specific, more tangible and realistic goals that are clear to all stakeholders involved.