Appendix 1: Promising practices and models from other countries
Our proposal for investment instructions is based in part on models developed in other countries. USAID’s Development Innovation Venture (DIV) has a model based on systematic testing of pilot projects through three steps: (1) pilot, (2) testing and impact evaluation, and (3) scaling to new populations and contexts.
It is particularly relevant to the framework we are introducing here that the DIV model is based on a method for assessing the return on an investment. Here, DIV uses the concept of ‘social rate of return’ (SROR), which is designed to assess the return on a development investment.146 DIV translates social benefit into monetary value to provide a common unit (for an alternative, see the box on Disability-Adjusted Life Years (DALY) below) across topics and sectors. The method is intended to provide a more complete and comprehensive assessment of the return on social investments than traditional economic methods, such as economic return (present value), as it also includes, in addition to the economic effects of investments, social and environmental effects (direct and indirect effects).
Not all projects provide good data or a basis for estimating results. Instead, DIV uses a portfolio approach where they assume a large bias in goal attainment and therefore use the best available knowledge to arrive at a lower estimate of return for the entire portfolio. For projects that have not been or cannot/will not be evaluated for returns, existing knowledge is used.147 We believe methods such as this should be used as part of the investment instructions for Norwegian development policy.
What is important here, more generally, is the use of benchmarks to be able to explicitly measure programmes and initiatives against each other and prioritise between them (see box below). Explicit use of benchmarks is uncommon in Norwegian development policy. The investment instructions we are proposing will require extensive use of this methodology.
Greater use of methods to calculate the return of portfolios is an important means of ensuring that we are doing the right things, but it does not provide sufficient information about whether we are doing things in the right way. A crucial next step is therefore to ensure that the implementation of programmes and measures, portfolios, is carried out in a way that allows appropriate experimentation, adjustment and learning along the way in order to optimise and maximise the effect.148
A particularly promising model called ‘trial and adopt ‘has been developed by the World Bank’s DIME. This simple model entails DIME working closely with partners to implement measures in a way that allows step-by-step experimentation along the way. Here, the World Bank uses parallel arm-controlled experiments, which are largely reminiscent of a ‘stepped wedge’ design. This approach is increasingly used in global health as a standard method to ensure effectiveness.
The World Bank has tested the approach on a wide range of projects and can demonstrate significant effects. In a project aimed at improving the sewage and drainage system in Indonesia, they used experimentation and scaling to achieve a 30 % decrease in diarrhoea and prevented the loss of nearly 20,000 DALY, without increasing the budget. Furthermore, a project in India that was intended to contribute to climate adaptation by funding rain insurance proved to hardly have any effect at all and was discontinued while still in the testing phase. Findings from across the projects where DIME has tested the method indicate that by making a moderate investment in knowledge in the programme, at around 1 % of the project costs, the effect can be increased by an average of as much as 50 %, within the same budget. These models should thus be considered in order to increase the impact of both Norway’s and the development banks’ investments.
Boks 8.1 ‘Benchmarking’, ‘best in class’ and scoringA decidedly comprehensive approach to development has often prevented more widespread use of benchmarks, which are a standard against which to judge if assistance is well spent. The use of benchmarks is an issue that is frequently raised (including by major donors such as the USA), particularly in light of research on cash transfers showing that this method has an impact on far more goals than just poverty reduction. However, it is difficult to set a single standard against which all efforts towards the SDGs can be measured, despite individual initiatives and measures, such as cash transfers, appearing to have positive effects across goals and in the short and medium term. The complexity of the sustainability agenda means that we should rather adopt a ‘best in class’ approach to effectiveness. When it comes to information about past results, we currently have little in the way of systematic approaches, quality assurance or comparisons. This does not only apply to Norway. There is no international database of readily available information about the evaluation of impacts, costs and results of a myriad initiatives in international development cooperation. One feasible way of improving the overview and learning opportunities is the use of scoring. An immediate summary of the assessment can be obtained by setting a score for each assessment of each aid initiative. It is often a matter of finding one or more common denominators – properties or characteristics of the results that make it easier to compare them. It is crucial that the assessments are available in order to understand why the respective scores have been given. The score is necessary for aggregation purposes and provides a quick overview and an indication of progress for an entire portfolio. It corresponds to the requirement in the Regulations on Financial Management in Central Government to assess the degree of achievement of objectives. A number of objections may be raised, but the methodology provides clearer guidelines, frameworks and a systematic approach to assessments that are in any case based on the discretionary judgement of the executive officer. There is already a methodology for this, but it is has not been put into use (assessments and scoring of results are only available for 151 projects out of the 4,104 aid agreements between 2019 and 2021). The different countries and contexts involved also complicate using a benchmark or ‘universal score’. One option would be to take a selection of SDGs and their indicators and consider these in light of each country’s progress. This would, at the very least, measure a country’s progress against its own adopted benchmarks. |
Boks 8.2 Disability-adjusted life years and economic rate of returnDisability-Adjusted Life Years (DALY) is a unit of measurement used to evaluate the burden of disease in a population. DALY was developed by the World Health Organization (WHO) as a way of quantifying both mortality and disability resulting from disease. It is calculated by combining two main components: premature mortality and years of healthy life lost due to disability. DALY is thus a measure of the loss of health status and life expectancy in a population and can be used to compare the burden of various diseases and health problems. It is a useful measuring device to help decision-makers prioritise health resources and set health improvement targets. By estimating the total burden of disease in a population, DALY can be used to identify the biggest health challenges and to prioritise effective measures to reduce the burden of disease. DALY can also be used to evaluate the effects of health interventions and treatments by comparing the figures before and after a particular intervention. DALY has a number of limitations, however. One example is the difficulty in comparing DALYs between different countries and regions due to differences in health systems and data quality. DALY is nonetheless a valuable unit of measurement that can help inform decision-making and prioritise resources to reduce the burden of disease and improve the health of the population. Economic Rate of Return The American Millennium Challenge Corporation (MCC) provides assistance to developing countries to promote economic growth and reduce poverty. MCC differs from other bilateral donors in its very explicit use of Economic Rate of Return (ERR) in project assessment and decision-making. ERR is a tool for measuring the return on an investment relative to the total cost. It takes into account both income and expenses over a certain period of time and calculates a percentage that indicates the return an investment will generate for each USD invested. MCC expects an ERR of at least 10 %to be willing to invest. MCC also considers other factors such as environmental impact and social impact but ERR is an important factor in decision-making. ERR makes it possible to quantify the expected return on an investment and provides decision-makers with a clear and objective way of comparing different investment options. This helps to ensure that MCC invests in projects with the greatest potential to contribute to economic growth and poverty reduction in the recipient countries. MCC also uses ERR to evaluate the success of investments after completion. This provides a way of measuring the effectiveness of investments and adjusting future investment decisions in accordance with the results. |