Agricultural Systems 152 (2017) 145–153

Contents lists available at ScienceDirect

Agricultural Systems

journal homepage: www.elsevier.com/locate/agsy

Perspectives

Responding to global change: A theory of change approach to making

agricultural research for development outcome-based
PK Thornton a,⁎, T Schuetz a, W Förch a,1, L Cramer a,b, D Abreu b, S Vermeulen c, BM Campbell b,c
a
CCAFS, International Livestock Research Institute (ILRI), PO Box 30709, Nairobi 00100, Kenya
b
International Centre for Tropical Agriculture (CIAT), AA6713 Cali, Colombia
c
CCAFS Coordinating Unit, University of Copenhagen, Faculty of Science, Department of Plant and Environmental Sciences, Rolighedsvej 21, DK-1958 Frederiksberg C, Denmark

a r t i c l e i n f o a b s t r a c t

Article history: Agricultural research for development has made important contributions to poverty reduction and food security
Received 11 August 2016 over the last 40 years. Nevertheless, it is likely that both the speed of global change and its impacts on natural and
Received in revised form 27 December 2016 socio-economic systems are being under-estimated. Coupled with the moral imperative to justify the use of pub-
Accepted 10 January 2017
lic resources for which there are multiple, competing claims, research for development needs to become more
Available online 22 January 2017
effective and efficient in terms of contributing towards longer-term development goals. Currently there is consid-
Keywords:
erable debate about the ways in which this may be achieved. Here we describe an approach based on theory of
Agricultural research for development change. This includes a monitoring, evaluation and learning system that combines indicators of progress in re-
Theory of change search along with indicators of change aimed at understanding the factors that enable or inhibit the behavioural
Impact pathway changes that can bring about development impacts. Theory of change represents our best understanding of how
Outcome engagement and learning can enable change as well as how progress towards outcomes might be measured. We
Adaptive management describe the application of this approach and highlight some key lessons learned. Although robust evidence is
currently lacking, a theory of change approach appears to have considerable potential to achieve impacts that bal-
ance the drive to generate new knowledge in agricultural research with the priorities and urgency of the users
and beneficiaries of research results, helping to bridge the gap between knowledge generation and development
outcomes.
© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction with rapidly rising demand (Alexandratos and Bruinsma, 2012). At the
same time, climate change is already affecting agriculture in many de-
The last 25 years have seen substantial improvements in human veloping countries, and the effects will become increasingly challenging
wellbeing. Between 1990–92 and 2012–14, there was a 42% reduction in the future (Thornton et al., 2014a).
in the prevalence of undernourished people in developing regions Several approaches are being used to address poverty, and in devel-
(FAO, 2015). Considerable regional differences exist in the progress oping countries agricultural development is one. The role of agriculture
that has been made against poverty and hunger in the time span, how- in reducing poverty is relatively well studied; enhancing agriculture is
ever: in South Asia progress has been limited, and in sub-Saharan Africa often seen as a critical entry-point in designing effective poverty reduc-
the situation regarding poverty and hunger has become worse (FAO, tion strategies (Christiaensen et al., 2006; Alston, 2010), with agricul-
2015). There were still 805 million people who were chronically under- tural research for development (AR4D) a key mechanism. The
nourished in 2012–2014 (FAO, 2015), almost all in developing coun- adoption of improved agricultural practices, technologies and policies,
tries. Clearly, there is much to be done to reach the targets for 2030 as such as high-yielding rice and wheat varieties, fertilizers, pesticides, ir-
articulated in the Sustainable Development Goals (UN, 2015), particu- rigation and enabling policies, has had strong and positive impacts rel-
larly Goal 2 on ending hunger, achieving food security and improved ative to research investment (Renkow and Byerlee, 2010; Raitzer and
nutrition and promoting sustainable agriculture. With an expected Kelley, 2008). Nevertheless, the world food system continues to face
extra 2–3 billion people to feed over the next 40 years, this will require challenges of persistent food insecurity and rural poverty in places.
targeted efforts to achieve making 70% more food available to keep up The adoption of improved agricultural technologies and practices by
farmers has often been less than expected, despite demonstrated bene-
fits. There are many contributing factors, including inherent limitations
⁎ Corresponding author.
E-mail address: [email protected] (P.K. Thornton).
of supply-led approaches to development and dissemination, limited at-
1
Current affiliation: Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) tention to context-specificity and to farmers' priorities beyond in-
GmbH, Private Bag X12 (Village), Gaborone, Botswana. creased agricultural productivity, and lack of appreciation of the socio-

http://dx.doi.org/10.1016/j.agsy.2017.01.005
0308-521X/© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
146 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153

economic, political and institutional contexts within which smallholder research next users such as development practitioners, extension ser-
farmers operate (Orr, 2012). A technology or intervention may need to vices, farmers and policy makers. These outcomes in turn lead to impact,
be much more than “scientifically proven” if it is to be adopted; good so- such as increased food security or reduced poverty. Fig. 1 is no more
cial management and appropriate implementation processes are likely than a caricature of these processes, but it illustrates that while research
to be needed as well (Pachico and Fujisaka, 2004; Hartmann and Linn, focuses mostly on producing research outputs and development on pro-
2008). In addition, the rate of change in many socio-economic and ducing outcomes and impact, AR4D is an attempt to bridge the two.
earth system trends appears to be accelerating (Steffen et al., 2015), Some of the key characteristics of agricultural research, development
perhaps to the point where the past is no longer a good indicator of and AR4D are listed in Table 1. The boundaries of these realms are nec-
the future. Considerable behavioural shifts will be needed on the part essarily fuzzy, and the characteristics related to evaluation and
of all stakeholders if food security is to be achieved for the more than timeframes in particular are somewhat idealised and may not reflect
9 billion people on the planet by 2050. current practice in use-orientated research (Nowotny et al., 2003). Nev-
AR4D has huge challenges ahead, and ways are needed to do it more ertheless, the distinctions are important; the aim of AR4D is not to take
effectively and efficiently. Here we outline one approach to AR4D that over the work of development agencies but to ensure that the outputs of
may have some potential for addressing issues of effectiveness and effi- research maintain their integrity and are appropriately contextualized
ciency – an approach based on theory of change and impact pathway (translated, communicated, and disseminated). Working in this way at
thinking. This approach is illustrated with reference to the CGIAR Re- the boundaries of science, knowledge and action means that different
search Program on Climate Change, Agriculture and Food Security kinds of partnership are needed if AR4D is to be effective (Clark et al.,
(CCAFS), a global partnership that unites organisations engaged in re- 2011). AR4D has to tread a careful line between the “R” and the “D”.
search and capacity development for a food secure future. This is On the one side, research is a risky business, its results uncertain, and
among the first examples of a large AR4D program being orientated its application sometimes very far from obvious (for instance, the devel-
this way. Although we are not yet at the stage of being able to carry opment of quantum physics and computers in the early and mid-
out a robust evaluation of CCAFS with respect to the effectiveness of a twentieth century, respectively – daily life now is unimaginable without
theory of change approach, its implementation to date has generated them). On the other side, the nature of development is very different to
important lessons that we believe can enhance its effectiveness at that of research, involving different aims, skills, partners, and time
scale. In the next section, we provide some background on theory of frames.
change. In Section 3 we discuss progress so far in implementing the ap- The different framings of agricultural research in a development
proach in CCAFS, focusing on program design and systems for planning context have been driven largely by development agencies and funding
and reporting. We conclude with a discussion of some of the lessons agencies. Such organisations often face common challenges: how to
learnt regarding institutional change, monitoring and evaluation, and strengthen their accountability for the use of public resources, how to
behavioural change. deal with analytical issues of attributing impacts and aggregating re-
sults, how to establish effective performance measurement systems,
2. Background how to ensure a distinct yet complementary role for evaluation, and
how to establish organisational incentives and processes to stimulate
AR4D can be thought of as a set of applied research approaches that the use of performance information in management decision-making
aim to contribute directly to the achievement of international develop- (Binnendijk, 2000). Often, such organisations have been instrumental
ment targets such as the Sustainable Development Goals (UN, 2015) in implementing new or modified approaches to AR4D.
through growth of and innovation in the agricultural sector. This Canada's International Development Research Centre (IDRC) made
broad definition allows for a wide understanding of the concept. In early efforts to articulate how AR4D could contribute to desired behav-
what follows, we assume that the research element of AR4D is carried ioural changes or outcomes (Earl et al., 2001). This articulation revolves
out with broader development outcomes in mind, and that this involves around defining in some detail, during project planning, how the project
demand-led prioritization of research, participatory and action re- team envisages the logical chain of Fig. 1 to unfold in practice. The
search, and stakeholder involvement and capacity development resulting theory of change represents the team's best understanding
(Harrington and Fisher, 2014). or hypothesis, at that point in time, of how engagement and other ap-
Over the last 40 years, agricultural research has undergone several proaches can bridge the gap between research outputs and outcomes
different “framings” regarding the role of research and its effect upon in development. There is no single definition of a theory of change and
the world, but current ideas generally crystallise around a logical se- no set methodology; rather, the approach allows flexibility according
quence of events as shown in Fig. 1, though recognising that this is to the needs of the user or implementer (Vogel, 2012). A theory of
never a linear process. Resources are utilised in a set of research activi- change provides a detailed narrative description of an impact pathway
ties, which produce research outputs that are then used. The use of (the logical causal chain from input to impact as shown in Fig. 1) and
these outputs contributes to behavioural changes, manifested in chang- how changes are anticipated to happen, based on assumptions made
es in knowledge, attitudes, skills and practices of a wide set of non- by the people who are undertaking the work. (While theories of change

Fig. 1. A logical causal chain from research inputs to impact, and the domains of research, development, and agricultural research for development (AR4D). This is highly simplified from
what may be a complex, iterative process.
 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153 147

Table 1
Comparison between agricultural research, agricultural research for development (AR4D) and international development.
(Adapted from Schuetz et al., 2016.)

Characteristic Research AR4D International Development

Organisation of Research centres with a key scientific focus Interdisciplinary research programs built around a NGOs, development aid agencies, UN
activity development challenge and partnership approach agencies
Mandate and Outputs Outcomes Impacts
performance focus
Mechanism for Provision of solid science knowledge and Strong partnerships incorporated within the research Implementation
achieving impact technologies program
Type of One-directional communication Communications for development (based on research), Communications for development,
communication, engagement as part of the research process engagement
knowledge
management
Type of partners International, regional, national research International and national research partners, and Local/district implementing
partners development agencies agencies, central/national
governments
Program evaluation Traditionally focused on quantitative measures: Learning-based approaches dealing with contribution rather Focused on traditional impact
number of publications, quality of journals, than attribution, balancing quantitative with qualitative assessment and quantitative
number of citations measures, assessment of outcomes achieved measurements building on baselines
Timeframe for Traditionally, outcomes/impact often not Achieving outcomes at scale within 5 years and impact within Long-term impacts at large scale in
achieving considered; much research may never achieve 15 years 10–20 years, at local scale in a few
outcomes/impact these months
Languages of International standards Both international and locally appropriate languages Both international and locally
products appropriate languages

may describe a range of different processes and mechanisms, those (Douthwaite et al., 2003; Jordan and Warner, 2010; Klerkx et al.,
based on the direct use of research outputs (as in Fig. 1) are likely to 2010). This is the main reason why approaches based on theory of
be appropriate for many A4RD programs and projects; such “use-orient- change hold out such promise, even if robust evidence for their effec-
ed” theories of change are the subject of the work reported here.) Vogel tiveness in delivering desired outcomes and impact is still in the process
(2012) traces theory of change to the development of program theory of being generated – and not just in the agricultural development arena
approaches in the 1960s. These approaches, built on theoretical under- (Jackson, 2013). Theory of change approaches are no panacea: Valters
pinnings, revolve around clear articulation of the linkages between in- (2014) notes that while such approaches can be used to communicate,
puts and outcomes, and the assumptions that accompany these to learn and to be held accountable, these will often exist with some ten-
linkages, with the aim of improving evaluation and program perfor- sion between them, particularly the accountability element. Neverthe-
mance (Funnell and Rogers, 2011). Theories of change can thus provide less, theory of change approaches can help facilitate the broad
a means to make explicit the implicit, often elusive, hypotheses on the commitment to learning from individuals and organisations that is in-
processes that bridge the gaps between research design, outputs, use, creasingly being seen as an essential element of sustainable develop-
and outcomes (Fig. 1). ment (Pahl-Wostl, 2009; Vogel, 2012; Kristjanson et al., 2014; Valters,
Logical Framework Analysis (LFA) originated at USAID in the early 2015).
1970s before spreading to UNDP and beyond (Bell, 2000). LFA has Approaches based on theory of change have several implications for
been widely used over the last 30 years for project planning; indeed, it implementation in practice. These include the necessity for qualitative
has often been an integral part of applications for funding. LFA has as well as quantitative monitoring of the performance of research pro-
been widely criticised for being overly prescriptive: it adheres to a rela- jects (Springer-Heinze et al., 2003; Young and Mendizabal, 2009), the
tively rigid framework, with a hierarchy of objectives converging on a need to formalise a project's theory of change by involving a wide
single goal, a set of measurable and time-bound indicators of achieve- range of people in its design (Chen, 2005), and regularly examining
ment, checkable sources of information, and assumptions of other im- the assumptions associated with it and adjusting program management
pinging factors (Gasper, 2000). A substantial literature exists on the accordingly (Douthwaite et al., 2013). A major challenge is how to eval-
advantages and weaknesses of LFA (e.g., Bakewell and Garbutt, 2005; uate projects appropriately. In the last 15 years, many development
Vogel, 2012). Despite the fact that LFA has been a mainstay of project agencies and funding agencies have developed evaluation systems
planning for several decades, some questions have been raised as to that are based on metrics of performance: so-called results-based man-
its overall suitability as an approach for ensuring the use of research re- agement (RBM), by which is meant a life-cycle approach to manage-
sults and their translation into outcomes (Crawford and Bryce, 2003). ment in which actors seek to ensure that their actions contribute to
Perhaps its major failings are that LFA does not pay enough attention the achievement of desired results by iteratively using actual results to
to involving key stakeholders and their networks to achieve impact, inform future actions (Mayne, 2007a, 2007b; Bester, 2012). RBM can
providing managers with information to learn and report to funding build on the same logical causal chain (Fig. 1) and can force more explic-
agencies, and establishing a research framework to examine the change it thinking about output use: strategies that directly engage next-users
processes that projects seek to initiate (Douthwaite et al., 2008; Vogel, in the research process, for instance through stakeholder platforms
2012). and user-oriented communication products. There is considerable de-
Theory of change approaches can help to address these drawbacks of bate as to whether RBM leads to efficiency and effectiveness gains in
LFA, because of their explicit focus on all the key participants in the pro- AR4D compared with other evaluation mechanisms. We touch on
cess of AR4D, and because the research framework and hypotheses that these issues below, although here our focus is primarily on the theory
are developed in a particular project lend themselves readily to moni- of change approach itself, rather than on RBM as a method of evaluation.
toring and evaluation, and to being changed or reformulated if found Before that, Section 3 describes the design and implementation of a par-
wanting. Perhaps more fundamentally, approaches based on theory of ticular research management approach to AR4D based on theory of
change can address the well-documented need for learning as a (per- change, in relation to the processes themselves; learnings from these
haps the) critical element of innovation in complex systems processes are discussed in Section 4.
148 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153

3. Implementing a theory-of-change based approach to AR4D: CCAFS representatives, several CCAFS scientists participated, so that in-house
as a case study capacity was built. The training, in combination with theory of change
facilitation guides (Jost et al., 2014; Schuetz et al., 2014a) and learning
The CGIAR Research Program on Climate Change, Agriculture and notes (listed in Annex 1) helped highlight the opportunities and con-
Food Security (CCAFS) is made up of the 15 international agricultural re- straints of rolling out a theory of change approach to an entire AR4D
search centres of CGIAR, integrating thematic work across multiple program. An online community of practice and wiki were established,
global, regional and local partners. The goal of CCAFS is to overcome so that experiences could be documented and shared during the pilot
the additional threats posed by a changing climate to achieving food se- phase.
curity, enhancing livelihoods and improving environmental manage-
ment. The program works to identify and test pro-poor adaptation
3.2. Insights from researchers and partners during the pilot phase
and mitigation practices, technologies and policies for food systems,
adaptive capacity and rural livelihoods, and to provide diagnosis and
CCAFS's approach to theory of change is centred on adaptive man-
analysis that will ensure cost-effective investments, the inclusion of ag-
agement, regular communications between program and projects, and
riculture in climate change policies, and the inclusion of climate issues
facilitated learning within and between projects. Besides periodic virtu-
in agricultural policies, from the sub-national to the global level in
al meetings, project participants in the pilot phase were surveyed for a
ways that bring benefits to the rural poor (Vermeulen et al., 2012;
more in-depth and standardised reflection, and for capturing lessons
Förch et al., 2014).
and achievements from their experiences. These lessons, from both a
CCAFS started in 2010 with research activities clustered into six
project and programmatic perspective, were documented in reports
(later four) thematic areas, working in three regions: East Africa, West
and a series of learning notes (see Annex 1). The approach to developing
Africa, and South Asia. Initially, CCAFS was required to use LFA to plan
theories of change was simplified over time, mostly by reducing the
and monitor project activities across its portfolio. In 2013, the scope of
type and number of indicators and the level of complexity so that the
CCAFS expanded to include two new target regions, Southeast Asia
wider group of people who were expected to work with them would
and Latin America. At the same time, opportunities arose because of
continue to buy into the approach. This simplification led to a heavily re-
changes in CGIAR to implement and test a theory of change approach,
vised version of the training and facilitation guide (version 1: Jost et al.,
with a greatly-expanded set of partners and longer-term objectives.
2014; version 2: Schuetz et al., 2014a).
This was done as a pilot, involving six new, multiannual projects that
Many project participants and partners were willing to take on the
were set up via a competitive process (Thornton et al., 2014b); it was
challenge to develop new ways of collaborating and working beyond
envisaged that the approach would be extended to the entire project
delivering outputs. After one year of the pilot phase, several projects
portfolio in subsequent years. At the time, these projects accounted
had made considerable progress, although making fundamental shifts
for approximately 7% of CCAFS' portfolio, in terms of both total budget
in the way of working takes time and (initially at least) additional re-
and the number of projects (US$ 3.7 million in six projects). These
sources, as well as iteration and learning. It also may affect team compo-
new pilot activities were tasked with designing their projects using a
sition. Some projects recognised that additional skills beyond
theory of change approach, which included not just planning research
disciplinary expertise would be required, such as skills in coordination,
outputs but also planning for outcomes.
facilitation, engagement, communications, and participatory and
learning-oriented monitoring and evaluation. Stakeholder buy-in and
3.1. Building capacity and learning within the program for a theory of
a supportive organisational environment were also seen by most pro-
change approach
jects as necessary elements in implementing the approach.
The CCAFS pilot project teams were thrown in at the deep end. Used
to a more traditional LFA, they were tasked with shifting to a theory of 3.3. Rolling out the approach for CCAFS as a whole
change and learning-based approach for planning their projects within
the trial. It was quickly apparent that capacity to plan and implement Opportunities for changing the programmatic approach to project
projects using this new approach had to be built within CCAFS and its planning and implementation emerged towards the end of 2014, largely
partners. in response to a desire to refine the project portfolio. Impact pathways
Using theory of change approaches within AR4D requires the and theories of change were developed for all four thematic areas and
strengthening of scientists' capacities to do research differently, work for the five regional programs in CCAFS, as one step in refining the pro-
across research disciplines, and work with non-research partners for gram portfolio. The iterative development of the theory of change and
impact, and that institutions facilitate such shifts. Several authors high- impact pathways for all 90 projects in the portfolio took a considerable
light the importance of building capacity for institutional learning (Hall amount of effort. Initial meetings were held virtually, for the most part,
et al., 2003; Horton and Mackay, 2003; Eade, 1997; Springer-Heinze building on a considerable amount of previous engagement and region-
et al., 2003). Johnson et al. (2003) show that participation of non- al priority-setting. The process was completed in five regional face-to
research stakeholders early on in the research process is important, as face meetings with key next-users and stakeholders actively participat-
it can inform institutional learning in research organisations to change ing. These regional workshops were of three-to-four days' duration
priorities and practices. It can also enhance the relevance of agricultural each, with approximately 50–60 scientists and partners in each work-
technologies and the capacity of these stakeholders to design their own shop and a total cost of approximately US$500,000 (excluding staff
action research processes. Horton and Mackay (2003) outline the links time). In these workshops, as noted above, the theory of change devel-
between monitoring and evaluation (M&E), learning and institutional opment and facilitation process, along with guidance documentation,
change and highlight the importance of institutional learning as a were revised to make them leaner, more contextualized, less time con-
means to develop the capacities of the organisation and of individual re- suming and easier to implement (Schuetz et al., 2014a). This was impor-
searchers, and empowering non-research partners as key stakeholders tant for maintaining buy-in into the process of using theories of change
in the process. as a management tool across the entire project portfolio (some US$55
CCAFS implemented a one-week training course on using theory of million per annum and 90 projects). The workshops resulted in several
change for project and program planning (Alvarez et al., 2014). Partici- outputs: impact pathways for many of the projects, key partners trained
pants were chosen strategically so that capacity would be available in in theory of change development, and a coherent set of outcome targets,
CGIAR Centres at the time when proposals would need to be developed as well as workshop documentation and learning notes. It was envis-
following theory of change principles. In addition to project aged that the outputs would help to facilitate changes in people's
 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153 149

Fig. 2. Theories of change: three examples at different levels. Acronyms: CCPAP, Climate Change Priorities Action Plan. COP, Conference of the Parties. GACSA, Global Alliance on Climate-
Smart Agriculture. M&E, Monitoring and Evaluation. SBSTA, Subsidiary Body for Scientific and Technological Advice. UNFCCC, United Nations Framework Convention on Climate Change. A.
Global level: UNFCCC and global paradigms and approaches around climate change. B. National level: Cambodia's CCPAP. C. Sub-national level: weather index insurance in Maharashtra
State, India.
150 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153

practice, such as working towards implementing more effective AR4D

and proactively changing organisational norms.
Three examples of project theories of change are shown in Fig. 2. In
Fig. 2A, an approach is shown for bridging between science and policy in
the climate and agriculture space at the global level, to contribute to
outcomes for global policies and investments that in turn achieve posi-
tive impacts on smallholder livelihoods and food security under climate
change. Activities in this theory of change emphasize partnerships, put-
ting a large proportion of effort into understanding and responding to
next-users' needs. The political economy of climate change and food se-
curity is highly complex at the global level, and the theory of change, set
out with a wide range of partners in the project design phase, has
helped to navigate this by encouraging a focus on a small number of re-
search outputs and events each year. This has been facilitated by a small
group of close partners, which often varies from one year to the next.
This global policy engagement work contributed to the inclusion of ag-
riculture and food security in Parties' contribution to the Paris Agree-
ment adopted by the UNFCCC in 2015 at COP21, for example
(Vermeulen, 2016). It may be several more years before there is robust
evidence as to the achievement of impact as a result of this decision, but
as noted above, impact pathways may be very long.
A second example of a theory of change is shown in Fig. 2B, at the na-
tional level. A set of regional scenarios was developed through a partic-
ipatory process for Southeast Asia, and these were quantified with
research partners and used in the formulation of the Climate Change
Priorities Action Plan (CCPAP) of Cambodia's Ministry of Agriculture,
Fig. 3. Elements of monitoring, evaluation, learning and impact assessment.
Forestry and Fisheries. The scenarios were used to test the plan, the (Adapted from Schuetz et al., 2016.)
main purpose of which is to enhance the resilience of the agricultural
sector and farmers' livelihoods. The project's theory of change was de-
veloped with relevant stakeholders at the same time as the scenarios the strategy are prescribed by program governance bodies, including
process was undertaken, to help ensure high relevance and the inclu- the carrying out of baselines, independent impact assessments, and pe-
sion of the partners likely to be necessary for success. This work has riodic external evaluations, for example.
the potential to benefit the entire population in different ways in the Several elements are needed to implement the strategy (Fig. 3). The-
coming years - over 15 million people in Cambodia, 12 million of ories of change and impact pathways have to be harmonized so that all
whom live in rural areas (Vervoort et al., 2015). As for many AR4D activ- projects are contributing to program targets. With appropriate theories
ities, participatory development and engagement processes are key for of change defined, indicators and baselines are needed so that the as-
attaining outcomes. Without stakeholder buy-in from the very begin- sumptions underlying them can be continually tested and projects' con-
ning, research activity outputs may have no traction among partners tributions checked for alignment and plausibility. In its first three years,
and intended next users. CCAFS undertook a set of baseline studies at key sites in all five regions
A third example, at the sub-national level, is shown in Fig. 2C, focus- (Förch et al., 2014), so that behaviour and practice changes of farmers
ing on work in one state of India to improve weather-based index insur- and other decision-makers could be evaluated through time. Not all
ance in several states. By providing technical assistance to insurance the indicators needed for all outcome targets are covered in these base-
companies to improve the indices used by the insurance industry and lines, so projects themselves are responsible for conducting the addi-
also facilitating interaction between state government and the private tional baselines needed to monitor progress over time related to their
sector, the aim is to increase the satisfaction of farmers using crop insur- specific research activities. Several elements around self-reflection are
ance. With better satisfaction rates, more farmers will be encouraged to needed, if adaptive management that provides for flexibility and correc-
make use of insurance products, and it is envisaged that this will in- tive action is to be implemented. Change often happens not as anticipat-
crease their resiliency in the face of climate variability and extremes. ed, and appropriate mechanisms are critical for allowing us to make
As in the other examples above, the theory of change for this work well-documented and well-justified adjustments in response to the in-
was co-developed early in the project cycle in an effort to ensure rele- sights gained through our work. A key component is an ICT-supported
vance and buy-in from all key stakeholders. After three seasons of program and project management processes in the form of an online
A4RD, more than 1 million farmers in Maharashtra state are now platform, building on the online community of practice mentioned in
being reached with improved insurance products to help them cope Section 3.1 above, used by project teams to plan activities, report on
with climate risk in their soybean, rice, cotton and pearl millet crops them, and monitor progress against outcome target indicators. This
(Aggarwal and Shirsath, 2015). platform is accompanied by a “support pack” that provides practical
guidance and tools for quantitative and qualitative monitoring. Project
3.4. Monitoring, evaluation, learning and impact assessment evaluation and synthesis are carried out in the wake of project reporting
to simplify reporting to funding agencies and facilitate learning and
A monitoring, evaluation, learning and impact assessment strategy knowledge brokerage across the program portfolio and beyond. Evalua-
was developed to support the new approach in a comprehensive man- tion criteria include traditional output-focused criteria, as well as
ner (Schuetz et al., 2014b). This strategy was developed to help promote progress towards outcomes, partnership and learning. Incentive mech-
an “evaluative culture” within the program. It includes a conceptual anisms are being introduced, recognising that these do not always have
framework, guided by overall program principles for partnership, en- to relate to budgetary bonuses. A final element encompasses research-
gagement and communications in a modular way, so that the demands able issues that some projects may contribute to, including issues
can be met of the program as a whole, its projects, and the wider re- around institutional change, incentives, and social learning, for
search system within which CCAFS is embedded. Some elements of example.
 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153 151

4. Implications for policy, practice and research 4.2. Learning

Working with theories of change has major implications in several Robust knowledge needs to be generated that can feed into develop-
dimensions. In relation to monitoring, evaluation, learning and impact ment policy and investment decision making, and this in turn requires a
assessment, it implies a shift in focus to contribution rather than attribu- cumulative and broad-based approach in choosing evaluation and im-
tion, to acknowledge the role and inputs of partners and other actors pact assessment methods at different levels (Maredia, 2009). An ap-
both in achieving outcomes and in providing evidence for those out- proach based on theory of change supports adaptive management:
comes. Building in triple-loop learning (Kristjanson et al., 2014) can because the former is based on learning processes, it allows mid-
make a major contribution to reflection and to supporting adaptive course corrections to be made (including modifying the assumptions
management, so that project teams can better deal with uncertainty. and hypotheses that originally helped to define the theory of change),
At the same time, not everything can be measured; this highlights the and so facilitates dealing with uncertainty and emerging priorities and
need for narratives that can complement and support quantitative in- opportunities. The ICT process supported through the online platform
formation. Bringing both qualitative and quantitative information to for project planning, reporting and evaluation procedures has proved
bear remains a substantial challenge (Vogel, 2012), and impact assess- to be a good vehicle for learning as well as project management. The
ment methodology needs to evolve considerably to address social pro- platform is far from perfect, but it does foster the inclusive involvement
cesses and outcomes in robust ways; it is not yet up to the task of as wide a range of stakeholders as possible in project planning, imple-
(Befani et al., 2016). mentation and reporting. There are substantial costs involved (either
Project implementation in CCAFS aspires to a “three-thirds prin- using commercial products or developing something from scratch
ciple” in relation to engagement effort: a third working with next- over many months), but in CCAFS's case, development and maintenance
users to build relationships and define their needs from research, a costs are being shared among several programs.
third on the research itself, and a third on enhancing next-users'
capacity to improve the uptake of research outputs (Fullana i 4.3. Effectiveness
Palmer et al., 2011; Vermeulen and Campbell, 2015). This does not
necessarily translate into a third for each of these three elements in The necessity of providing value for money has to be embraced.
relation to financial resources, as engagement processes themselves Many funding agencies now require that grantees demonstrate value
may not be that costly, though they may be time-consuming. As part for money. “The purpose of the VfM [Value for Money] drive is to devel-
of creating a program enabling environment, embracing this three- op a better understanding (and better articulation) of costs and results
thirds principle facilitates investment into solid science, critical part- so that we can make more informed, evidence-based choices” (DFID,
nerships, ownership and buy-in by partners, and capacity enhance- 2011). Some have critiqued the whole notion of payment by results as
ment at all levels both internally and externally. Embracing the applied to development and AR4D on the basis that it provides perverse
three-thirds principle also implies different budgeting and funding incentives that actually diminishes cost-effectiveness (for example, see
structures, so that appropriate levels of resources are allocated to ca- Chambers, 2014). As noted above, there is much work still to do on ap-
pacity building, communications and engagement with the wide propriate measurement mechanisms, but this does not diminish the
range of different partners likely to be needed to produce outcomes. need to demonstrate accountability and results. The jury is still out on
These elements need to be budgeted for explicitly within a project the question of whether theory-of-change-based AR4D is more effective
life-cycle, rather than as an after-thought and left to others or in leading to outcomes and impact compared with other approaches;
outsourced. At the same time, there is still much work to be done nevertheless, the conceptual grounding of theory of change in iterative
on how to monitor outcomes effectively, how to evaluate the real learning processes, as noted in Section 2 above (Douthwaite et al.,
share of contribution towards the observed change, and how to as- 2003), provides grounds for optimism. The portfolio of projects within
sess value for money. Similarly, delivery of outcomes, especially at CCAFS is continuing to shift in response to newly-formulated theories
scale, may take time for AR4D programs. Longer funding cycles of change and impact pathways for the various thematic areas, includ-
could be expected to facilitate this considerably, as well as explicit ing the expansion of activities around nutrition scenarios and modelling
recognition that research does not start from scratch, allowing out- to contribute to targets on food and nutrition security under climate
comes from previous investment cycles to be reported on. change, for instance. Similarly, the number of non-research partners
As noted above, the processes described in Section 3 have been doc- that CCAFS interacts with continues to increase, in response to the
umented in reports and learning briefs (see Annex 1), which included need for engagement with next- and end-users of research outputs;
participant surveys and self-reflection activities. On the basis of this in- currently, some 35% of CCAFS's strategic partners are in this category
formation, below we summarise the CCAFS experience in implementing (CCAFS, 2016). Within CGIAR, a similar theory-of-change approach
an AR4D approach based on theory of change in relation to four over- was piloted over a ten-year period (2004–2013) by the Challenge Pro-
arching elements: flexibility, learning, effectiveness, and incentives. gram on Water and Food. An external review of that program placed
their work at the leading edge of global research, and this research re-
sulted in developmental outcomes although limited impact at scale by
4.1. Flexibility the time the review was undertaken (Hall et al., 2014).

The need for flexibility is key. Rigid application of a specific ap- 4.4. Incentives
proach most likely will not work. Early interaction with users led to
a considerable simplification of the CCAFS process (Section 3.2), to Research is often curiosity-driven, and traditional indicators of suc-
arrive at something that was not seen as overly burdensome to en- cess centre on peer-reviewed publications in high-profile academic
gage with. In designing and implementing systems and solutions, journals. In today's highly competitive research environment another
this may well involve abandoning the goal of best practice for a crucial success factor relates to fundraising: the ability to write and
goal of “good enough”. The CCAFS experience involved several false win competitive research proposals. Neither of these motivations for re-
starts and considerable frustration before implementing a lean and search is guaranteed to deliver development outcomes. For that, new
simple model that most users felt they were able to buy into. At the investments of time and effort may be needed to identify and work
same time, a flexible framework is needed to allow aggregation of with non-traditional partners to promote behavioural change in shared
output, outcomes and targets across the different program units theories of change. This suggests the need for AR4D organisations to ex-
(projects, research areas, regions). pand their incentive system to reward different types of excellence in
152 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153

addition to great science, such as strategic team and partnership build- Annex 1 (continued)
ing, engagement, communications and capacity development, for ex- Title Permanent web link
ample. The development of theories of change can lead all
and evaluating Knowledge to Action.
participants, whatever their skills, to give thought to what lies between CCSL Learning Brief No. 9. Copenhagen,
solid science, great interventions, and their positive developmental im- Denmark: CGIAR Research Program on
pact, by allowing teams to monitor, reflect, evaluate and learn. Climate Change, Agriculture and Food
Security (CCAFS)
3 Alvarez S, Jost C, Schuetz T, Förch W, http://hdl.handle.net/10568/52992
5. Conclusions
Schubert C, Kristjanson P. 2014. Lessons
in Theory of Change from the Introduc-
Strong incentives from funding agencies for a move towards tory Training on Theories of Change, Im-
outcome-oriented research programs are having considerable impact pact Pathways and Monitoring &
on the way in which agricultural research is conceived, planned, imple- Evaluation. CCSL Learning Brief No. 10.
Copenhagen, Denmark: CGIAR Research
mented and evaluated. A key requirement for such work is flexibility - Program on Climate Change, Agriculture
the flexibility to adjust so that the outcome orientation works as a sup- and Food Security (CCAFS)
port mechanism and enabler rather than a one-size-fits-all straitjacket 4 Schuetz T, Förch W, Thornton P, http://hdl.handle.net/10568/52990
without space for innovation, serendipity and creativity. The shift to Wollenberg L, Hansen J, Jarvis A, Coffey
K, Bonila-Findji O, Loboguerrero
an AR4D approach based on theory of change is fostering real transition
Rodriguez AM, Martínez Barón D,
in the international agricultural research community, much of it for the Aggarwal P, Sebastian L, Zougmoré R,
better in our view. However, it also comes with considerable challenges. Kinyangi J, Vermeulen S, Radeny M,
Defining the necessary adjustments, and developing new processes and Moussa A, Sajise A, Khatri-Chhetri A,
mechanisms, needs time and resources, which are often grossly Richards M, Jost C, Jay A. 2014. Learning
Brief: Lessons in Theory of Change from a
underestimated and for which planning is often inadequate. Some of
Series of Regional Planning Workshops.
these challenges arise because of the nature of research: the results Learning Brief No 11. Copenhagen,
are not known from the start, in contrast to engineering, for example, Denmark: CGIAR Research Program on
where the results are much less uncertain. A second challenge lies in Climate Change, Agriculture and Food
Security (CCAFS)
striving to balance the need to do great science with the need for impact
5 Schuetz T, Förch W, Schubert C, Thornton http://hdl.handle.net/10568/56629
delivery at scale. A third challenge lies in generating an evidence base P, Cramer L. 2014. Lessons and Insights
that can rigorously address whether and how theory-of-change-based from CCAFS Results-Based Management
approaches lead to efficiency and effectiveness gains in AR4D compared Trial. Learning Brief No 12. CGIAR Re-
with other approaches. An evaluation of the outcomes and impacts of search Program on Climate Change, Ag-
riculture and Food Security (CCAFS).
CCAFS's work is planned for 2020–2021, building in part on revisiting
Copenhagen: Denmark
the baselines carried out in the five target regions in 2011–2013 6 Jost C, Kristjanson P, Ferdous N. 2015. http://hdl.handle.net/10568/61900
(Förch et al., 2014). In general, this kind of evidence generation will al- Lessons in Theory of Change: Gender and
most certainly involve multi- and trans-disciplinary research that mixes Inclusion. Learning Brief No 14.
Copenhagen, Denmark: CGIAR Research
quantitative approaches to measure outcome variables with qualitative
Program on Climate Change, Agriculture
approaches that establish the causal mechanisms involved, however and Food Security (CCAFS).
difficult this may be in relation to social processes and human behaviour 7 Schuetz T, Förch W, Thornton PK 2015. http://hdl.handle.net/10568/67362
(Carr, 2013). Above all, we need to avoid the focus on outcomes being CCAFS reporting and evaluation in a
seen as disadvantageous to science, and development being seen as in results-based management framework.
Learning Brief 15. Copenhagen,
competition with the science. Rather, they need to be seen as comple-
Denmark: CGIAR Research Program on
mentary, enabling, and ultimately liberating. Climate Change, Agriculture and Food
Security (CCAFS)
Acknowledgements 8 Förch W, Schuetz T, Abreu D, Tobon H, http://hdl.handle.net/10568/69497
Thornton P. 2016. Building an online
platform in support of outcome-focused
We thank the many people who have contributed to the work results-based program management.
outlined here, both in its implementation as well as in its design. We CCSL Learning Brief No. 16. Copenhagen,
thank two anonymous referees for constructive comments on an earlier Denmark: CGIAR Research Program on
version of the paper. We acknowledge the CGIAR Fund Council, Climate Change, Agriculture and Food
Security (CCAFS).
Australia (ACIAR), European Union, International Fund for Agricultural
Development (IFAD), New Zealand, Netherlands, Switzerland, UK and
Thailand for funding to the CGIAR Research Program on Climate Change,
Agriculture and Food Security (CCAFS). References

Annex 1 Aggarwal, P.K., Shirsath, P., 2015. Better designed weather-based insurance holds promise
Published reports and learning briefs related to the CCAFS theory of change approach. for Maharashtra farmers. https://ccafs.cgiar.org/research-highlight/better-designed-
weather-based-insurance-holds-promise-maharashtra-farmers.
Title Permanent web link Alexandratos, N., Bruinsma, J., 2012. World agriculture towards 2030/2050: the 2012 re-
vision. ESA Working Paper 12–03. FAO, Rome, Italy.
1 Jost CC, Sebastian L, Kristjanson P, Förch http://hdl.handle.net/10568/42447 Alston, J.M., 2010. The Benefits from Agricultural Research and Development, Innovation,
W 2014. Lessons in theory of change: and Productivity Growth. OECD Food, Agriculture and Fisheries Papers, No. 31. OECD
CCAFS Southeast Asia Research for De- Publishing, Paris DOI. http://dx.doi.org/10.1787/5km91nfsnkwg-en.
velopment Workshop. CCSL Learning Alvarez, S., Jost, C., Schuetz, T., Förch, W., Schubert, C., Kristjanson, P., 2014. Lessons in
Theory of Change from the Introductory Training on Theories of Change, Impact Path-
Brief No. 8. Copenhagen, Denmark:
ways and Monitoring & Evaluation. CCSL Learning Brief No. 10. CGIAR Research Pro-
CGIAR Research Program on Climate gram on Climate Change, Agriculture and Food Security (CCAFS), Copenhagen,
Change, Agriculture and Food Security Denmark. http://hdl.handle.net/10568/52992.
(CCAFS). Bakewell, O., Garbutt, A., 2005. The Use and Abuse of the Logical Framework Approach.
2 Jost CC, Kristjanson P, Vervoort J, Alvarez http://hdl.handle.net/10568/42446 SIDA, Stockholm.
S, Ferdous N, Förch W. 2014. Lessons in Befani, B., D'Errico, S., Booker, F., Giuliani, A., 2016. Clearing the fog: newtools for improv-
theory of change: monitoring, learning ing the credibility of impact claims. IIED Briefing Paper . pubs.iied.org/17359IIED.
html.
 P.K. Thornton et al. / Agricultural Systems 152 (2017) 145–153 153

Bell, S., 2000. Logical frameworks, Aristotle and soft systems: a note on the origins, values Jost, C., Alvarez, S., Schuetz, T., 2014. CCAFS Theory of Change Facilitation Guide. CGIAR
and uses of logical frameworks, in reply to Gasper. Public Adm. Dev. 20, 29–31. Research Program on Climate Change, Agriculture and Food Security (CCAFS), Copen-
Bester, A., 2012. Results-based management in the United Nations development system: hagen, Denmark. http://hdl.handle.net/10568/41674.
progress and challenges. A Report Prepared for the United Nations Department of Klerkx, L., Aarts, N., Leeuwis, C., 2010. Adaptive management in agricultural innovation
Economic and Social Affairs, for the Quadrennial Comprehensive Policy Review systems: the interactions between innovation networks and their environment.
Final Report. Agric. Syst. 102, 390–400.
Binnendijk, A., 2000. Results-based management in the development co-operation agen- Kristjanson, P.M., Harvey, B., Van Epp, M., Thornton, P.K., 2014. Social learning and sus-
cies: a review of experience. DAC Working Party on Aid Evaluation Report. OECD. tainable development. Nat. Clim. Chang. 4, 5–7.
Carr, E.R., 2013. Causality isn't the same as causal mechanisms: why development needs Maredia, M.K., 2009. Improving the Proof - Evolution of and Emerging Trends in Impact
more research into the latter. http://www.edwardrcarr.com/opentheechochamber/ Assessment Methods and Approaches in Agricultural Development. IFPRI Discussion
2013/02/04/causality-isnt-the-same-as-causal-mechanisms-why-development- Paper. IFPRI, Washington DC.
needs-more-research-into-the-latter/. Mayne, J., 2007a. Best Practices in Results-Based Management: A Review of Experience A
CCAFS, 2016. CGIAR Research Program on Climate Change, Agriculture and Food Security, Report for the United Nations Secretariat Volume 1: Main Report.
Full Proposal 2017–2022. Online at. http://www.cgiar.org/our-strategy/second-call- Mayne, J., 2007b. Challenges and lessons in implementing results-based management.
for-cgiar-research-programs/cgiar-research-programs-and-platforms-revised-full- Evaluation 13 (1), 187-109.
proposals-submitted-for-review/. Nowotny, H., Scott, P., Gibbons, M., 2003. “Mode 2” revisited: the new production of
Chambers, R., 2014. Perverse payment by results: frogs in a pot and straitjackets for obsta- knowledge. Minerva 41 (3), 179–194.
cle courses. https://participationpower.wordpress.com/2014/09/03/perverse- Orr, A., 2012. Why were so many social scientists wrong about the green revolution?
payment-by-results-frogs-in-a-pot-and-straitjackets-for-obstacle-courses/. Learning from Bangladesh. J. Dev. Stud. 48 (11), 1565–1586.
Chen, H.T., 2005. Practical Program Evaluation: Assessing and Improving Planning, Imple- Pachico, D., Fujisaka, S. (Eds.), 2004. Scaling Up and Out: Achieving Widespread Impact
mentation, and Effectiveness. Sage Publications, CA. through Agricultural Research 3. CIAT, Cali.
Christiaensen, L.J., Demery, L., Kuhl, J., 2006. The Role of Agriculture in Poverty Reduction: Pahl-Wostl, C., 2009. A conceptual framework for analysing adaptive capacity and multi-
An Empirical Perspective. 4013. World Bank Publications. level learning processes in resource governance regimes. Glob. Environ. Chang. 19
Clark, W.C., Tomich, T.P., van Noordwijk, M., Guston, D., Catacutan, D., Dickson, N.M., (3), 354–365.
McNie, E., 2011. Boundary work for sustainable development: Natural resource man- Raitzer, D., Kelley, T., 2008. Benefit-cost analysis of investment in the international agri-
agement at the Consultative Group on International Agricultural Research (CGIAR). cultural research centers of the CGIAR. Agric. Syst. 96, 108–123.
Proc. Natl. Acad. Sci. 200900231. Renkow, M., Byerlee, D., 2010. The impacts of CGIAR research: a review of recent evi-
Crawford, P., Bryce, P., 2003. Project monitoring and evaluation: a method for enhancing dence. Food Policy 35, 391–402.
the efficiency and effectiveness of aid project implementation. Int. J. Proj. Manag. 21 Schuetz, T., Förch, W., Thornton, P., 2014a. Revised CCAFS Theory of Change Facilitation
(5), 363–373. Guide. CGIAR Research Program on Climate Change, Agriculture and Food Security
Department for International Development (DFID), 2011. DFID's approach to Value for (CCAFS), Copenhagen, Denmark.
Money (VfM). https://www.gov.uk/government/uploads/system/uploads/ Schuetz, T., Förch, W., Thornton, P., 2014b. CCAFS Monitoring and Evaluation Strategy.
attachment_data/file/67479/DFID-approach-value-money.pdf. CGIAR Research Program on Climate Change, Agriculture and Food Security
Douthwaite, B., Alvarez, S., Thiele, G., Mackay, R., 2008. Participatory Impact Pathways (CCAFS), Copenhagen, Denmark. http://hdl.handle.net/10568/41913.
Analysis: A Practical Method for Project Planning and Evaluation. CPWF, Colombo: Schuetz, T., Förch, W., Thornton, P.K., Vasileiou, I., 2016. Pathway to impact: supporting
Sri Lanka. and evaluating enabling environments for research for development. In: Uitto, J.I.,
Douthwaite, B., Kamp, K., Longley, C., Kruijssen, F., Puskur, R., Chiuta, T., Apgar, M., Dugan, Puri, J., van den Berg, R.D. (Eds.), Evaluating Climate Change for Sustainable Develop-
P., 2013. Using Theory of Change to Achieve Impact in AAS. AAS Working Paper. ment. Springer, Dordrecht.
Douthwaite, B., Kuby, T., van de Fliert, E., Schulz, S., 2003. Impact pathway evaluation: an Springer-Heinze, A., Hartwich, F., Henderson, J.S., Horton, D., Minde, I., 2003. Impact path-
approach for achieving and attributing impact in complex systems. Agric. Syst. 78 (2), way analysis: an approach to strengthening the impact orientation of agricultural re-
243–265. search. Agric. Syst. 78 (2), 267–285.
Eade, D., 1997. Capacity-building: An Approach to People-centred Development. Oxfam, Steffen, W., Broadgate, W., Deutsch, L., Gaffney, O., Ludwig, C., 2015. The trajectory of the
Oxford. Anthropocene: the great acceleration. Anthropol. Rev. 2, 81–98.
Earl, S., Carden, F., Smutylo, T., 2001. Outcome Mapping, Building Learning and Reflection Thornton, P.K., Ericksen, P.J., Herrero, M., Challinor, A.J., 2014a. Climate variability and vul-
Into Development Programs. IDRC Publications. nerability to climate change: a review. Glob. Chang. Biol. 20 (11), 3313–3328.
FAO, 2015. Climate Change and Food Systems: Global Assessments and Implications for Thornton, P.K., Schuetz, T., Förch, W., Campbell, B., 2014b. The CCAFS Flagship 4 Trial on
Food Security and Trade. FAO, Rome. www.fao.org/documents/card/en/c/2d309fca- Results-based Management: Progress Report. CGIAR Research Program on Climate
89be-481f-859e-72b27a3ea5dc/. Change, Agriculture and Food Security (CCAFS), Copenhagen, Denmark. http://hdl.
Förch, W., Kristjanson, P., Cramer, L., Barahona, C., Thornton, P., 2014. Back to Baselines: handle.net/10568/52261.
Measuring Change and Sharing Data. 3(13). Agriculture and Food Security online UN, 2015. Transforming Our World: The 2030 Agenda for Sustainable Development.
at. www.agricultureandfoodsecurity.com/content/3/1/13. https://sustainabledevelopment.un.org/post2015/transformingourworld/
Fullana i Palmer, P., Puig, R., Bala, A., Baquero, G., Jordi Riba, J., Raugei, M., 2011. From life publication.
cycle assessment to life cycle management. J. Ind. Ecol. 15, 458–475. Valters, C., 2014. Theories of change in international development: Communication,
Funnell, S., Rogers, P., 2011. Purposeful Programme Theory. Effective Use of Theories of learning or accountability? JSRP Paper 17. The Asia Foundation, San Francisco
Change and Logic Models. Jossey-Bass, San Francisco. Valters, C., 2015. Theories of Change. Time for a Radical Approach to Learning in Develop-
Gasper, D., 2000. Evaluating the ‘logical framework approach’ towards learning-oriented ment. ODI, London.
development evaluation. Public Adm. Dev. 20, 17–28. Vermeulen, S., 2016. Agriculture is not excluded from the post-2015 UNFCCC agreement
Hall, A., Bullock, A., Adolph, B., 2014. Forward-looking review of the CGIAR challenge pro- announced in Paris. https://activities.ccafs.cgiar.org/data/projects/91/caseStudy/
gram on water and food. https://waterandfood.org/. Project91Evidenceforoutcomecase.docx.
Hall, A., Sulaiman, V.R., Clark, N., Yoganand, B., 2003. From measuring impact to learning Vermeulen, S., Campbell, B., 2015. Ten principles for effective AR4D programs. CCAFS Info
institutional lessons: an innovation systems perspective on improving the manage- Note. CGIAR Research Program on Climate Change, Agriculture and Food Security
ment of international agricultural research. Agric. Syst. 78 (2), 213–241. (CCAFS), Copenhagen, Denmark.
Harrington, L.W., Fisher, M.J. (Eds.), 2014. Water Scarcity, Livelihoods and Food Security, Vermeulen, S., Zougmore, R., Wollenberg, E., Thornton, P., Nelson, G., Kristjanson, P.,
Research and Innovations for Development. Routledge, New York. Kinyangi, J., Jarvis, A., Hansen, J., Challinor, A., Campbell, B., 2012. Climate change, ag-
Hartmann, A., Linn, J.F., 2008. Scaling up: a framework and lessons for development effec- riculture and food security: a global partnership to link research and action for low-
tiveness from literature and practice. Wolfensohn Center for Development Working income agricultural producers and consumers. Curr. Opin. Environ. Sustain. 4 (1),
Paper 5. Brookings Institute. 128–133.
Horton, D., Mackay, R., 2003. Using evaluation to enhance institutional learning and Vervoort, J., Peou, R., Veeger, M., 2015. Scenario-guided policy formulation: Cambodia's
change: Recent experiences with agricultural research and development. Agric. Climate Change Priorities Action Plan. In: Westermann, O., Förch, W., Thornton, P.K.
Syst. 78, 127–142. (Eds.), Reaching More Farmers: Innovative Approaches to Scaling Up Climate Smart
Jackson, E.T., 2013. Interrogating the theory of change: evaluating impact investing where Agriculture. CCAFS Working Paper no. 135. Copenhagen, Denmark, pp. 69–76.
it matters most. J. Sustain. Finance Invest. 3 (2), 95–110. Vogel, I., 2012. ‘Theory of Change’ in International Development. Review Report for the
Johnson, N., Lilja, N., Ashby, J.A., 2003. Measuring the impact of user participation in agri- UK Department of International Development.
cultural and natural resource management research. Agric. Syst. 78, 127–142. Young, J., Mendizabal, E., 2009. Helping researchers become policy entrepreneurs. ODI
Jordan, N., Warner, K.D., 2010. Enhancing the multifunctionality of US agriculture. Biosci- Briefing Paper 53 . www.odi.org/sites/odi.org.uk/files/odi-assets/publications-
ence 60 (1), 60–66. opinion-files/1730.pdf.