aes24 International Evaluation Conference

Keynote address: Plenary: Elizabeth Hoffecker 8.30-9.30, followed by plenary panel 9:30-10:00

Lead Research Scientist, Local Innovation Group, Massachusetts Institute of Technology (MIT), USA

What does a wayfinding approach look like when seeking to learn from and evaluate interventions into complex systems? 

Many of the most intractable challenges facing communities around the world are system challenges requiring system-level responses. Development-focused donors and implementers at various levels are recognizing this and funding system-strengthening and systems-change work across a variety of systems. Monitoring, evaluation, and learning work, however, has traditionally been focused at the project level, not the level of the dynamic local systems in which projects operate. A new kind of evaluation is needed for this work and is in the early stages of being developed, tested, and improved through learning-by-doing.

In forums such as the UNDP’s M&E Sandbox and the BMGF-funded Systems Monitoring, Learning, and Evaluation initiative, development donors, implementers, and evaluators are asking questions such as: what evaluation designs and approaches are most suitable for learning from and evaluating system and portfolio-level interventions? And “how do we know if we are making progress, generating results, and contributing to positive change in a complex system?”
Drawing on experience implementing “complexity-aware” evaluations of system-change interventions in Northern India and Guatemala, this session develops and explores responses to these questions. The presentation shares an evaluation approach and six related tools that are being used to evaluate, learn, and implement adaptively in these two very different system contexts. The tools--while humble and likely familiar--can become powerful wayfinding instruments for navigating complexity when combined with a systems-informed evaluation design. This session introduces this approach through a keynote presentation and then further develops it through an interactive panel with systems-informed evaluators working both internationally and domestically in Australia.

Panel: Exploring learning and evaluation tools for complex systems  9.30-10
 
Panel: Elizabeth Hoffeker, Matt Healey, Donna Loveridge, Tony Keissler
Chair: Jess Dart

Elizabeth Hoffecker will be joined by a panel to explore how the learning and evaluation tools she presents in the keynote address are being applied across different contexts.

Authors: Eleanor Williams (Australian Department of Treasury ),Josephine Norman (Victorian Department of Health, AU),Melissa Kaltner (Lumenia, AU),Skye Trudgett (Kowa Collaboration, AU),George Argyrous (Paul Ramsay Foundation, AU),Luke Craven (National Centre for Place-Based Collaboration (Nexus), AU)

Delivering great evaluations requires a strong professional relationship between those commissioning and delivering the evaluation, as well as all relevant stakeholders.

Traditional evaluation commissioning approaches have tended to treat evaluation as a one-off exchange focusing on the completion of pre-defined tasks. However, the evolving landscape of policies and programs tackling complex issues demands a more nuanced and relational approach to get the most out of the journey of evaluation.

This big room panel session brings together speakers who are at the forefront of thinking around collaborative commissioning partnerships from the perspectives of government, not-for-profit and Indigenous-led organisations, and the private sector who can play the full suite of roles on the commissioning journey. The discussion will delve into the experiences of a range of organisations involved in commissioning who are seeking to build enduring relationships, and in some case partnerships, between the commissioners, the evaluators and the stakeholders to whom we are accountable.

Drawing on real-world case studies and empirical evidence, the discussion will highlight the challenges and rewards of transitioning from a transactional model to a relational model. It will explore how this paradigm shift can enhance collaboration and ultimately lead to a range of positive outcomes.

Attendees will be invited to respond to engage in dialogue with the panel to bring the collective wisdom of attendees together and consider how the destination of better commissioning relationships would look, the practical obstacle we face on our pathway, and how we can reach our destination. To facilitate this active discussion, attendees will have the opportunity to use Sli.do throughout the session to provide input on key questions, share experience in real-time and ask questions of the expert panel.

Authors: Phillip Belling (NSW Department of Education), Liam Downing (NSW Department of Education, AU)

This paper is aimed at early career and experienced evaluators interested in realist evaluation, but with concerns about the time a realist approach might take. The authors respond to this concern with an innovative blending of realist and developmental evaluation. Participants will exit the room with a working understanding of realist developmental evaluation, including its potential for adaptive rigour that meets the needs of policy makers and implementers.

Realist evaluation is theoretically and methodologically robust, delivering crucial insights about how, for whom and why interventions do and don't work (House, 1991; Pawson & Tilley, 1997; Pawson, 2006). It aims to help navigate unfamiliar territory towards our destination by bringing assumptions about how and why change happens out in the open.

But even realism's most enthusiastic practitioners admit it takes time to surface and test program theory (Marchal et al., 2012; van Belle, Westhorp & Marchal, 2021). And evaluation commissioners and other stakeholders have understandable concerns about the timeliness of obtaining actionable findings (Blamey & Mackenzie, 2007; Pedersen & Rieper, 2008).

Developmental evaluation (Patton, 1994, 2011 2021; Patton, McKegg, & Wehipeihana, 2015) is more about what happens along the way. It appeals because it provides a set of principles for wayfinding in situations of complexity and innovation. Realist and developmental approaches do differ, but do they share some waypoints to reliably unpack perplexing problems of practice?

This paper documents a journey towards coherence and rigour in an evaluation where developmental and realist approaches complement each other, and deliver an evidence base for program or policy decision-making that is not only robust but also timely.

We show that, in complex environments, with programs involving change and social innovation, realist developmental evaluation can meet the needs of an often-varied cast of stakeholders, and can do so at pace, at scale, and economically.

Authors: Moderator:  Prof Rick Cummings, Emeritus Professor Murdoch University, AES Fellow
Key Speaker: Eleanor Williams, Managing Director, Australian Centre for Evaluation
Panelists: Prof Lisa Cameron, Professional Research Fellow, Melbourne Institute of Applied Economic and Social Research, University of Melbourne
Dr Wendy Jarvie, Adjunct Professor, Public Service Research Group, University of NSW
Bruce Cunningham, Assistant Secretary, Employment Evaluation Branch, Commonwealth Department of Employment and Workplace Relations
Commentators: Prof Patricia Rogers, Former Professor of Public Sector Evaluation, RMIT University, AES Fellow
Scott Bayley, Principal, Scott Bayley Evaluation Service, AES Fellow

The Commonwealth Government has established the Australian Centre for Evaluation (ACE) to put evaluation evidence at the heart of policy design and decision-making by improving the volume, quality, and use of evaluation evidence to support better policy and programs that improve the lives of Australians. This aligns well with the aim of the AES to improve the theory, practice and use of evaluation for people involved in evaluation. The creation of ACE provides an excellent opportunity for the AES and its members to work with a government agency on our common purposes. This collaboration has already commenced through shared activities and, in particular, the involvement of the responsible Minister, Dr Andrew Leigh, as a keynote speaker at the 2023 AES Conference.

An area that has attracted considerable attention is the mandate for ACE to include randomised control trials (RCTs) in at least some of their evaluation studies of Commonwealth programs. This issue was the central topic of Minister Leigh’s keynote address and created considerable debate and discussion at the conference. This demonstrates that this is a topic of importance for the AES and its members.

The aim of the session is to explore the appropriate use of RCTs in evaluation studies of public policy in Australia. The strategy is to commence a communication process on this key topic between ACE and the evaluation community as represented by the AES. Ideally, this will lead to collaboration between ACE and the AES to improve evaluation practice in Australia.

The Fellows Forum session will commence with a prepared presentation by a senior staff member of ACE explaining its mandate and outlining its approach to including RCTs in evaluation studies. This will be followed by a panel of evaluators who have experience with RCTs to explain how they included RCTs in an evaluation study or where they chose not to include an RCT and the reasons why. They will also explore what they learned from this experience to inform their future evaluation practice. Finally, one or two Fellows will act as discussants, responding to the previous presentations with their thoughts on this issue. The session will be moderated by a Fellow and there will be time for audience members to ask questions of the panel members and discussants.

Moderator:  Prof Rick Cummings, Emeritus Professor Murdoch University, AES Fellow
Key Speaker: Eleanor Williams, Managing Director, Australian Centre for Evaluation
Panelists:  Prof Lisa Cameron, Professional Research Fellow, Melbourne Institute of Applied Economic and Social Research, University of Melbourne
                       Dr Wendy Jarvie, Adjunct Professor, Public Service Research Group, University of NSW
                       Bruce Cunningham, Assistant Secretary, Employment Evaluation Branch, Commonwealth Department of Employment and Workplace Relations
Commentators:  Prof Patricia Rogers, Former Professor of Public Sector Evaluation, RMIT University, AES Fellow
Scott Bayley, Principal, Scott Bayley Evaluation Service, AES Fellow

Authors: Amanda Mottershead (Tetra Tech International Development), Qudratullah Jahid (Oxford Policy Management Australia, AU),Eroni Wavu (Pacific Community, FJ)

The sustainability of the evaluation sector requires emerging evaluators to be supported in pursuing high-quality practice. What this support needs to be and how it should be developed is much less certain. What topics should we focus on? How should we deliver it? Who should we deliver it to? How can the broader evaluation community support emerging evaluators?

Global experiences in emerging evaluator support contain a treasure trove of lessons which can fill this knowledge gap and inform effective support here in Australia. Experiences show that fostering a strong evaluation community, that includes emerging evaluators, can nurture, ignite and shape future evaluation practices. A variety of approaches are being adopted across the region, and the globe, to foster this sense of community, that range from formal approaches to capacity building to more informal approaches that focus on experience sharing.

In this session, we bring together current and former emerging evaluator leaders from across the Asia Pacific region to answer some of these questions and understand what approaches could work best for the Australian context. This will include presentations and discussion on in-demand topics, how to formulate support, how to target emerging evaluators and the best means of delivery. The session will be highly interactive, engaging the audience in a question-and-answer forum on this important topic. All panel members have been engaged with emerging evaluator networks in their countries or regions and bring diverse experiences to facilitate cross learning. The session will provide practical ways forward for the broader evaluation community to grow and support the future of evaluation.