aes24 International Evaluation Conference

Authors: Kabira Namit (Abt Global ),Kathryn Lee (Abt Global, AU)

This hands-on session is designed to strengthen intuition of causality in non-experimental evaluations among emerging evaluators.

In environments where conventional RCTs are unethical or unfeasible, identifying causal factors within the complex weave of societal factors and individual behaviours presents a significant challenge. Centred on a novel research project from Papua New Guinea, this session navigates this maze through real-world research (exploring the intricate relationship between the consumption of carcinogenic betel nut and its impact on educational outcomes). By focusing on this specific case study, we provide a concrete context for participants to understand the broader implications of causal explorations in fragile and sensitive settings.

Participants will actively engage in small group discussions in a collaborative learning environment where they can practice and refine their skills in causal evaluation by discussing scenarios that are reflective of real-world complexities.

This session aims to move beyond simply documenting correlations, encouraging a deep dive into the underlying dynamics of causal linkages. Through this exploration, we aim to eventually guide participants to discussions on pathways for targeted interventions and policy formulations which take causal chains into account.

Additionally, we aim to spark dialogue on the ethical dimensions of 'activist research,' exploring how evaluators can navigate moral dilemmas while advocating for meaningful change. This hands-on session not only seeks to build evaluative skills but also to inspire participants to consider the broader implications of their work on societal well-being and ethical research practices.

Authors: Laura Glynn (Simna)

The measurement space has seen many new actors, terms, approaches and “gold standards” emerge in the last two decades. More than ever before has it become difficult to navigate and explore our intended destination in the space of measurement and evaluation. What schools of thought are worth exploring? What value do they offer to an existing evaluation skillset? We are also traversing through heightened levels of complexity, with cost of living, environmental and society fabric crises. In this busy and crowded environment, the Social Impact Measurement Network (SIMNA) led panel will seek to explore the similarities and differences between evaluation and social impact measurement (SIM) as mindsets to help steer us towards our destination.

The panel will involve 3 speakers from diverse sectoral backgrounds – government, not-for-profit, and private spheres, all commenting (broadly) on the questions: Are evaluation and social impact measurement the same? To what extent do they differ? How can they complement one another? While the questions themselves will be more nuanced than that, the answers will hold broad value for attendees in considering how they can bring complementary approaches and mindsets to navigating the work they do in measurement and evaluation. The panellists will draw on their unique perspectives across different sectoral and practice spaces to discuss this complementarity.

Authors: John Gargani (Gargani + Co)

The world has never faced larger problems—climate change, refugee crises, and Covid19, to name just three. And organizations have responded by scaling solutions to unprecedented size—sustainable development goals, global refugee policies, and universal vaccination programs. But scaling is a journey to a destination imperfectly imagined at the onset and difficult to recognize upon arrival. At what point is scaling a program, policy, or product successful? Under what conditions should scaling stop? Or "descaling" begin? Robert McLean and I posed these and other questions to innovators in the Global South and shared what we learned in our recent book *Scaling Impact: Innovation for the Public Good*. In this session, we outline the book's four research-based scaling principles—justification, optimal scale, coordination, and dynamic evaluation. Then we discuss how to (1) define success as achieving impact at optimal scale, (2) choose a scaling strategy best suited to achieve success, and (3) judge success with dynamic evaluation. My presentation goes beyond the book, reflecting our most current thinking and research, and I provide participants with access to free resources, including electronic copies of the book.

Authors: Mallory Notting (First Person Consulting)

The ACT Government implemented a suite of programs over the ten year period between 2012 and 2022 aiming to increase the rates of students actively travelling to and from school. 102 schools in the ACT participated in at least one of the three programs during this time which targeted well-known barriers to active travel, including parental perceptions of safety and infrastructure around school. The programs were intended to contribute towards a range of broader priorities, including health, safety, and environmental outcomes.

This short-paper session will share learnings from evaluating long-term behaviour change at a population level, based on the school active travel evaluation. The evaluation represents a unique case study, as the evaluators needed to look retrospectively over ten years of program delivery and assess whether the combination of programs had created changes within the system and had resulted in the achievement of wider goals.

The presenter will illustrate that the line between short-term and long-term outcomes is rarely linear or clear, as is the relationship between individual interventions and whole of system change. This will be done by summarising the approach taken for the evaluation and sharing the diversity of information collated for analysis, which included individual program data and attitudinal and infrastructure-level data spanning the whole school environment.

Evaluators are often only able to examine the shorter term outcomes of an intervention, even in complex policy areas, and then rely on a theory of change to illustrate the assumed intended wider impacts. The presenter was able to scrutinise these wider impacts during the active travel evaluation, an opportunity not regularly afforded to evaluators. The lessons from the active travel evaluation are therefore pertinent for other evaluations in complex policy areas and may carry implications for program design as the focus shifts increasingly towards population-level, systems change.

Authors: Skye Trudgett (Kowa ),Haley Ferguson (Kowa, AU),Tara Beattie (Kowa, AU),Levi McKenzie-Kirkbright (Kowa, AU),Jess Dart (Clear Horizon, AU)

In the pursuit of understanding and honouring the depth of First Nations wisdom, this hands-on session at the AES conference introduces the Ancestral Knowledge Tapestry —a living guide for developing a repository of ancestral knowledge, practices, and philosophies. Participants will actively engage in co-creating a 'Wisdom Base,' a collective endeavour to encapsulate the richness of old and new First Nations knowledges and their application to contemporary evaluative practices.

Through interactive exercises, collaborative dialogue, and reflective practices, attendees will delve into the components of the Ancestral Knowledge Tapestry, exploring the symbiosis between deep knowing, artefacts, deep listening and truth-telling. The session aims to empower participants, particularly those from First Nations communities, to identify, document, and share their unique wisdom in ways that foster self-determination and cultural continuity.
Attendees will emerge from this workshop not only with a deeper appreciation for the intrinsic value of First Nations knowledge systems but also with practical insights into how to cultivate a Wisdom Base that not only preserves but actively revitalises First Nations wisdom for future generations.