aes24 International Evaluation Conference

Authors: Awuor PONGE (African Policy Centre (APC) )

Indigenous Knowledge Systems (IKS) in Africa have long been marginalized and undervalued, despite their potential to offer sustainable solutions to pressing challenges faced by communities across the continent. This presentation explores the perspectives, challenges, and opportunities for incorporating IKS into evaluation practices that create value for both people and the planet.

From a people-centric perspective, IKS offer a holistic and culturally relevant approach to understanding local contexts, priorities, and value systems. By embracing these knowledge systems, evaluations can better capture the multidimensional nature of well-being, including spiritual, social, and environmental aspects that are often overlooked in conventional evaluation frameworks. However, challenges arise in reconciling IKS with dominant Western paradigms and navigating power dynamics that have historically suppressed indigenous voices.

From a planetary perspective, IKS offer invaluable insights into sustainable resource management, biodiversity conservation, and climate change adaptation strategies that have been honed over generations of lived experiences. Integrating these knowledge systems into evaluation can shed light on the intricate relationships between human activities and ecosystem health, enabling more informed decision-making for environmental sustainability. Nonetheless, challenges exist in bridging the divide between traditional and scientific knowledge systems, as well as addressing concerns around intellectual property rights and benefit-sharing.

This presentation will explore innovative approaches to overcoming these challenges, such as participatory and community-based evaluation methodologies, capacity-building initiatives, and cross-cultural dialogue platforms. By fostering a deeper appreciation and understanding of IKS, evaluation practices can become more inclusive, relevant, and effective in creating value for both people and the planet in the African context.

Authors: John Gargani (Gargani + Co ),Julian King (Julian King & Associates, NZ)

In this participatory session, we pose the question, "How should evaluators innovate the practice of value-for-money assessment to meet the needs of an expanding set of actors that include governments, philanthropists, impact investors, social entrepreneurs, program designers, and Indigenous and First Nations communities?" We begin by framing value for money as an evaluative question about an economic problem. How well are we using resources, and are we using them well enough to justify their use? Then we suggest new methods intended to help innovate the practice of value for money based on our body of published and current research spanning over 10 years.
These include new methods that (1) produce "holistic" assessments of value for money, (2) reflect rather than hide multiple value perspectives even when values conflict, (3) estimate social benefit-cost ratios without monetizing benefits or costs, and (4) adjust monetary and nonmonetary value for risk using Bayesian methods. Along the way, we facilitate discussions with participants, asking them to consider if, how, and by whom these innovations should be pursued, and what other innovations may be needed. We provide participants with access to a collection of our published and draft papers, and invite them to comment and continue our discussion after the conference.

Authors: Rhiannon Birch (Victorian Department of Education ),Hayden Jose (Urbis, AU),Joanna Petkowksi (Victorian Department of Education, AU),Ekin Masters (Victorian Department of Education, AU)

How can an evaluation balance the need to generate pragmatic insights while meeting central agency requirements for rigorous measurement of outcomes? What ingredients can facilitate the effective evaluation of a government initiative and achieve improved outcomes? This paper explores the essential ingredients for evaluating a large-scale government program using an example of a statewide initiative aimed at attracting and retaining suitably qualified teachers in hard-to-staff positions in Victorian government schools.

We showcase how an adaptive and evidence-led method of enquiry helped identify program implementation bottlenecks and probe potentially unintended program outcomes over a three-year evaluation. We discuss enablers for the integration of evaluation recommendations into program implementation and future policy direction, particularly on participatory action approaches and deep relationships with policy and implementation teams. We will also present the robust and varied methodology, particularly the novel use of system data to facilitate a quasi-experimental design that aligned with central agency requirements and met stakeholder needs.
This presentation will benefit policymakers, program evaluators, and others interested in evaluating government programs, by sharing key learnings on how evaluations can balance pragmatic insights with central agency requirements and identifying the key elements for influencing such programs and achieving improved outcomes.

Authors: Laura Faulker (NSW Treasury)

This paper presents the key findings from research and engagement on how cost-benefit analysis (CBA) has been applied to First Nations initiatives to date. CBA is an important tool used by governments to help prioritise budget funding decisions. It assesses the potential impacts of an initiative - economic, social, environmental, and cultural - to determine whether it will deliver value for money.

The paper explores the methods in which the value of First Nations cultures has been incorporated into CBAs, along with the associated challenges and opportunities to improve current practice. The findings have informed the development of an investment framework for the design and evaluation of initiatives that affect First Nations people and communities. The framework focuses on the key principles for embedding First Nations perspectives and ensuring culturally informed evaluative thinking.

Authors: Alli Burness (Tetra Tech), Lisa Faerua (Vanuatu), Nathan Sentance (Museum of Applied Arts and Sciences, AU), David Lakisa (Talanoa Consultancy, AU)

In the inaugural ANU Coral Bell Lecture on Indigenous Diplomacy, Dr Mary Graham outlined a powerful legacy of Aboriginal and Torres Strait Islander relational methods that have operated across a spectacular time scale. She envisioned a compelling future for its renewed application and spoke of these practices as a type of "thinking in formation, a type of slow, collective, and emergent process".

Inspired by Dr Graham's vision, this panel explores synergies, distinctions, and complementarities in local and Indigenous research methods across Australia and the Pacific. The panel features Wiradjuri, Samoan (Polynesian), Ni-Vanuatu (Melanesian) and settler-background (Australian) researchers from a range of fields who will explore, engage and showcase locally specific methodologies that connect across Australia and the Pacific continents, as ways of knowing, doing, and relating with the land, the moana (ocean) and air.

This session frames evaluation and research approaches as reflecting their contextual political order. While the panel will critique the legacies of individualist and survivalist research methods, it will focus on exploring the futures that relational research methods could realize. How do we evolve current institutional approaches to become more commensurate with Indigenous methods? Would institutionalizing these methods resolve the legacy, structure, and form of colonialist political approaches? Panelists will speak to their experience in working to evolve institutions in this way and the research and evaluation methodologies used within them.

The session also situates evaluation within a cannon of contextualizing evidence-based practices (such as political economy analysis, GEDSI analysis or feasibility.

Authors: Skye Trudgett (she/her) - Kowa, AU, Katie Stubley (she/her) - Griffith University, AU, Terri Reid (she/her), Chloe Wegener(she/her), Banok Rind (she/her), Sophie Spry (she/her), Niamh Kealy (she/her)- Wiyi Yani U Thangani Institute for First Nations Gender Justice, AU

Measurement and data does not need to be all about numbers—it is about our heart and spirits, it is about voice, story, emotion—it is about truth. Numbers can tell us all sorts of lies, Wiyi Yani U Thangani is and always has been about—your voice—what you are saying about your lives, how you see your future and what matters to you.' June Oscar AO, Chair of Wiyi Yani U Thangani Institute.
Quantitative data often dominate measurement and evaluation, yet true understanding requires tapping into the heart and spirit of communities. As June Oscar AO poignantly reminds us, it is the voice, story, emotion, and truth that bring depth to data. This session at the AES conference offers a hands-on experience that transcends traditional data collection, engaging participants in co-creating collaborative Impact Yarns through deep listening, yarning, and creative expression.
Join us in a dynamic workshop where sculpture and artwork become powerful tools for storytelling and knowledge sharing. Participants will learn to capture the nuanced experiences of First Nations communities, reflecting on how these creative practices can reveal the interconnectedness of our lives and contribute to systemic change. By integrating Indigenous methodologies, we will collectively explore the rich, qualitative data that emerges from individuals' lived realities and aspirations.
As we craft and shape our narratives, we will reflect on how these stories can inform and transform policies and initiatives. This immersive session is not just about creating art; it is about embodying the principles of gender justice and equality, respecting cultural heritage, and acknowledging the diverse ways communities envision their future.
Experience the power of collaborative creation, where every word spoken, and every form sculpted enriches our collective understanding of impact. This workshop is an invitation to step away from the spreadsheet and into a space where every voice contributes to a tapestry of change. Come, let us shape a more empathetic and embracing approach to measurement—one that values the stories and truths of all peoples. truths of all peoples.

Authors: Kathryn Erskine (Cube Group), Michael Maher (Cube Group, AU)

The session is intended to provide a practical example of how a traditional program evaluation was re-orientated to allow the application of findings to inform broader system transformation. This echoes current discourse in evaluation - particularly since the global pandemic - whereby traditional notions about how the field of evaluation is viewed and developed are being challenged (Ofir, Z. 2021). Specifically, there are calls to rethink and elevate evaluation practice to actively contribute to and support systems transformation (see Dart 2023; Norman 2021), beyond a narrow programmatic focus.

This session will illuminate this discussion by examining a mental health program evaluation in the context of significant service reform across the Victorian mental health system. The presentation will outline insights and techniques about how to lift and reconfigure a tightly defined program evaluation into one which can have broader application to the system ecosphere. It outlines how and why the pivot was made; changes we made to the methodology and the key benefits that arose from taking an expansive view of the sector in which the program operated within.

The design of the session will be a presentation format supported by a PowerPoint slide deck, comprising:
•    Introduction and purpose of session
•    Overview of the program we evaluated
•    Key challenges which required an evaluation 'pivot' - and how we worked with our client
•    Key changes made to the methodology
•    Key benefits from elevating from a programmatic to systems focus.

Authors: Allan Mua Illingworth (Mua'akia Consulting and Insight Pasifika) Fiona Fandim (Pacific Community (SPC), FJ), Eroni Wavu (MEL Officer for Pacific Women Lead at Pacific Community (SPC) and cofounder of the Fiji Monitoring, Evaluation & Learning Community), Mereani Rokotuibau (Balance of Power Program, FJ) and Chris Roche (La Trobe University),

In recent years there have been a number of Pacific driven initiatives designed to promote monitoring and evaluation practice which is culturally and contextually appropriate. These have occurred with projects and programs as well as at national and regional levels. At the same time geo-political interest in the Pacific region has resulted in an increased number of bi and multilateral donor agencies becoming present in the region and/or funding development programs, local organisations, national governments and regional bodies. This has in turn led to an evaluation landscape where notions of 'international best practice' as well as donor policies and practices and associated international researcher and consulting companies, risk crowding out emergent Pacific led evaluation initiatives.

This panel will bring together key participants who are leading four examples of these Pacific experiences: the Rebbilib process initiated by the Pacific Community (SPC ), Insight Pasifika (an emerging Pacific led and owned collective focused on evaluation in the first instance): the Fiji Monitoring, Evaluation & Learning Community and the Balance of Power program (a Pacific-led initiative, supported by the Australian Government, focused improving the political, social and economic opportunities for women and girls) each of whom are seeking to create space for processes of monitoring, evaluation and learning which are consistent with Pacific ways of knowing and being. They will share their experience, the challenges they face and ideas about what forms of support might be provided by international donors, consultants and advisors which are enabling rather than undermining.

Moderated by Prof. Chris Roche the panel and audience will also draw out the lessons from these four cases about what might contribute to more systemic change in the evaluation landscape more generally.

Authors: Skye Trudgett (Kowa), Rachel Kerry (CAGES Foundation, AU)

The philanthropic sector has long grappled with power imbalances inherent in funding relationships, particularly in the context of support for First Nations communities. This panel session explores a groundbreaking Monitoring, Evaluation, and Learning (MEL) approach that inverts traditional power structures, placing First Nations communities in the driver's seat to assess the adherence of a leading philanthropic organisation to their stated values and principles.

Drawing from the collaborative efforts of one foundation and its MEL partner, this session showcases a MEL model that exemplifies shared power and mutual accountability. The panel will consist of thought leaders from the philanthropic sector, First Nations community representatives, and MEL experts who have been at the forefront of developing and implementing this innovative approach.

Through a facilitated discussion, panellists will delve into the process of co-creating a MEL framework that empowers communities to evaluate the performance of philanthropists against a set of mutually agreed-upon criteria. This approach ensures that philanthropic actions align with community expectations, cultural protocols, and contribute to genuine and sustainable impact.

Attendees will gain insights into the challenges and successes of operationalising this community-centric MEL method. The session aims to inspire other philanthropic entities to reflect on their practices and adopt similar approaches that truly shift power to First Nations communities.

Authors: Ossie Lang (RMCG ),Donna Lucas (RMCG ),Carl Larsen (RMCG ),Zarmeen Hassan (AUSVEG ),Cherry Emerick (AUSVEG ),Olive Hood (Hort Innovation )

How do you align ten regionally delivered projects with differing focus topics to nationally consistent outcomes? Take advantage of this opportunity to explore the journey of building and implementing a robust Monitoring and Evaluation (M&E) program that showcases regional nuances and aligns national outcomes, making a significant contribution to the success of this horticultural industry extension project.

Join us for an insightful presentation on how a national vegetable extension project focused on adoption of best management practices on-farm, has successfully implemented a dynamic M&E program. Over the two and a half years of project delivery, the national M&E manager, in collaboration with ten regional partners, has crafted a program that demonstrates regional impact consistently on a national scale and adapts to the project's evolving needs.

The presentation will highlight the team's key strategies, including the upskilling of Regional Development Officers in M&E practices. Learn how templates and tools were designed to ensure consistent data collection across approximately 40 topics. The team will share the frameworks utilised to capture quantitative and qualitative monitoring data, providing a holistic view of tracking progress against national and regional outcomes and informing continuous improvement in regional delivery.

Flexibility has been a cornerstone of the M&E program, allowing it to respond to the changing needs of growers, industry, and the funding partner and seamlessly incorporate additional data points. Discover how this adaptability has enhanced the project's overall impact assessment and shaped its delivery strategy.

The presentation will not only delve into the national perspective but also feature a firsthand account from one of the Regional Development Officers. Gain insights into how the M&E program has supported their on-the-ground delivery, instilling confidence in providing data back to the national project manager. This unique perspective offers a real-world understanding of the national program's effectiveness at a regional level.

Authors: Phillip Belling (NSW Department of Education), Liam Downing (NSW Department of Education, AU)

This paper is aimed at early career and experienced evaluators interested in realist evaluation, but with concerns about the time a realist approach might take. The authors respond to this concern with an innovative blending of realist and developmental evaluation. Participants will exit the room with a working understanding of realist developmental evaluation, including its potential for adaptive rigour that meets the needs of policy makers and implementers.

Realist evaluation is theoretically and methodologically robust, delivering crucial insights about how, for whom and why interventions do and don't work (House, 1991; Pawson & Tilley, 1997; Pawson, 2006). It aims to help navigate unfamiliar territory towards our destination by bringing assumptions about how and why change happens out in the open.

But even realism's most enthusiastic practitioners admit it takes time to surface and test program theory (Marchal et al., 2012; van Belle, Westhorp & Marchal, 2021). And evaluation commissioners and other stakeholders have understandable concerns about the timeliness of obtaining actionable findings (Blamey & Mackenzie, 2007; Pedersen & Rieper, 2008).

Developmental evaluation (Patton, 1994, 2011 2021; Patton, McKegg, & Wehipeihana, 2015) is more about what happens along the way. It appeals because it provides a set of principles for wayfinding in situations of complexity and innovation. Realist and developmental approaches do differ, but do they share some waypoints to reliably unpack perplexing problems of practice?

This paper documents a journey towards coherence and rigour in an evaluation where developmental and realist approaches complement each other, and deliver an evidence base for program or policy decision-making that is not only robust but also timely.

We show that, in complex environments, with programs involving change and social innovation, realist developmental evaluation can meet the needs of an often-varied cast of stakeholders, and can do so at pace, at scale, and economically.

Authors: Samiha Barkat (Launch Housing) Edgar Daly (Launch Housing)

Embedded evaluation roles challenge conventional boundaries between evaluator, commissioner, strategist, advocate, program manager and critical friend.

Three years into establishing an embedded impact and evaluation team at a large not-for-profit community service organisation, Samiha and Edgar share their insights on operating as internal evaluators to advance their organisation’s mission. They draw from their recent projects, including developing and implementing an award-winning impact measurement framework. Central to their learnings is the importance of valuing relationships with diverse stakeholders and maintaining an adaptive approach.

Their insights are particularly valuable for organisations seeking to enhance the effectiveness, cost-efficiency, and impact of their evaluation services, including internal evaluation teams, program managers, and executives from both non-government and government organisations.

Samiha and Edgar work for Launch Housing, a secular Melbourne-based community organisation that delivers homelessness services and life-changing housing supports to disadvantaged Victorians. Launch Housing employs over 400 staff and runs more than 50 programs aimed at ending homelessness in Melbourne.

Authors: John Gargani (Gargani + Co)

The world has never faced larger problems—climate change, refugee crises, and Covid19, to name just three. And organizations have responded by scaling solutions to unprecedented size—sustainable development goals, global refugee policies, and universal vaccination programs. But scaling is a journey to a destination imperfectly imagined at the onset and difficult to recognize upon arrival. At what point is scaling a program, policy, or product successful? Under what conditions should scaling stop? Or "descaling" begin? Robert McLean and I posed these and other questions to innovators in the Global South and shared what we learned in our recent book *Scaling Impact: Innovation for the Public Good*. In this session, we outline the book's four research-based scaling principles—justification, optimal scale, coordination, and dynamic evaluation. Then we discuss how to (1) define success as achieving impact at optimal scale, (2) choose a scaling strategy best suited to achieve success, and (3) judge success with dynamic evaluation. My presentation goes beyond the book, reflecting our most current thinking and research, and I provide participants with access to free resources, including electronic copies of the book.

Authors: Skye Trudgett (Kowa ),Haley Ferguson (Kowa, AU),Tara Beattie (Kowa, AU),Levi McKenzie-Kirkbright (Kowa, AU),Jess Dart (Clear Horizon, AU)

In the pursuit of understanding and honouring the depth of First Nations wisdom, this hands-on session at the AES conference introduces the Ancestral Knowledge Tapestry —a living guide for developing a repository of ancestral knowledge, practices, and philosophies. Participants will actively engage in co-creating a 'Wisdom Base,' a collective endeavour to encapsulate the richness of old and new First Nations knowledges and their application to contemporary evaluative practices.

Through interactive exercises, collaborative dialogue, and reflective practices, attendees will delve into the components of the Ancestral Knowledge Tapestry, exploring the symbiosis between deep knowing, artefacts, deep listening and truth-telling. The session aims to empower participants, particularly those from First Nations communities, to identify, document, and share their unique wisdom in ways that foster self-determination and cultural continuity.
Attendees will emerge from this workshop not only with a deeper appreciation for the intrinsic value of First Nations knowledge systems but also with practical insights into how to cultivate a Wisdom Base that not only preserves but actively revitalises First Nations wisdom for future generations.

Authors: Julie Elliott (Independent Evaluator), Jill Thomas (J.A Thomas & Associates, AU), Martina Donkers (Independent Evaluator, AU)

Mentors and mentees from the AES Group Mentoring Program share rich experiences of group learning, knowledge sharing, and reflective practice, exploring the Wayfinding skills, knowledge, and expertise they have found through the program and the valuable lessons learned.

AES remains committed to mentoring, and this session provides a unique opportunity to hear perspectives across the mentoring spectrum, from Fellows to emerging evaluators, and the ways that sharing our professional practice enhances our work. Since 2021, the AES Group Mentoring Program has been a trailblazer in fostering professional growth and competencies for emerging and mid-career evaluators, enabling mentors and peers to help navigate unfamiliar territories, incorporating various tools and strategies.

Our dynamic panel will discuss how evaluators have adapted their approaches to mentoring and to evaluation practice with the support of the program. It's a session where personal and professional growth intersect and will offer a unique perspective on the transformative power of mentorship.

This discussion is for evaluators who are passionate about learning - both their own and that of other AES members! Whether you're a seasoned professional eager to contribute to your community, an emerging talent or a mid-career evaluator navigating contemporary evaluation ecosystems, this session is for you. Don't miss this opportunity to hear directly from mentors and mentees who value the shared, continuous journey of social learning and adaptation.

Authors: Amy Gullickson (University of Melbourne), Taimur Siddiqi (Victorian Legal Services, AU)

AES in collaboration with learnevaluation.org offers a competency self-assessment to members. The aim to help individuals understand their strengths and plan their learning journey, to help the AES continue to tailor its professional development offerings and develop pathways to professionalisation, and to contribute to ongoing research about evaluation learners. In this session, members of the AES Pathways Committee will briefly summarise the findings from the self-assessment and then invite participants into groups by their discipline and sector to discuss: Which competencies are really core and why? Reporting out from groups will will reveal whether the core competencies differ based on the sectors/background of the evaluators. The follow up discussion will then explore: What do the findings mean for evaluation practice, and teaching and learning? How do they relate to professionalisation? If we want to increase clarity about what good evaluation practice looks like - what are our next steps related to the competencies?

Participants will benefit from reflecting on their own competency self-assessment in relation to the findings and discussion, and discovering how the backgrounds of learners influences their ideas about core competencies. The session findings will be shared with the AES Pathways Committee to inform AES' next steps for the competencies, self-assessment, and ongoing discussion of pathways to professionalisation.

Authors: Sharon Marra-Brown (ARTD Consultants), Moya Johansson (ARTD Consultants, AU)

Think it's important to enable children and young people to have a voice in evaluations, but find it challenging? This paper presents tried and tested strategies for ensuring ethical engagement with children and young people and encouraging meaningful participation.

Involving children and young people in evaluation is critical to ensure that we arrive at evaluations that accurately reflect their experiences and capture the outcomes they consider most important. Children and young people have the right to have a say about their experiences, and evaluations that avoid their involvement risk perpetuating ongoing inequities.

However, involving children and young people in evaluations can prompt ethical concerns in relation to their comprehension of research, capacity to provide consent, potential coercion by parents, and the potential conflicting values and interests between parents and children. Depending on the subject, it can also create concerns about safety and readiness.

Based on our experience successfully achieving ethics approval for multiple evaluations of services for children and young people across Australia, which include interviews with children and young people who have accessed these services, we will talk through considerations for ensuring the voice of children and young people in evaluation while safeguarding them from unnecessary risks.

We will then take you through how we've overcome challenges engaging children and young people in evaluations with youth-centred innovative solutions, including carefully considering the language we use and how we reach out. We will demonstrate the developmental benefits of meaningful participation of children and young people once ethical considerations have been carefully considered and navigated.

Finally, we will take you through our tips for ensuring meaningful and safe engagement with children and young people. We will point you in the direction of Guidelines and practice guides for involving young people in research and evaluation in a safe and meaningful way.

The presenters are evaluators with extensive experience in designing, delivering and reporting on evaluations that include data collection with children and young people. This includes recently achieving ethics approval and commencing interviews with children as young as seven, accessing a suicide aftercare service.

While much attention is devoted to ensuring safe and inclusive data collection with various demographics, specific considerations for engaging children and young people remain relatively uncommon. Recognising the unique needs of this population, coupled with the understandably cautious stance of ethics committees, underscores the necessity for a thoughtful and deliberate approach to evaluations involving children and young people.

Given the additional complexities and ethical considerations involved, the default tendency can be to exclude children and young people from evaluation processes. However, it is important that children and young people are able to have a say in the programs, policies and services that they use. Participation in evaluations can be a positive experience, if risks are managed and the process is designed to be empowering.

This session will provide valuable insights, actionable strategies, and an opportunity for participants to reflect on their own practices, fostering a culture of inclusivity and responsiveness in evaluation.

Authors: Anne Smyth (LDC Group), Lesley Thornton (LDC Group, AU), Kym Coupe (First Step, AU), Caroline Lynch (Launch Housing, AU)

As all Wayfinders would understand, when we embarked on a developmental evaluation of the Road Home, we really had no idea how the program or evaluation would play out in practice. We did know however that the usual way of delivering crisis housing services was not working well for either clients or staff. Something needed to change. We needed to change. So, we did - we being the Road Home team working with the evaluators.

Road Home centres on a strong and engaged multidisciplinary team to deliver mental health, medical, legal and housing services to people in crisis accommodation, where they are, and when they need it the most. This integrated way of working is in stark contrast to the conventional, single discipline outreach and in-reach approaches that characterise service delivery in the community sector - its impact has been significant.

This panel will bring leading representatives of the Road Home team and the evaluators together to explore with our audience what we have learned; what it takes to do this well, the benefits to clients, staff and participating organisations, the pitfalls and challenges and the value of developmental evaluation and its methods.

We now have a much better idea of what Road Home looks like, what it takes to support and enable it, to achieve valued outcomes and to meaningfully evaluate it. The role of evaluators and the project manager in holding the uncertain and evolving space characteristic of developmental evaluation and wayfinding is central - it has taken clarity, alignment of purpose, a lot of patience and much persistence, not to mention flexibility. It has been and remains quite the journey!

Authors: John Guenther (Batchelor Institute Of Indigenous Tertiary Education), Anthea Rutter (University of Melbourne, AU), Yvonne Zurynski (Macquarie Univesity, AU)

The Evaluation Journal of Australasia (EJA) supports evaluators who wish to share their knowledge and practical experiences in a peer-reviewed article. Documenting evidence, including for programs which do not achieve expected results, is critical for improving evaluation practice, building the evidence base, and advancing evaluation methodologies that are rigorous and ethical.

The EJA depends on volunteer reviewers who can offer critical feedback on articles that are submitted. Reviewers help to improve the quality of manuscripts the Journal receives.

The focus of this presentation is on how to write a good review: how to be academically critical, while at the same time providing constructive feedback that will benefit authors and readers. The presenters will offer step-by-step advice on what to look for, how to judge the quality of a manuscript, and how to make constructive suggestions for authors to consider.

The presentation will also explain how reviewing fits within the publication process, from submission to production. It will be most helpful to potential authors and current and potential reviewers. Authors will learn how to prepare their articles so they receive a favourable review, and reviewers will receive clear guidance on presenting their review feedback to authors.

Authors: Julian King (Julian King & Associates Limited), Adrian Field (Dovetail Consulting Limited, NZ)

Evaluation rubrics are increasingly used to help make evaluative reasoning explicit. Rubrics can also be used as wayfinding tools to help stakeholders understand and participate meaningfully in evaluation. Developing rubrics is conceptually challenging work and the search is on for additional navigation tools and models that might help ease the cognitive load.

As a preliminary step toward rubric development it is often helpful to co-create a theory of change, proposing a chain of causality from actions to impacts, documenting a shared understanding of a program, and providing a point of reference for scoping a logical, coherent set of criteria.

However, it's easy to become disoriented when getting from a theory of change to a set of criteria, because the former deals with impact and the latter with value. Implicitly, a theory of change may focus on activities and impacts that people value, but this cannot be taken for granted - and we argue that value should be made more explicit in program theories.

Specifying a program's value proposition can improve wayfinding between a theory of change and a set of criteria, addressing the aspects of performance and value that matter to stakeholders. Defining a value proposition prompts us to think differently about a program. For example, in addition to what's already in the theory of change, we need to consider to whom the program is valuable, in what ways it is valuable, and how the value is created.

In this presentation, we will share what we've learnt about developing and using value propositions. We'll share a simple framework for developing a value proposition and, using roving microphones, engage participants in co-developing a value proposition in real time. We'll conclude the session by sharing some examples of value propositions from recent evaluations.

Authors: Linda Kelly (Praxis Consultants), Elizabeth Jackson (Latrobe University, AU)

This presentation focuses on a multi-year program funded by Australia that aims to empower people marginalised by gender, disability and other factors. Like similar programs, the work is subject to regular monitoring and evaluation - testing the effectiveness of program activities largely from the perspective of the Australian and national country Government.
But what of the views of the people served by the program? Is the impact of the various activities sufficient to empower them beyond their current condition? How significant are the changes introduced by the program, given the structural, economic, social and other disadvantages experienced by the marginalised individuals and groups.
Drawing from feminist theory, qualitative research methods and managed with local research and communication experts this presentation outlines the study focused on the long-term impact of the program.

The presentation will outline the methodology and practical considerations in the development of the approach and data collection methodologies. It will highlight the value of exploring impact from a qualitative perspective, while outlining the considerable management and conceptual challenges required in designing, introducing and supporting such an approach. It will consider some of the implications in shifting from traditional evaluation methods to more open-ended enquiry and consider whose values are best served through evaluation versus impact assessment?

Authors: Andrew McLachlan (NSW Department of Education)

At first glance, realist methodology seems ideally suited to investigating implementation problems (Dalkin et al., 2021). It is versatile in that it draws on theories from diverse fields of social inquiry. It is pragmatic in that the theories it adopts are deemed useful only in so far as they offer explanatory insight. And it is transferable; realist methodology is less concerned with generalising findings than in understanding how programs work under different conditions and circumstances.

As for implementation science, its founding aim is purpose built for realist work; it seeks to improve the uptake of evidence-based practices by investigating the barriers and facilitators to implementation. Yet despite the affinity between realist methodology and implementation science, so far there have been few attempts to formalise the relationship (Sarkies et al., 2022).

This paper offers insights into how evaluators can harness realist methodology to better understand challenges of program implementation. It demonstrates how implementation concepts like fidelity (the degree to which a program is delivered as intended), adaptation (the process of modifying a program to achieve better fit), and translation (the ability to transfer knowledge across organisational borders) can be combined with realist concepts to develop a more active understanding of context.

In showing how to construct program theories that are responsive to changing conditions, the paper promises to equip evaluators with tools that can help them navigate the complexities of program implementation in their own work.

Authors: Samantha Abbato (Visual Insights People)

Values guide all evaluation processes, methods and judgements. Although evaluators are often not aware of the values shaping their work and can't readily name them, they know when we are straying off their values path through the experience of conflict or unease. Reflecting on the evaluation literature and two decades of evaluation practice using a 'values' perspective, it is argued that there has never been a more important time to build values literacy. This presentation demonstrates how values literacy can guide conversations with yourself, your team and others and provide signposting and illumination of a more rigorous and ethical evaluation journey.

Authors: Kirstin Clements (Arts Centre Melbourne)

How does a creative institution develop an effective evaluation framework that honours the artistic process while meeting rigorous research standards?

At Arts Centre Melbourne we asked ourselves, 'what if?'... What if we integrated the economic story into a fit-for-purpose value proposition? What if we see the emotive, subjective nature of the arts as an asset, rather than a challenge in our evaluation design? What if we tried to embed systems thinking and extend our approach beyond individual projects?

Like many purpose-driven industries, the arts face an increasingly competitive funding landscape and heightened expectations from stakeholders for evidence-based reporting on the value generated by initiatives. Historically, in the arts such reporting has been responsive to external demands and formats. One of our core goals has been to equip the organisation with the capability and capacity to pro-actively drive its own public value narrative through a transparent, consistent approach.

In this presentation, we spotlight Arts Centre Melbourne's innovative approach to building appetite for evaluation and to designing a fit-for-purpose organisational impact model and evaluation function. We offer insights into the conceptual and methodological approaches we've adopted to achieve our objectives: supporting effective advocacy for the public value of the arts, enhancing accountability to stakeholders, and fostering a culture of continuous learning.

In sharing how we have creatively navigated challenges and opportunities at Arts Centre Melbourne, we aim to provide valuable advice and inspiration for evaluators and supporting professionals, particularly those working in sectors where evaluation is yet to be understood as 'business-as-usual' activity.