aes24 International Evaluation Conference

Authors: Charlie Tulloch (Policy Performance)

Many people will be attending their first AES conference - welcome!
This workshop is targeted at new and emerging evaluators who are seeking to build their familiarity with the key concepts and language used in evaluation and the main elements of evaluation project delivery.
This will provide you with a plotted history of the evaluation field so you feel more comfortable engaging in deeper-dive topics over coming days. The session is grounded in theory, drawing on leading thinkers and methods that are central to our practice.
Key concepts to be covered include: what is evaluation? why should we evaluate? what can be evaluated? when to evaluate? how to evaluate? where can I learn more about evaluation?
The forum will help many brave young and emerging evaluators to navigate through often choppy evaluation waters so they don't feel 'all at sea' over coming days.

Authors: Emily Lapinski (Central Land Council ),Malkirdi Napaljarri Rose (Centre For Human Security and Social Change, La Trobe University, AU), Glenda Napaljarri Wayne (Central Land Council, AU), Geoffrey Jungarrayi Barnes (Central Land Council, AU), Alex Gyles (Centre For Human Security and Social Change, La Trobe University, AU)

For evaluation to support transformational change, research suggests strategies must focus on localised Indigenous values, beliefs and worldviews. Decolonising evaluation involves identifying and addressing power and considering what is being evaluated, by whom and how. In this paper we argue that these developments are necessary but insufficient and suggest a possible way forward for further decolonising the field of evaluation. To support change for Indigenous Australians the emphasis needs to move from simple evaluation of individual programs to more critical examination of their combined impact on communities from local perspectives.

This paper explores how Warlpiri and non-Indigenous allies are collaborating to create and use their own community-level impact assessment tool. The 5-year Good Community Life Project is supporting Warlpiri residents of Lajamanu in the Northern Territory to define, assess and take action for a 'good community life'. Warlpiri will explain how they created the approach for assessing wellbeing in Lajamanu, and how they are using emerging results to give voice to their interests and advocate for the life they envision for future generations.

The project involves collaboration between Warlpiri community members, land council staff and university researchers, drawing on Indigenous concepts of 'two-way' seeing and working, relationality, and centring Indigenous voice and values. Applying these concepts in practice is challenging, particularly for non-Indigenous allies who must constantly reflect and use their privilege to challenge traditional views on 'robust' evaluation methodology.

Warlpiri and the land council see potential for this work to improve life in Lajamanu and as an approach that could be applied across Central Australian communities. Going beyond co-designed and participatory evaluation to critical examination of impact is the next step in supporting change with Indigenous communities. This paper will focus on Warlpiri perspectives, plus brief reflections from non-Indigenous allies, with time for the audience to discuss broader implications.

Authors: Mark Power (Murawin ),Carol Vale (Murawin, AU)

This presentation explores the paramount importance of culturally sensitive engagement methodologies in ensuring meaningful contributions from Aboriginal and Torres Strait Islander communities to mission programs. Murawin, an Aboriginal-led consultancy, has developed a robust Indigenous Engagement Strategy Framework grounded in the principles of reciprocity, free, informed and prior consent, mutual understanding, accountability, power sharing, and respect for Indigenous knowledge systems. Our session aims to share insights into the necessity of prioritising Aboriginal and Torres Strait Islander voices in engagement, co-design, and research, highlighting the significance of cultural competence in fostering mutual respect and understanding.
We will discuss three key messages: the imperative of deep knowledge and understanding of Aboriginal and Torres Strait Islander cultures in engagement practices; the success of co-design processes in facilitating genuine and respectful engagement; and the strategic partnership with CSIRO to enhance cultural competence and inclusivity in addressing Indigenous aspirations and challenges. These points underscore the critical role of acknowledging cultural interactions and ensuring cultural sensitivity in building strong, respectful productive relationships with Indigenous communities.
To achieve our session's objectives, we have designed an interactive format that blends informative presentations with the analysis of case studies, complemented by engaging intercultural discussions. This approach is intended to equip participants with actionable insights drawn from real-world examples of our collaborative ventures and co-designed projects. Through this comprehensive exploration, we aim to enrich participants' understanding of successful strategies for engaging Aboriginal and Torres Strait Islander communities, ultimately contributing to the achievement of more inclusive and impactful outcomes in mission programs and beyond.

Authors: Rachel Wilks (Grosvenor), Kristy Hornby (Grosvenor)

As we learn the practice of evaluation, we often have experienced evaluators around us. They provide us guidance and guardrails within which to learn and grow our capabilities - but what happens when we no longer have this support? How do we stop ourselves from getting lost?

This session will provide tips and tricks for evaluators going out into the world on their own. Attendees will leave with practical advice and strategies to help keep them on track while they build their autonomy.

Authors: Susie Fletcher (Australian Healthcare Associates)

Auslan is a visual, signed language that was developed by and for the Australian Deaf community. People who use Auslan as their primary or preferred language are not necessarily fluent in English. Our team was engaged to review access to interpreter services for Auslan users, a population group that is often underrepresented in evaluation. In this presentation we will highlight some of the issues evaluators need to consider when working with this marginalised community, and share practical skills and techniques for making their evaluations more accessible.

Authors: Skye Trudgett (she/her) - Kowa, AU, Katie Stubley (she/her) - Griffith University, AU, Terri Reid (she/her), Chloe Wegener(she/her), Banok Rind (she/her), Sophie Spry (she/her), Niamh Kealy (she/her)- Wiyi Yani U Thangani Institute for First Nations Gender Justice, AU

Measurement and data does not need to be all about numbers—it is about our heart and spirits, it is about voice, story, emotion—it is about truth. Numbers can tell us all sorts of lies, Wiyi Yani U Thangani is and always has been about—your voice—what you are saying about your lives, how you see your future and what matters to you.' June Oscar AO, Chair of Wiyi Yani U Thangani Institute.
Quantitative data often dominate measurement and evaluation, yet true understanding requires tapping into the heart and spirit of communities. As June Oscar AO poignantly reminds us, it is the voice, story, emotion, and truth that bring depth to data. This session at the AES conference offers a hands-on experience that transcends traditional data collection, engaging participants in co-creating collaborative Impact Yarns through deep listening, yarning, and creative expression.
Join us in a dynamic workshop where sculpture and artwork become powerful tools for storytelling and knowledge sharing. Participants will learn to capture the nuanced experiences of First Nations communities, reflecting on how these creative practices can reveal the interconnectedness of our lives and contribute to systemic change. By integrating Indigenous methodologies, we will collectively explore the rich, qualitative data that emerges from individuals' lived realities and aspirations.
As we craft and shape our narratives, we will reflect on how these stories can inform and transform policies and initiatives. This immersive session is not just about creating art; it is about embodying the principles of gender justice and equality, respecting cultural heritage, and acknowledging the diverse ways communities envision their future.
Experience the power of collaborative creation, where every word spoken, and every form sculpted enriches our collective understanding of impact. This workshop is an invitation to step away from the spreadsheet and into a space where every voice contributes to a tapestry of change. Come, let us shape a more empathetic and embracing approach to measurement—one that values the stories and truths of all peoples. truths of all peoples.

Authors: Kate Nichols (Department of Jobs, Skills, Industry and Regions - Victoria), Milbert Gawaya (Department of Jobs, Skills, Industry and Regions, AU)

First Peoples/settler state treaties have been likened to marriage - an evolving and changeable (political) relationship, not an endpoint or divorce (Blackburn, 2007). But what does this look like in practice given marriage's checkered connection with power imbalance and violence through to romance and deep, trusting companionship?

Contemporary colonial 'settlerism' (after Aunty/Dr Lilla Watson, in Watego, 2021) is undergoing structural change in Victoria, with Victoria's First Peoples sitting down with the Victorian State Government in 2024 to commence statewide treaty negotiations. Treaty is an acknowledgement that British sovereignty did not extinguish Aboriginal sovereignty, opening-up a "third space of sovereignty" (after Bruyneel, 2007) where co-existing sovereigns can further contest the "sovereignty impasse" (ibid., 2007), while Indigenous people control their own affairs.

Treaty is expected to reshape how the Victorian state government operates, challenging state laws, institutions, policies, programs and processes, which together, have contributed to First Nations disadvantage and suffering. Government evaluation practices will need their own shake-up.

How can public sector evaluators help establish an equal, strong and nourishing treaty marriage? This short paper shares emerging ally insights into how local practices are evolving to support Victoria's Treaty and self-determination. It shares reflections from a recent evaluation of Traditional Owner grant programs, conducted in partnership between key Aboriginal and non-Aboriginal public sector staff. It is a story of both-ways practice and the time, trust and bravery required to achieve deep change. It also highlights the role of lifelong cultural learning and behaviour change for ally evaluators. Culturally responsive evaluation, Indigenous research practices, restorative justice and the AES First Nations Cultural Safety Framework provide useful framing. Although focused on the Victorian treaty context, the paper may be transferable to other jurisdictions and evaluations involving or impacting Aboriginal and Torres Strait Islander peoples in support of their sovereignty and self-determination.
ion trainers and facilitators.

Authors: Kristy Hornby (Grosvenor)

There is a lot of interest in rapid M&E planning and rapid evaluations at present; partially borne out of rapid contexts in a COVID-19 policy context; and partially borne out of constricting appetites for time and money spent on evaluations. It is unlikely this trend will reverse in the short-term, so what do we do about it to acquit our responsibilities as evaluators, ethically and appropriately, in a rapid context? This session sets out a step by step approach to conducting a rapid evaluation, inviting attendees to follow along with their own program in mind, to come away from the session with a pathway for conducting their own rapid evaluation. The session uses a fictional case study as the construct to move the rapid evaluation approach forward, describing throughout the session how you can use literature reviews, qualitative and quantitative data collection and analysis techniques, and report writing approaches innovatively to save you time while not compromising rigour.

We contend it is possible to do a rapid evaluation ethically and appropriately, but the backbone of doing so is good planning and execution. This session shares practical tips and approaches for doing so through each key phase of an evaluation, so attendees are well-equipped for their next rapid evaluation.

To consolidate the learning, attendees will be provided a framework to come away from the session with a high level plan of how to conduct their own rapid evaluation, to increase their chance of success.

Authors: Eleanor Kerdo (Attorney Generals Department ),Claudia Oke (Attorney Generals Department, AU),Michael Amon (Attorney Generals Department, AU),Anthony Alindogan (Attorney Generals Department, AU)

To embed a culture of evaluation across the Australian Public Service (Commonwealth of Australia, 2021), we must first have an accurate understanding of the current state of evaluation capability and priorities across Commonwealth agencies. This paper shares tools on how to build an effective measurement framework for evaluation culture, and discusses how to use these for evaluation capability uplift.
We explore quantitative and qualitative methods to gather and analyse data to measure an organisation's readiness to change its culture towards evaluation. This includes assessing staff attitudes towards evaluation, the level of opportunity for staff to conduct and use evaluation, and confidence in their knowledge of evaluation.
We discuss the development of a staff evaluation culture survey based on Preskill & Boyle's ROLE and how behavioural insight tools can be utilised to boost engagement. The paper discusses the utility of holding focus groups with senior leaders to understand authorising environments for evaluation and key leverage points. Also discussed, are challenges and innovative solutions that were encountered throughout the assessment process.
This paper will be valuable for those who work in, or with, any government agency with an interest in evaluation capacity building and driving an evaluation culture within organisations. This paper explains each stage of measurement design, data analysis and results, and discussing opportunities for action.
1 Preskill, H., & Boyle, S. (2008). A Multidisciplinary Model of Evaluation Capacity Building. American Journal of Evaluation, 29(4), 443-459. ://journals.sagepub.com/doi/10.1177/1098214008324182

2 Michie S, Atkins L, West R. (2014) The Behaviour Change Wheel: A Guide to Designing Interventions. London: Silverback Publishing. www.behaviourchangewheel.com.

3 Lahey, R. (2009). A Framework for Developing an Effective Monitoring and Evaluation System in the Public Sector: Key considerations from International Experience. Framework for developing an effective ME system in the public sector (studylib.net)

Authors: Skye Trudgett (Kowa), Rachel Kerry (CAGES Foundation, AU)

The philanthropic sector has long grappled with power imbalances inherent in funding relationships, particularly in the context of support for First Nations communities. This panel session explores a groundbreaking Monitoring, Evaluation, and Learning (MEL) approach that inverts traditional power structures, placing First Nations communities in the driver's seat to assess the adherence of a leading philanthropic organisation to their stated values and principles.

Drawing from the collaborative efforts of one foundation and its MEL partner, this session showcases a MEL model that exemplifies shared power and mutual accountability. The panel will consist of thought leaders from the philanthropic sector, First Nations community representatives, and MEL experts who have been at the forefront of developing and implementing this innovative approach.

Through a facilitated discussion, panellists will delve into the process of co-creating a MEL framework that empowers communities to evaluate the performance of philanthropists against a set of mutually agreed-upon criteria. This approach ensures that philanthropic actions align with community expectations, cultural protocols, and contribute to genuine and sustainable impact.

Attendees will gain insights into the challenges and successes of operationalising this community-centric MEL method. The session aims to inspire other philanthropic entities to reflect on their practices and adopt similar approaches that truly shift power to First Nations communities.

Authors: Kizzy Gandy (Verian (formerly Kantar Public) )

As evaluators, one of our key ethical responsibilities is to not cause psychological harm or distress through our methods. We often start workshops or interviews with a warning that the topic may be upsetting and provide contact information for mental health services to participants under the assumption this is the most ethical practice.

Trigger warnings are used with good intentions and are often recommended in evaluation ethics guidelines. However, what do we know about their impact? Is there a risk they actually trigger people more?

This talk examines the evidence on whether trigger warnings are an effective strategy for reducing the risk of trauma and re-traumatisation when discussing topics such as sexual assault, mental health, violence, drug use, and other sensitive issues. It also touches on new evidence from neuroscience about how emotions are understood differently now compared to in the past.

This session will not provide a definitive answer on when or how to use trigger warnings but aims to challenge the audience to think critically about whether trigger warnings are useful in their own work.

Authors: Candice Butler (Queensland Aboriginal and Torres Strait Islander Child Protection Peak ),Michelle McIntyre (Queensland Aboriginal and Torres Strait Islander Child Protection Peak, AU),John Prince (JKP Consulting, AU)

There is increasing recognition that evaluations of Aboriginal and Torres Strait Islander programs must be culturally safe and appropriate, and represent the worldviews, priorities, and perspectives of Aboriginal and Torres Strait Islander communities. Aboriginal and Torres Strait Islander peoples have the cultural knowledge and cultural authority to design appropriate evaluations that are safe, and that tell the true story of the impacts of our ways of working.

As a peak body for Aboriginal and Torres Strait Islander community-controlled organisations we wanted to ensure that the worldviews and perspectives of our members and communities are embedded in any evaluations of services delivered by our member organisations. This is a necessary step towards building an evidence base for our ways of working, developed by and for Aboriginal and Torres Strait Islander people. To that end we developed an evaluation framework to enable self-determination and data sovereignty in evaluation, and to build capacity among our member organisations to undertake and/or commission culturally grounded evaluations. Culturally grounded evaluations are led by Aboriginal and Torres Strait Islander people and guided by our worldviews and knowledge systems - our ways of knowing, being and doing.

This paper reports on the development and implementation process used in the project and describes the standards and principles which underpin the framework. An example of how the framework is being applied in practice is also outlined. Our principles for evaluation describe the core values which underpin culturally grounded and safe evaluation including self-determination; cultural authority; truth-telling; two-way learning; and holistic approaches. The evaluation standards and associated elements operationalise our principles and embed them in evaluative practice.