Loading…
Conference hashtag #aes24MEL
arrow_back View All Dates
Wednesday, September 18
 

9:00am AEST

Opening plenary: Welcome to Country followed by June Oscar "Wiyi Yani U Thangani - re-imagining evaluation with a gender justice lens"
Wednesday September 18, 2024 9:00am - 10:30am AEST
Welcome to Country & Smoking Ceremony: Wurundjeri Elder Perry Wandin
Opening address: Kiri Parata, President, Australian Evaluation Society

Keynote address: "Wiyi Yani U Thangani - re-imagining evaluation with a gender justice lens"

June Oscar AO

As the first woman to be the Aboriginal and Torres Strait Islander Social Justice Commissioner, June led the ground-breaking national project, Wiyi Yani U Thangani (Women’s Voices). In March 2024, the Wiyi Yani U Thangani Institute for First Nations Gender Justice was launched at the Australian National University to carry the legacy of the project—which has gathered a powerful evidence base of the rights, issues and aspirations of thousands of First Nations women and girls.
The Institute is developing an applied measurement, evaluation and learning approach formed by the voices and cultures of First Nations women and girls. This approach will guarantee that women can own their own evidence and identify areas of research which respond to their strengths and priorities. June’s address will explore this unique approach and how evaluation methods shaped by women’s voices can lead to systemic transformation benefiting all Australians.



Chair
avatar for Jess Dart

Jess Dart

Chief Evaluator and Founder, Clear Horizon Consulting
Dr Jess Dart is the founder and Chief Evaluator of Clear Horizon, an Australian-based specialist evaluation company. Having received the 2018 Outstanding Contribution to Evaluation Award from the Australian Evaluation Society (AES), Jess is a recognised leader with over 25 years of... Read More →
Speakers
avatar for June Oscar AO

June Oscar AO

Inaugural Chair, The Wiyi Yani U Thangani Institute for First Nations Gender Justice
June Oscar AO is a proud Bunuba woman from the remote town of Fitzroy Crossing in Western Australia’s Kimberley region. She is a strong advocate for Indigenous Australian languages, social justice, women’s issues, and has worked tirelessly to reduce Fetal Alcohol Spectrum Disorder... Read More →
avatar for Kiri Parata

Kiri Parata

President, Australian Evaluation Society
Kia ora, greetings allI'm always excited to attend AES conferences and the Wayfinding theme of aes24 speaks to my heritage and culture. This year with my family I holidayed in beautiful Rarotonga in the Cook Islands. We visited the site where the waka (Māori watercraft/vessel) of... Read More →
Wednesday September 18, 2024 9:00am - 10:30am AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

Evaluation that adds value for People and Planet: Perspectives, Challenges, and Opportunities for Indigenous Knowledge Systems in Africa.
Wednesday September 18, 2024 11:00am - 11:30am AEST
104
Authors: Awuor PONGE (African Policy Centre (APC) )

Indigenous Knowledge Systems (IKS) in Africa have long been marginalized and undervalued, despite their potential to offer sustainable solutions to pressing challenges faced by communities across the continent. This presentation explores the perspectives, challenges, and opportunities for incorporating IKS into evaluation practices that create value for both people and the planet.

From a people-centric perspective, IKS offer a holistic and culturally relevant approach to understanding local contexts, priorities, and value systems. By embracing these knowledge systems, evaluations can better capture the multidimensional nature of well-being, including spiritual, social, and environmental aspects that are often overlooked in conventional evaluation frameworks. However, challenges arise in reconciling IKS with dominant Western paradigms and navigating power dynamics that have historically suppressed indigenous voices.

From a planetary perspective, IKS offer invaluable insights into sustainable resource management, biodiversity conservation, and climate change adaptation strategies that have been honed over generations of lived experiences. Integrating these knowledge systems into evaluation can shed light on the intricate relationships between human activities and ecosystem health, enabling more informed decision-making for environmental sustainability. Nonetheless, challenges exist in bridging the divide between traditional and scientific knowledge systems, as well as addressing concerns around intellectual property rights and benefit-sharing.

This presentation will explore innovative approaches to overcoming these challenges, such as participatory and community-based evaluation methodologies, capacity-building initiatives, and cross-cultural dialogue platforms. By fostering a deeper appreciation and understanding of IKS, evaluation practices can become more inclusive, relevant, and effective in creating value for both people and the planet in the African context.


Chair
avatar for Alice Muller

Alice Muller

Senior Monitoring & Evaluation Advisor: FMNR Scale Up, World Vision Australia
An environmental scientist, working in international development, interested in evaluation and learning about all things community, trees, ecosystem restoration, climate action, scaling and systems transformation.  I also really like coffee and chatting about gardening, travel and... Read More →
Speakers
avatar for Awuor Ponge

Awuor Ponge

Senior Associate Research Fellow, African Policy Centre (APC)
Dr. Awuor Ponge, is a Senior Associate Fellow, in-charge of Research, Policy and Evaluation at the African Policy Centre (APC). He is also the Vice-President of the African Evaluation Association (AfrEA). He holds a Doctor of Philosophy (PhD) Degree in Gender and Development Studies... Read More →
Wednesday September 18, 2024 11:00am - 11:30am AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

Appreciating First Nations voices: Using appreciative inquiry and participation in the evaluation of Community Justice Groups
Wednesday September 18, 2024 11:00am - 12:00pm AEST
Authors: Michael Limerick (Myuma Pty Ltd ),Melinda Mann (Myuma Pty Ltd, AU),Melissa Osborn (Myuma Pty Ltd, AU)

Emerging best practice principles for Indigenous evaluations encourage evaluators to find new ways of conducting evaluations of programs delivered in First Nations settings. The impetus for this work is a growing awareness that evaluation activity carries the risk of perpetuating colonising impacts on First Nations people, especially in relation to the sovereignty over knowledge and data, the level of consent and self-determination in the process, the level of appreciation of cultural insights and community strengths, and the sharing of the benefits of evaluation activity. For the evaluation of the Community Justice Group (CJG) Program in Queensland, the Department of Justice & Attorney General engaged our organisation, an Aboriginal social enterprise from Queensland, to deliver an evaluation guided by best practice Indigenous evaluation principles. Encouraged by the Department's evaluation brief, our organisation assembled a team of predominantly Indigenous people with deep community connections to facilitate a strengths-based and collaborative approach that would put First Nations voices and perspectives at the centre of the evaluation. Over three years, the team followed a process of working with CJG staff and members to co-design and deliver place-based 'local evaluations' in 25 locations, as the central feature of the Statewide program evaluation. The goal was to 'walk alongside' CJGs to respect their agency and afford them growth opportunities, and to seek out stories of success rather than evidence of deficit. Working in partnership, our organisation and the Department learned much on this journey. Fully implementing Indigenous ethical evaluation principles was not without its challenges - for example, meaningful participation can only occur by relationship-building that takes time and stretches evaluation budgets, and principles such as Indigenous data sovereignty can be difficult to implement in government contexts. However, the value of the approach is evident in firstly, the way that many CJGs embraced the local evaluations, and secondly, in the powerful qualitative evidence of program success yielded by the Appreciative Inquiry-inspired storytelling methods.
Chair
avatar for Carol Vale

Carol Vale

CEO & Co-founder, Murawin, Murawin
Carol Vale is a Dunghutti entrepreneur, businesswoman, CEO and co-founder of Murawin, who’s passion, determination and commitment have driven her impressive 40-year career as a specialist in intercultural consultation, facilitation, and participatory engagement, and an empathetic... Read More →
Speakers
avatar for Michael Limerick

Michael Limerick

Lead Consultant, Myuma
Dr Michael Limerick is a Brisbane-based consultant and lawyer specialising in Indigenous governance and policy.  He is Lead Consultant for the research and evaluation arm of Aboriginal social enterprise, Myuma Pty Ltd, and an Adjunct Associate Professor at the Institute for Social... Read More →
avatar for Melinda Mann

Melinda Mann

Academic Lead Jilbay First Nations RHD Academy, CQUniversity
Melinda Mann is a Darumbal and South Sea Islander woman based in Rockhampton, Qld. Her work focuses on Indigenous Nation building, Pacific sovereignties, and regional and rural communities. Melinda has a background in student services, learning design, school and tertiary education... Read More →
Wednesday September 18, 2024 11:00am - 12:00pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

Design-stage evaluative thinking: helping NGOs and grant makers learn to love evaluation from the start
Wednesday September 18, 2024 11:00am - 12:00pm AEST
103
Authors: Claire Grealy (Rooftop Social ),Duncan Rintoul (Rooftop Social, AU),Virginia Poggio (Paul Ramsay Foundation, AU),Luciana Campello (NSW Department of Communities and Justice, AU),Kirsty Burow (NSW Department of Communities and Justice, AU),Jacqueline Webb (National Association for the Prevention of Child Abuse and Neglect (NAPCAN), AU)

The evaluation of grant programs has long frustrated grantees and perplexed fund managers.
Evaluators often arrive at the end, and may find a strong narrative about the funded activity (assuming the project staff are still in place) but less of the documentation and data that demonstrates the impact or learning, or shows the link between each project to the fund objectives.

Fund managers have often had to be content with the limited results available to them, sometimes as basic as acquittals on activity and expenditure. This limits funders' ability to capture learning, feed into new fund designs, mount budget bids, or tell a compelling story about the work grant holders are doing.

This panel brings together a cross-section of key players and beneficiaries from a variety of contexts:
* a state government fund manager in the human services sector
* an evaluation lead from a large national philanthropic organisation
* an experienced project manager from a national NGO that receives grants from various sources
* two evaluation specialists who have deep experience working in this space, developing and delivering this kind of support.

Drawing on case studies from practice, this panel will share some innovative approaches from their work, which bring the right mix of expectation and support to the design stage of grant-based projects, from the time of submitting an EOI through to the point of evaluation readiness.

The fruit that hangs off this tree includes:
* strengthening the 'evaluability' of each project and the overall fund
* testing each project's assumptions and ambitions
* deep conversations between grant makes and grant holders about outcome alignment
* building the evaluative thinking and capability of project teams and organisations, activating the 'ripple effect' as participants share their newfound commitment and skills with their colleagues.
"You couldn't drag me to program logic workshop before this. And now look at me - I took that process you did with us and yesterday I ran it with my team on another project."
Chair
avatar for Christina Kadmos

Christina Kadmos

Principal, Kalico Consulting
Speakers
avatar for Duncan Rintoul

Duncan Rintoul

Managing Director, Rooftop Social
ECB devotee, mentor in the AES group mentoring program, used to be on the AES board, run a rad consulting firm that specialises in evaluation, lifelong learner. Keeping busy doing research and evaluation and facilitation work in education and justice and sustainability and health... Read More →
avatar for Claire Grealy

Claire Grealy

Director, Rooftop Social
So looking forward to AES 2024! We are Silver Sponsors this year, which means we're keeping your devices charged up through the conference, and you'll find us next to the charging stations. I welcome any and all conversation about evaluation, strategy and design, research, facilitation... Read More →
avatar for Jacqueline Webb

Jacqueline Webb

Strategic Projects Manager, NAPCAN
As Strategic Projects Manager at NAPCAN, I am leading an important DCJ grant initiative aimed at enhancing NSW workforce capabilities to support children and young people affected by sexual violence. With guidance from Rooftop Social, we’ve adopted an innovative evaluation approach... Read More →
avatar for Virginia Poggio

Virginia Poggio

MERL Associate, Paul Ramsay Foundation
As a Measurement, Evaluation, Research, and Learning (MERL) Associate at the Paul Ramsay Foundation, I lead teams to deliver evidence-based advice to inform the Foundation’s strategic initiatives. My role involves commissioning, supporting, and managing independent evaluations of... Read More →
avatar for Luciana Campello

Luciana Campello

Senior Policy and Projects Officer, NSW Department of Communities and Justice
Wednesday September 18, 2024 11:00am - 12:00pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

The psychology of evaluation capacity building: Finding the way with the rider, elephant and the pathway
Wednesday September 18, 2024 11:00am - 12:00pm AEST
106
Authors: Samantha Abbato (Visual Insights People )

The psychology of evaluation capacity building: Finding the way with the rider, elephant and the pathway
Evaluation capacity building is increasingly becoming a core part of evaluation practice and a critical part of incorporating evaluation into the everyday activity of organisations (Preskill and Boyle, 2008, White, Percy and Small, 2018). Reaching the point where evaluation becomes the way of doing business requires a change of knowledge, skills, and attitudes.

Changes need to happen at the level of individuals, teams, organisations, and partnerships. This journey requires supporting and managing change to systematic enquiry processes as much as it requires evaluation expertise. In this skill-building session, we introduce Jonathan Haidt's 'rider, elephant and pathway' metaphor as a framework to support change and strengthen evaluation capacity (Haidt, 2018).

Haidt's metaphor for change includes the rider (our rational thinking side) atop an elephant (our emotional side). Behaviour change for individuals and collectives requires steps that (1) support the rider, such as giving clear directions, (2) motivate the elephant by tapping into emotions, and (3) shape a pathway to change, including clearing obstacles. In this interactive session, the facilitator will provide case studies applying Haidt's metaphor,spanning two decades Through these examples the power of this framework to support evaluation capacity building is demonstrated. Examples include using Haidt's framework for:
1. Building a Monitoring, Evaluation and Learning (MEL) system with a medium-sized community organisation;
2. Increasing the maturity of MEL in an existing large organisation; and
3. Increasing the impact of evaluation partnerships.

The active skill-building component incorporates:_
  • Cartoon elephant, rider and pathway flashcards;
  • A 'snakes and ladders' style game; and
  • Evaluation-specific examples.

The combination of examples and activities are designed to support participant learning. The session will encourage discussion of barriers, enablers and actions to build evaluation capacity relevant to different situations and contexts.

Learning objectives include:
  • Knowledge of a sound and memorable psychological framework for supporting evaluation capacity building;
  • Ability to apply Haidt's metaphor
Chair
avatar for Anthea Rutter

Anthea Rutter

Research Fellow, Centre for Program Evaluation. The University of Melbourne
Anthea Rutter is a Senior Research Fellow in the Assessment and Evaluation Research Centre (formerly the Centre for Program Evaluation) at The University of Melbourne. She has extensive experience working with a wide range of community, state and national organisations. She is particularly... Read More →
Speakers
avatar for Samantha Abbato

Samantha Abbato

Director, Visual Insights People
My twenty-plus years of evaluation experience are built on academic training in qualitative and quantitative disciplines, including mathematics, health science, epidemiology, biostatistics, and medical anthropology. I am passionate about effective communication and evaluation capacity-building... Read More →
Wednesday September 18, 2024 11:00am - 12:00pm AEST
106 102 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

Innovating Value for Money: Finding Our Way to Greater Value for All
Wednesday September 18, 2024 11:00am - 12:00pm AEST
105
Authors: John Gargani (Gargani + Co ),Julian King (Julian King & Associates, NZ)

In this participatory session, we pose the question, "How should evaluators innovate the practice of value-for-money assessment to meet the needs of an expanding set of actors that include governments, philanthropists, impact investors, social entrepreneurs, program designers, and Indigenous and First Nations communities?" We begin by framing value for money as an evaluative question about an economic problem. How well are we using resources, and are we using them well enough to justify their use? Then we suggest new methods intended to help innovate the practice of value for money based on our body of published and current research spanning over 10 years.
These include new methods that (1) produce "holistic" assessments of value for money, (2) reflect rather than hide multiple value perspectives even when values conflict, (3) estimate social benefit-cost ratios without monetizing benefits or costs, and (4) adjust monetary and nonmonetary value for risk using Bayesian methods. Along the way, we facilitate discussions with participants, asking them to consider if, how, and by whom these innovations should be pursued, and what other innovations may be needed. We provide participants with access to a collection of our published and draft papers, and invite them to comment and continue our discussion after the conference.
Chair
avatar for Jade Maloney

Jade Maloney

Partner & CEO, ARTD Consultants
I work with government agencies, not-for-profits and citizens to co-design, refine, communicate and evaluate social policies, regulatory systems and programs. I am passionate about ensuring citizens have a voice in shaping the policies that affect their lives, translating research... Read More →
Speakers
avatar for Julian King

Julian King

Director, Julian King & Associates
I’m an independent public policy consultant based in Auckland. I specialise in evaluation and Value for Investment. I’m affiliated with the Kinnect Group, Oxford Policy Management, the University of Melbourne and the Northern Institute. Subscribe to my weekly blog at https:/... Read More →
avatar for John Gargani

John Gargani

President (former President of the American Evaluation Association), Gargani + Company
Dr John Gargani is an evaluator with 30 years of experience and eclectic interests. He is President of the evaluation consulting firm Gargani + Company, served as President of the American Evaluation Association in 2016, coauthored the book Scaling Impact: Innovation for the Public... Read More →
avatar for Farida Fleming

Farida Fleming

Evaluation Principal, Assai
I'm an evaluator with over 25 years of experience in international development. I'm currently one of a core team supporting DFAT implement its Evaluation Improvement Strategy.
Wednesday September 18, 2024 11:00am - 12:00pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

11:00am AEST

Planning your conference voyage: Key evaluation concepts for novice sailors
Wednesday September 18, 2024 11:00am - 12:30pm AEST
Authors: Charlie Tulloch (Policy Performance)

Many people will be attending their first AES conference - welcome!
This workshop is targeted at new and emerging evaluators who are seeking to build their familiarity with the key concepts and language used in evaluation and the main elements of evaluation project delivery.
This will provide you with a plotted history of the evaluation field so you feel more comfortable engaging in deeper-dive topics over coming days. The session is grounded in theory, drawing on leading thinkers and methods that are central to our practice.
Key concepts to be covered include: what is evaluation? why should we evaluate? what can be evaluated? when to evaluate? how to evaluate? where can I learn more about evaluation?
The forum will help many brave young and emerging evaluators to navigate through often choppy evaluation waters so they don't feel 'all at sea' over coming days.
Chair Speakers
avatar for Charlie Tulloch

Charlie Tulloch

Director, Policy Performance
Policy Performance is a proud conference sponsor! Charlie delivers evaluation projects, capability building support and drives public sector improvement. Charlie loves to help those who are new to evaluation or transitioning from related disciplines. He is a past AES Board member... Read More →
Wednesday September 18, 2024 11:00am - 12:30pm AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia

11:30am AEST

Culturally Responsive Initiatives: Introducing the First Nations Investment Framework
Wednesday September 18, 2024 11:30am - 12:00pm AEST
104
Authors: Eugenia Marembo

Representatives of First Nations communities have been advocating for changes in the way initiatives are planned, prioritised, and assessed. This includes greater visibility on where funding is going, more partnerships on designing initiatives and more evaluation on the outcomes being achieved, to inform government decision making.

This paper presents key insights on what constitutes good practice when designing and appraising initiatives that affect First Nations people and communities. The National Agreement on Closing the Gap is built around four new Priority Reforms that will change the way governments work with Aboriginal and Torres Strait Islander people and communities. Priority Reform Three is about transforming government institutions and organisations. As part of this Priority Reform, parties commit to systemic and structural transformation of mainstream government organisations to improve accountability, and to respond to the needs of First Nations people.

The findings presented in this paper draw on insights from consultations with various First Nations community representatives and government stakeholders in New South Wales, and the subsequent process of developing a government department's First Nations investment framework which seeks to strengthen the evidence on what works to improve outcome for First Nations people. Additionally, the frameworks to improve practice across government processes and better inform how initiatives are designed, prioritised and funded.
Chair
avatar for Alice Muller

Alice Muller

Senior Monitoring & Evaluation Advisor: FMNR Scale Up, World Vision Australia
An environmental scientist, working in international development, interested in evaluation and learning about all things community, trees, ecosystem restoration, climate action, scaling and systems transformation.  I also really like coffee and chatting about gardening, travel and... Read More →
Speakers
avatar for Steven Legg

Steven Legg

Associate Director, NSW Treasury
avatar for Eugenia Marembo

Eugenia Marembo

NSW Treasury, Senior Analyst, First Nations Economic Wellbeing
Wednesday September 18, 2024 11:30am - 12:00pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

12:00pm AEST

From bottlenecks to breakthroughs: Insights from a teacher workforce initiative evaluation
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
104
Authors: Rhiannon Birch (Victorian Department of Education ),Hayden Jose (Urbis, AU),Joanna Petkowksi (Victorian Department of Education, AU),Ekin Masters (Victorian Department of Education, AU)

How can an evaluation balance the need to generate pragmatic insights while meeting central agency requirements for rigorous measurement of outcomes? What ingredients can facilitate the effective evaluation of a government initiative and achieve improved outcomes? This paper explores the essential ingredients for evaluating a large-scale government program using an example of a statewide initiative aimed at attracting and retaining suitably qualified teachers in hard-to-staff positions in Victorian government schools.

We showcase how an adaptive and evidence-led method of enquiry helped identify program implementation bottlenecks and probe potentially unintended program outcomes over a three-year evaluation. We discuss enablers for the integration of evaluation recommendations into program implementation and future policy direction, particularly on participatory action approaches and deep relationships with policy and implementation teams. We will also present the robust and varied methodology, particularly the novel use of system data to facilitate a quasi-experimental design that aligned with central agency requirements and met stakeholder needs.
This presentation will benefit policymakers, program evaluators, and others interested in evaluating government programs, by sharing key learnings on how evaluations can balance pragmatic insights with central agency requirements and identifying the key elements for influencing such programs and achieving improved outcomes.
Chair
avatar for Alice Muller

Alice Muller

Senior Monitoring & Evaluation Advisor: FMNR Scale Up, World Vision Australia
An environmental scientist, working in international development, interested in evaluation and learning about all things community, trees, ecosystem restoration, climate action, scaling and systems transformation.  I also really like coffee and chatting about gardening, travel and... Read More →
Speakers
avatar for Rhiannon Birch

Rhiannon Birch

Senior Evaluation and Research Officer, Department of Education
Rhiannon is a dedicated research and evaluation specialist committed to enhancing health, social, education, and environmental outcomes for people and the planet. With over 10 years of experience in evaluation, she has worked extensively across emergency services, public health, and... Read More →
avatar for Hayden Jose

Hayden Jose

Associate Director, Urbis
Hayden brings 13 years’ experience as an evaluator, applied researcher and policy practitioner with extensive work in complex evaluations in government and not-for-profit settings. Across his roles, he has worked to consider complex system problems and translate evidence effectively... Read More →
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

12:00pm AEST

Warlpiri ways of assessing impact - How an Aboriginal community is defining, assessing and taking action for a good life in their community.
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
Authors: Emily Lapinski (Central Land Council ),Malkirdi Napaljarri Rose (Centre For Human Security and Social Change, La Trobe University, AU), Glenda Napaljarri Wayne (Central Land Council, AU), Geoffrey Jungarrayi Barnes (Central Land Council, AU), Alex Gyles (Centre For Human Security and Social Change, La Trobe University, AU)

For evaluation to support transformational change, research suggests strategies must focus on localised Indigenous values, beliefs and worldviews. Decolonising evaluation involves identifying and addressing power and considering what is being evaluated, by whom and how. In this paper we argue that these developments are necessary but insufficient and suggest a possible way forward for further decolonising the field of evaluation. To support change for Indigenous Australians the emphasis needs to move from simple evaluation of individual programs to more critical examination of their combined impact on communities from local perspectives.

This paper explores how Warlpiri and non-Indigenous allies are collaborating to create and use their own community-level impact assessment tool. The 5-year Good Community Life Project is supporting Warlpiri residents of Lajamanu in the Northern Territory to define, assess and take action for a 'good community life'. Warlpiri will explain how they created the approach for assessing wellbeing in Lajamanu, and how they are using emerging results to give voice to their interests and advocate for the life they envision for future generations.

The project involves collaboration between Warlpiri community members, land council staff and university researchers, drawing on Indigenous concepts of 'two-way' seeing and working, relationality, and centring Indigenous voice and values. Applying these concepts in practice is challenging, particularly for non-Indigenous allies who must constantly reflect and use their privilege to challenge traditional views on 'robust' evaluation methodology.

Warlpiri and the land council see potential for this work to improve life in Lajamanu and as an approach that could be applied across Central Australian communities. Going beyond co-designed and participatory evaluation to critical examination of impact is the next step in supporting change with Indigenous communities. This paper will focus on Warlpiri perspectives, plus brief reflections from non-Indigenous allies, with time for the audience to discuss broader implications.
Chair
avatar for Carol Vale

Carol Vale

CEO & Co-founder, Murawin, Murawin
Carol Vale is a Dunghutti entrepreneur, businesswoman, CEO and co-founder of Murawin, who’s passion, determination and commitment have driven her impressive 40-year career as a specialist in intercultural consultation, facilitation, and participatory engagement, and an empathetic... Read More →
Speakers
EL

Emily Lapinski

Monitoring, Evaluation and Learning Coordinator, Central Land Council
avatar for Alex Gyles

Alex Gyles

Research Fellow - Monitoring and Evaluation, Institute for Human Security and Social Change, La Trobe University
Alex Gyles is a Research Fellow working in Monitoring, Evaluation and Learning (MEL) at the Institute for Human Security and Social Change, La Trobe University. He works closely with Marlkirdi Rose Napaljarri on the YWPP project and finds fieldwork with the YWPP team an exciting learning... Read More →
GN

Glenda Napaljarri Wayne

Glenda Wayne Napaljarri is a community researcher on the YWPPproject from Yuendumu. She has developed her practice workingas an adult literacy tutor in Yuendumu’s Community LearningCentre. In addition to conducting research in her home communityof Yuendumu, Glenda has travelled... Read More →
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

12:00pm AEST

Navigating collaborative practice - the role of evaluation in brokering shared outcomes
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
106
Authors: Caroline Crothers (Allen + Clarke Consulting)

A collaborative effort between community organisations and Victoria Police has demonstrated significant impact in addressing youth offending in Victoria's northwest metropolitan region. This initiative brought together 12 partner organisations from various sectors, including police, legal services, and youth support services around the shared goal of reducing youth offending. By diverting young offenders from the criminal justice system, the initiative seeks to enhance overall justice, health, and wellbeing outcomes for vulnerable youth. Allen + Clarke was commissioned to evaluate the success of this initiative during its inaugural year. In this presentation, we share key lessons learned from the evaluation including how minimally resourced and small-scale interventions can have an outsized impact on organisational change to culture and practice. We also reflect on the journey embarked upon and explore how the evaluation process itself serves as a tool for navigating through complex challenges and adapting to changes encountered along the way. Through critical reflection, the presentation delves into the differing perspectives of the delivery partners involved highlighting how the evaluation journey facilitates a shared understanding of the path forward and shaping future strategies and interventions.
Chair
avatar for Anthea Rutter

Anthea Rutter

Research Fellow, Centre for Program Evaluation. The University of Melbourne
Anthea Rutter is a Senior Research Fellow in the Assessment and Evaluation Research Centre (formerly the Centre for Program Evaluation) at The University of Melbourne. She has extensive experience working with a wide range of community, state and national organisations. She is particularly... Read More →
Speakers
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
106 102 Convention Centre Pl, South Wharf VIC 3006, Australia

12:00pm AEST

Enhancing Stakeholder Engagement Through Culturally Sensitive Approaches: A Focus on Aboriginal and Torres Strait Islander Communities
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
105
Authors: Mark Power (Murawin ),Carol Vale (Murawin, AU)

This presentation explores the paramount importance of culturally sensitive engagement methodologies in ensuring meaningful contributions from Aboriginal and Torres Strait Islander communities to mission programs. Murawin, an Aboriginal-led consultancy, has developed a robust Indigenous Engagement Strategy Framework grounded in the principles of reciprocity, free, informed and prior consent, mutual understanding, accountability, power sharing, and respect for Indigenous knowledge systems. Our session aims to share insights into the necessity of prioritising Aboriginal and Torres Strait Islander voices in engagement, co-design, and research, highlighting the significance of cultural competence in fostering mutual respect and understanding.
We will discuss three key messages: the imperative of deep knowledge and understanding of Aboriginal and Torres Strait Islander cultures in engagement practices; the success of co-design processes in facilitating genuine and respectful engagement; and the strategic partnership with CSIRO to enhance cultural competence and inclusivity in addressing Indigenous aspirations and challenges. These points underscore the critical role of acknowledging cultural interactions and ensuring cultural sensitivity in building strong, respectful productive relationships with Indigenous communities.
To achieve our session's objectives, we have designed an interactive format that blends informative presentations with the analysis of case studies, complemented by engaging intercultural discussions. This approach is intended to equip participants with actionable insights drawn from real-world examples of our collaborative ventures and co-designed projects. Through this comprehensive exploration, we aim to enrich participants' understanding of successful strategies for engaging Aboriginal and Torres Strait Islander communities, ultimately contributing to the achievement of more inclusive and impactful outcomes in mission programs and beyond.


Chair
avatar for Jade Maloney

Jade Maloney

Partner & CEO, ARTD Consultants
I work with government agencies, not-for-profits and citizens to co-design, refine, communicate and evaluate social policies, regulatory systems and programs. I am passionate about ensuring citizens have a voice in shaping the policies that affect their lives, translating research... Read More →
Speakers
avatar for Carol Vale

Carol Vale

CEO & Co-founder, Murawin, Murawin
Carol Vale is a Dunghutti entrepreneur, businesswoman, CEO and co-founder of Murawin, who’s passion, determination and commitment have driven her impressive 40-year career as a specialist in intercultural consultation, facilitation, and participatory engagement, and an empathetic... Read More →
avatar for Mark Power

Mark Power

Director, Evaluation & Research, Murawin
Mark is an experienced researcher with more than 20 years of experience in Australia and the Pacific. Mark manages Murawin’s evaluation and research practice and leads multiple evaluations for a variety of clients. Mark has overseen more than 30 high-profile, complex projects funded... Read More →
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

12:00pm AEST

Valuing First Nations Cultures in Cost-Benefit Analysis
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
103
Authors: Laura Faulker (NSW Treasury)

This paper presents the key findings from research and engagement on how cost-benefit analysis (CBA) has been applied to First Nations initiatives to date. CBA is an important tool used by governments to help prioritise budget funding decisions. It assesses the potential impacts of an initiative - economic, social, environmental, and cultural - to determine whether it will deliver value for money.

The paper explores the methods in which the value of First Nations cultures has been incorporated into CBAs, along with the associated challenges and opportunities to improve current practice. The findings have informed the development of an investment framework for the design and evaluation of initiatives that affect First Nations people and communities. The framework focuses on the key principles for embedding First Nations perspectives and ensuring culturally informed evaluative thinking.


Chair
avatar for Christina Kadmos

Christina Kadmos

Principal, Kalico Consulting
Speakers
avatar for Laura Faulkner

Laura Faulkner

Senior Analyst, First Nations Economic Wellbeing, NSW Treasury
Wednesday September 18, 2024 12:00pm - 12:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Enhancing evaluation value for small community organisations: A case example
Wednesday September 18, 2024 1:30pm - 2:00pm AEST
104
Authors: Stephanie Button (Assessment and Evaluation Research Centre, University of Melbourne), Allison Clarke, Carolyn McSporran (Blue Light Victoria, AU)Authors: Stephanie Button (Assessment and Evaluation Research Centre, University of Melbourne), Allison Clarke (Assessment and Evaluation Research Centre, University of Melbourne, AU), Elissa Scott (Blue Light Victoria, AU)

This presentation aims to provide a case example of how two small-scale, standard process/outcomes evaluations for a low-budget community organisation increased value for the organisation by identifying and seizing opportunities for evaluation capacity building. Formal evaluations represent a significant financial commitment for low-budget community organisations. By maximising the value provided by such evaluations, evaluators can contribute more to these organisations' mission and ultimately to social betterment.

There are numerous different evaluation capacity building models and frameworks, many of which appear to be quite complex (for example: Volkov & King, 2007; Preskill & Boyle, 2008). Many emphasise planning, documentation, and other resource intensive components as part of any evaluation capacity building effort. This session provides a case example of intentional but light-touch and opportunistic evaluation capacity building. Through such an approach, evaluators may need to do only minimal additional activities to provide extra value to an organisation. Reflection-in-action during the evaluation process is as important as the final reporting (Schwandt & Gates, 2021). The session emphasises, though, that a critical enabler will be the organisation's leadership and culture, and willingness to seize the opportunity offered by a formal evaluation. The session is co-presented by two members of the evaluation team and the Head of Strategy, Insights, and Impact of the client organisation.
Chair
avatar for Duncan Rintoul

Duncan Rintoul

Managing Director, Rooftop Social
ECB devotee, mentor in the AES group mentoring program, used to be on the AES board, run a rad consulting firm that specialises in evaluation, lifelong learner. Keeping busy doing research and evaluation and facilitation work in education and justice and sustainability and health... Read More →
Speakers
avatar for Allison Clarke

Allison Clarke

Evaluator
- Allison is passionate about using monitoring and evaluation for organisational learning. She has over 20 years experience in the private and not-for-profit sectors in industrial research, probate research, and program development. She completed her Master of Evaluation at the Centre... Read More →
avatar for Stephanie Button

Stephanie Button

Research Associate & Evaluator, Assessment & Evaluation Research Centre
Stephanie has worked as a policy manager, analyst, strategist, researcher, and evaluator across the social policy spectrum in the public and non-profit sector for over 12 years. She is passionate about evidence-based policy, pragmatic evaluation, and combining rigour with equitable... Read More →
avatar for Carolyn McSporran

Carolyn McSporran

Head of Strategy, Insights and Impact, Blue Light Victoria
Passionate about social inclusion, Carolyn's work has spanned diverse portfolios across the justice and social services sectors. With a fervent belief in the power of preventative and early intervention strategies, she is committed to unlocking the full potential of individuals and... Read More →
Wednesday September 18, 2024 1:30pm - 2:00pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Envisioning and Encountering Relational Aboriginal and Pacific Research Futures
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
105
Authors: Alli Burness (Tetra Tech), Lisa Faerua (Vanuatu), Nathan Sentance (Museum of Applied Arts and Sciences, AU), David Lakisa (Talanoa Consultancy, AU)

In the inaugural ANU Coral Bell Lecture on Indigenous Diplomacy, Dr Mary Graham outlined a powerful legacy of Aboriginal and Torres Strait Islander relational methods that have operated across a spectacular time scale. She envisioned a compelling future for its renewed application and spoke of these practices as a type of "thinking in formation, a type of slow, collective, and emergent process".

Inspired by Dr Graham's vision, this panel explores synergies, distinctions, and complementarities in local and Indigenous research methods across Australia and the Pacific. The panel features Wiradjuri, Samoan (Polynesian), Ni-Vanuatu (Melanesian) and settler-background (Australian) researchers from a range of fields who will explore, engage and showcase locally specific methodologies that connect across Australia and the Pacific continents, as ways of knowing, doing, and relating with the land, the moana (ocean) and air.

This session frames evaluation and research approaches as reflecting their contextual political order. While the panel will critique the legacies of individualist and survivalist research methods, it will focus on exploring the futures that relational research methods could realize. How do we evolve current institutional approaches to become more commensurate with Indigenous methods? Would institutionalizing these methods resolve the legacy, structure, and form of colonialist political approaches? Panelists will speak to their experience in working to evolve institutions in this way and the research and evaluation methodologies used within them.

The session also situates evaluation within a cannon of contextualizing evidence-based practices (such as political economy analysis, GEDSI analysis or feasibility.
Chair
avatar for Martina Donkers

Martina Donkers

Independent Evaluator
I'm an independent freelance evaluator with a background in program design, grants, and science communication. I have a Master of Evaluation, and I'm finding my sweet spot in qualitative and mixed methods evaluation with a complexity and systems lens. I like rubrics, semi-structured... Read More →
Speakers
avatar for Lisa Faerua

Lisa Faerua

Lisa Faerua is a Pacific Freelance Consultant. She brings 17 years of experience in international and community development in the areas of leadership, design, monitoring and evaluation. Lisa has provided technical support to DFAT, MFAT, and Non-Government Organisations such Oxfam... Read More →
avatar for Nathan Sentance

Nathan Sentance

Nathan “mudyi” Sentance is a cis Wiradjuri librarian and museum collections worker who grew up on Darkinjung Country. Nathan currently works at the Powerhouse Museum as Head of Collections, First Nations and writes about history, critical librarianship and critical museology from... Read More →
avatar for David Lakisa

David Lakisa

Managing Director, Talanoa Consultancy
Dr David Lakisa specialises in Pacific training and development, educational leadership and diversity management. He is of Samoan (Polynesian) ancestry and completed his PhD on 'Pacific Diversity Management' at the University of Technology Sydney (UTS) Business School.
avatar for Alli Burness

Alli Burness

Director, Australian Consulting, Tetra Tech
Alli is an Australian strategic designer and researcher with settler heritage, born and living on Bunurong Country. As Director of the Australian Consulting Practice at Tetra Tech International Development, Alli works to address the colonial legacy she has inherited by taking an inclusive... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Learning from failure at a NFP - pitfalls and pointers
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103
Authors: Victoria Pilbeam (WWF-Australia)

Across social and environmental movements, we are often reticent to talk about failure. But as innovation and learning gain greater emphasis across the sector, Not-for Profits are finding new ways to share and learn from their failures (eg: Engineers Without Borders failure reports, Save the Children Fail Fest, etc.). In this presentation, I will share both insights from the available research and reflect on my own journey developing failure programming at WWF-Australia. This presentation will provide practical guidance to evaluators and organisations navigating the challenging terrain of learning from failure.
Chair Speakers
avatar for Victoria Pilbeam

Victoria Pilbeam

MEL Adviser, The Pacific Community (SPC)
At the Pacific Community, I support MEL for fisheries, aquaculture and marine ecosystems across the region. Previously, I worked for WWF-Australia and inconsulting with a range of not-for profit, government , and philanthropic partners. I love MEL that is approachable, equitable... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

No help? No worries! How to find your way on your own
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103
Authors: Rachel Wilks (Grosvenor), Kristy Hornby (Grosvenor)

As we learn the practice of evaluation, we often have experienced evaluators around us. They provide us guidance and guardrails within which to learn and grow our capabilities - but what happens when we no longer have this support? How do we stop ourselves from getting lost?

This session will provide tips and tricks for evaluators going out into the world on their own. Attendees will leave with practical advice and strategies to help keep them on track while they build their autonomy.
Chair Speakers
avatar for Rachel Wilks

Rachel Wilks

Senior Consultant, Grosvenor
Rachel is a management consultant and an emerging evaluator at Grosvenor. She took her first steps into the evaluation world two years ago, and since then has been increasingly interested in how evaluation can be used in and across the public sector and not-for-profit space. Rachel... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Revitalising Survey Engagement: Strategies to Tackle Low Response Rates
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103
Authors: Kizzy Gandy

Surveys are an excellent data collection tool when they reach their target response rate, but low response rates hinder the generalisability and reliability of the findings.

This Ignite presentation will discuss techniques Verian evaluators have applied to increase survey response rates while also assessing the efficacy and efficiency of these techniques. We will also explore other evidence-based strategies for boosting response rates and the value of drawing on other data sources if your response rates are still low.
Chair Speakers
avatar for Hannah Nguyen

Hannah Nguyen

Analyst, Verian Group
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Sign here: Supporting Deaf participation in evaluation
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103
Authors: Susie Fletcher (Australian Healthcare Associates)

Auslan is a visual, signed language that was developed by and for the Australian Deaf community. People who use Auslan as their primary or preferred language are not necessarily fluent in English. Our team was engaged to review access to interpreter services for Auslan users, a population group that is often underrepresented in evaluation. In this presentation we will highlight some of the issues evaluators need to consider when working with this marginalised community, and share practical skills and techniques for making their evaluations more accessible.
Chair Speakers
avatar for Susie Fletcher

Susie Fletcher

Senior consultant, Australian Healthcare Associates
Dr Susie Fletcher is an experienced health services researcher with over 50 peer reviewed journal articles and 3 book chapters in mental health and primary care. She is passionate about improving health outcomes through integrating services across sectors; her recent work has focused... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Beyond Numbers: Weaving Stories, Sculpting Change and Signal Spotting through Collaborative Impact Yarns
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
106
Authors: Skye Trudgett (she/her) - Kowa, AU, Katie Stubley (she/her) - Griffith University, AU, Terri Reid (she/her), Chloe Wegener(she/her), Banok Rind (she/her), Sophie Spry (she/her), Niamh Kealy (she/her)- Wiyi Yani U Thangani Institute for First Nations Gender Justice, AU

Measurement and data does not need to be all about numbers—it is about our heart and spirits, it is about voice, story, emotion—it is about truth. Numbers can tell us all sorts of lies, Wiyi Yani U Thangani is and always has been about—your voice—what you are saying about your lives, how you see your future and what matters to you.' June Oscar AO, Chair of Wiyi Yani U Thangani Institute.
Quantitative data often dominate measurement and evaluation, yet true understanding requires tapping into the heart and spirit of communities. As June Oscar AO poignantly reminds us, it is the voice, story, emotion, and truth that bring depth to data. This session at the AES conference offers a hands-on experience that transcends traditional data collection, engaging participants in co-creating collaborative Impact Yarns through deep listening, yarning, and creative expression.
Join us in a dynamic workshop where sculpture and artwork become powerful tools for storytelling and knowledge sharing. Participants will learn to capture the nuanced experiences of First Nations communities, reflecting on how these creative practices can reveal the interconnectedness of our lives and contribute to systemic change. By integrating Indigenous methodologies, we will collectively explore the rich, qualitative data that emerges from individuals' lived realities and aspirations.
As we craft and shape our narratives, we will reflect on how these stories can inform and transform policies and initiatives. This immersive session is not just about creating art; it is about embodying the principles of gender justice and equality, respecting cultural heritage, and acknowledging the diverse ways communities envision their future.
Experience the power of collaborative creation, where every word spoken, and every form sculpted enriches our collective understanding of impact. This workshop is an invitation to step away from the spreadsheet and into a space where every voice contributes to a tapestry of change. Come, let us shape a more empathetic and embracing approach to measurement—one that values the stories and truths of all peoples. truths of all peoples.
Chair
avatar for Melinda Mann

Melinda Mann

Academic Lead Jilbay First Nations RHD Academy, CQUniversity
Melinda Mann is a Darumbal and South Sea Islander woman based in Rockhampton, Qld. Her work focuses on Indigenous Nation building, Pacific sovereignties, and regional and rural communities. Melinda has a background in student services, learning design, school and tertiary education... Read More →
Speakers
KS

Katie Stubley

Griffith University Centre for Systems Innovation, the Presencing Institute, United in Diversity
avatar for Chloe Wegener

Chloe Wegener

Communications and Impact, Wiyi Yani U Thangani Institute for First Nations Gender Justice
Chloe is a proud Garrwa woman who grew up on Kaurna Country and now lives on Wadawurrung Country. With experience in physiotherapy, community engagement, learning and development and project management, Chloe brings a wide variety of experience and knowledges to the Institute.Chloe... Read More →
BR

Banok Rind

Co-lead for Impact and Engagement, Wiyi Yani U Thangani Institute for First Nations Gender Justice
avatar for Sophie Spry

Sophie Spry

Co-lead, Wiyi Yani U Thangani Institute for First Nations Gender Justice
Since 2020, Sophie has been a Policy Advisor at the Australian Human Rights Commission, working on the landmark Wiyi Yani U Thangani (Women’s Voices) Project, led by previous Aboriginal and Torres Strait Islander Social Justice Commissioner, June Oscar AO. Most recently, she has... Read More →
TR

Terri Reid

Wiyi Yani U Thangani Institute for First Nations Gender Justice:
avatar for Niamh Kealy

Niamh Kealy

Wiyi Yani U Thangani Institute for First Nations Gender Justice:
Niamh has been a member of the Wiyi Yani U Thangani team since 2021, coming on board through the completion of her Masters in Sustainability at the University of Sydney. In 2022, she took a break from the team to join the Australian Human Rights Commission’s Children’s Rights... Read More →
avatar for Skye Trudgett

Skye Trudgett

CEO, Kowa
Skye is a Gamilaroi researcher who has contributed to numerous evaluations and research projects including place-based, systems change and government initiatives. Skye has a particular interest in Indigenous Data Sovereignty & Governance and seeks to support the practical application... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
106 102 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Cultivating Equity: A Roadmap for New and Student Evaluators' Journeys
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
Authors: Ayesha Boyce (Arizona State University), Aileen Reid (UNC Greensboro, US)

Evaluation can be positioned as a social, cultural, and political force to address issues of inequity. We co-direct a 'lab' that provides new evaluators with hands-on applied research and evaluation experience to support their professional development. We are proud of our social justice commitments, and they show up in all aspects of our work. We believe the next generation of evaluators must be trained and mentored in high-quality technical, strengths-based, interpersonal, contextual, social justice-oriented, and values-engaged evaluation. We have found that novice evaluators are able to engage culturally responsive approaches to evaluation at the conceptual level, but have difficulty translating theoretical constructs into practice. This paper presentation builds upon our experiences and previous work of introducing a framework for teaching culturally responsive approaches to evaluation (Boyce & Chouinard, 2017) and a non-course-based, real-world-focused, adaptable training model (Reid, Boyce, et al., 2023). We will discuss how we have taught new evaluators three formal and informal methodologies that have helped them align their values with praxis. Drawing from our work across multiple United States National Science Foundation-funded projects we will overview how the incorporation of photovoice methodology, just-in-time feedback, and reflective practice have supported our commitments to meaningfully, and respectfully attend to issues of culture, race, diversity, power, inclusion, and equity in evaluation. We will also discuss our thoughts on the implications of globalization, Artificial Intelligence, and shifting politics on evaluation capacity building and training of new evaluators.

Chair
avatar for Nick Field

Nick Field

Director (Public Sector), Urbis
Nick has twenty years of public sector consulting experience, backed more recently by six years as a Chief Operating Officer in the Victorian Public Sector. A specialist generalist in a broad range of professional advisory services, Nick has expertise in the implementation of state-wide... Read More →
Speakers
avatar for Ayesha Boyce

Ayesha Boyce

Associate Professor, Arizona State University
Ayesha Boyce is an associate professor in the Division of Educational Leadership and Innovation at Arizona State University. Her research career began with earning a B.S. in psychology from Arizona State University, an M.A. in research psychology from California State University... Read More →
avatar for Aileen M. Reid

Aileen M. Reid

Assistant Professor, UNC Greensboro
Dr. Aileen Reid is an Assistant Professor of Educational Research Methodology in the Information, Library and Research Sciences department and a Senior Fellow in the Office of Assessment, Evaluation, and Research Services (OAERS) at UNC Greensboro. Dr. Reid has expertise in culturally... Read More →
Wednesday September 18, 2024 1:30pm - 2:30pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

1:30pm AEST

Link-Up Services and wayfinding: Co-creating and navigating a culturally safe national monitoring and evaluation strategy
Wednesday September 18, 2024 1:30pm - 3:00pm AEST
Authors: Kathleen Stacey (beyond...Kathleen Stacey & Associates Pty Ltd), Cheryl Augustsson (Yorgum Healing Services), Raelene Rosas (NT Stolen Generation Aboriginal Corporation), Pat Thompson (Link-Up (Qld) Aboriginal Corporation), Jamie Sampson (Link-Up (NSW) Aboriginal Corporation)

Link-Up Services support Aboriginal and/or Torres Strait Islander people who were forcibly removed, fostered or adopted from their families as children, and their descendants who live with the ongoing impact of forcible removal policies, to reconnect with family, community, culture and Country. Wayfinding is at the core of our work - navigating unfamiliar territory with clients towards a hoped for destination of a greater sense of 'home', wherever this is possible, in a culturally safe, appropriate and trauma-informed manner.

In 2019, the National Indigenous Australians Agency funded development of a national Link-Up monitoring and evaluation strategy with the eight Link-Up Services operate across six jurisdictions. Each Link-Up Service is either a stand-alone Aboriginal community controlled organisations or based in an Aboriginal community controlled organisation.

This interactive workshop invites participants into our collective experiences of co-creating and implementing the M&E Strategy on a national basis, presented from the voices and position of Link-Up Services. We believe our experiences and learnings will be instructive for monitoring and evaluation activity with other Aboriginal and Torres Strait Islander organisations and programs.

Travel with us in reflecting on our monitoring and evaluation wayfinding journey over three phases of work. Pause with us at key points throughout the session to exercise your critical self-reflection and analysis skills, share your ideas and learn what has worked well or presented challenges for us and why in creating, navigating and implementing a culturally safe monitoring and evaluation strategy in a complex and demanding service context.
Speakers
avatar for Kathleen Stacey

Kathleen Stacey

Managing Director, beyond…(Kathleen Stacey & Associates)
Kathleen Stacey is the Managing Director and Principal Consultant at beyond... She spent her formative working years within the public sector and academia, before establishing and expanding beyond... into its current form. The company conducts consultancy, evaluation, research and... Read More →
RR

Raelene Rosas

Interim CEO, Northern Territory Stolen Generations Corporation
avatar for Patricia Thompson AM

Patricia Thompson AM

CEO, Link-Up Queensland
CEO of Link-Up (Qld), an organisation that celebrates 40 years of supporting Stolen Generations this year. Has a wealth of management experience across all levels of government and importantly at a community level.  Has represented Aboriginal & Torres Strait Islander people at a... Read More →
Wednesday September 18, 2024 1:30pm - 3:00pm AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia

2:00pm AEST

Evaluation for whom? Shifting evaluation to increase its value for local actors
Wednesday September 18, 2024 2:00pm - 2:30pm AEST
104
Authors: Linda Kelly (Praxis Consultants), Mary Raori (UNDP Pacific, FJ)

This presentation outlines an approach to program assessment of a long-term governance program working across the Pacific, the UNDP Governance for Resilience program. It tells the story of the program’s maturing evaluation approach which has shifted from serving the information needs of those with money and power to focus more particularly on the values and interests of local participants and partners..
Despite the well-documented limitations of single methodology evaluation approaches for complex programs, many international development donors and corresponding international and regional organisations, continue to require program assessment that serves their needs and values. Typically, this includes narrowing evaluation to assessment against quantitative indicators. Notwithstanding the extensive limitations of this approach, it serves the (usually short-term) needs of international donors and other large bureaucracies. It generates simple information that can be communicated and showcased in uncritical forms. It provides numbers that are easily aggregated and used for concise reporting to senior and political masters.
Such approaches risk crowding out attention to the information needs of other participants and undermine attempts to support more locally led processes. This presentation will explain how this long-term and large-scale program has shifted, making use of a values-based evaluative approach to better serve the interests of partners and participants in the Pacific. This has involved both a methodological and political shift, broadening the range of data collection and analysis methodologies and approaches, increasing resourcing to accommodate different types of data and data collection and internal and external advocacy. This one program experience echoes wider views across the Pacific about the limitations of externally imposed measures and the lack of attention to what is valued by pacific countries and people.


Chair
avatar for Duncan Rintoul

Duncan Rintoul

Managing Director, Rooftop Social
ECB devotee, mentor in the AES group mentoring program, used to be on the AES board, run a rad consulting firm that specialises in evaluation, lifelong learner. Keeping busy doing research and evaluation and facilitation work in education and justice and sustainability and health... Read More →
Speakers
avatar for Linda Kelly

Linda Kelly

Director, Praxis Consultants
avatar for Lisa Buggy

Lisa Buggy

Strategy, Learning and Innovation Specialist, UNDP Pacific Office
Ms. Lisa Buggy commenced with the UNDP Pacific Office in Fiji in January 2021 and has recently transitioned into the role of Strategy, Learning and Innovation Specialist with the Governance for Resilient Development in the Pacific project. Her current role focuses on influencing systems... Read More →
avatar for Linda Vaike

Linda Vaike

Programme Adviser - Climate Risk Finance and Governance, Pacific Islands Forum Secretariat
Wednesday September 18, 2024 2:00pm - 2:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

2:30pm AEST

Navigating a path to system impact: designing a strategic impact evaluation of education programs
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
104
Authors: Amanda Reeves (Victorian Department of Education), Rhiannon Birch (Victorian Department of Education, AU), Eunice Sotelo (Victorian Department of Education, AU)

To provide insight to complex policy problems, evaluations need to adopt a systems perspective and examine the structures, relationships and contexts that influence program outcomes.

This paper outlines the design of a 4-year strategic evaluation that seeks to understand how a portfolio of over 25 education programs are interacting and collectively contributing to system-level outcomes. In this context, policy makers require evaluation to look beyond the boundaries of individual programs and assess the holistic impact of this investment to inform where and how resources can be directed to maximise system outcomes.

The strategic evaluation presented is theory-based and multi-layered, using logic modelling to identify outcomes at the program, cluster and system level and draw linkages to develop a causal pathway to impact. The strategic evaluation and the evaluations of individual education programs are being designed together to build-in common measures to enable meta-analysis and synthesis of evidence to assess system-level outcomes. The design process has been broad and encompassing, considering a diverse range of methods to understand impact including quantitative scenario modelling and value for money analysis.

The authors will describe how the strategic evaluation has been designed to respond to system complexity and add value. The evaluation adopts an approach that is:
• interdisciplinary, drawing on a range of theory and methods to examine underlying drivers, system structures, contextual factors and program impacts
• collaborative, using expertise of both internal and external evaluators, to design evaluations that are aligned and can tell a story of impact at the system-level
• exploratory, embracing a learning mindset to test and adapt evaluation activities over time.

This paper will be valuable for anyone who is interested in approaches to evaluating the relative and collective contribution of multiple programs and detecting their effects at the system level to inform strategic decision-making.
Chair
avatar for Duncan Rintoul

Duncan Rintoul

Managing Director, Rooftop Social
ECB devotee, mentor in the AES group mentoring program, used to be on the AES board, run a rad consulting firm that specialises in evaluation, lifelong learner. Keeping busy doing research and evaluation and facilitation work in education and justice and sustainability and health... Read More →
Speakers
avatar for Amanda Reeves

Amanda Reeves

Principal Evaluation Officer, Victorian Department of Education
Amanda is an evaluation specialist with over 12 years experience leading evaluation projects in government, not-for-profit organisations and as a private consultant. She has worked across a range of issues and sectors including in education, youth mental health, industry policy and... Read More →
avatar for Eunice Sotelo

Eunice Sotelo

Senior Evaluation & Research Officer, Department of Education (Victoria)
I'm here for evaluation but passionate about so many other things - education (as a former classroom teacher); language, neuroscience and early years development (recently became a mom so my theory reading at the moment is on these topics); outdoors and travel. Workwise, I'm wrangling... Read More →
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

2:30pm AEST

If treaty is like a marriage, state evaluation needs sustained deep work: Evaluation and Victoria's First Peoples Treaty
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
105
Authors: Kate Nichols (Department of Jobs, Skills, Industry and Regions - Victoria), Milbert Gawaya (Department of Jobs, Skills, Industry and Regions, AU)

First Peoples/settler state treaties have been likened to marriage - an evolving and changeable (political) relationship, not an endpoint or divorce (Blackburn, 2007). But what does this look like in practice given marriage's checkered connection with power imbalance and violence through to romance and deep, trusting companionship?

Contemporary colonial 'settlerism' (after Aunty/Dr Lilla Watson, in Watego, 2021) is undergoing structural change in Victoria, with Victoria's First Peoples sitting down with the Victorian State Government in 2024 to commence statewide treaty negotiations. Treaty is an acknowledgement that British sovereignty did not extinguish Aboriginal sovereignty, opening-up a "third space of sovereignty" (after Bruyneel, 2007) where co-existing sovereigns can further contest the "sovereignty impasse" (ibid., 2007), while Indigenous people control their own affairs.

Treaty is expected to reshape how the Victorian state government operates, challenging state laws, institutions, policies, programs and processes, which together, have contributed to First Nations disadvantage and suffering. Government evaluation practices will need their own shake-up.

How can public sector evaluators help establish an equal, strong and nourishing treaty marriage? This short paper shares emerging ally insights into how local practices are evolving to support Victoria's Treaty and self-determination. It shares reflections from a recent evaluation of Traditional Owner grant programs, conducted in partnership between key Aboriginal and non-Aboriginal public sector staff. It is a story of both-ways practice and the time, trust and bravery required to achieve deep change. It also highlights the role of lifelong cultural learning and behaviour change for ally evaluators. Culturally responsive evaluation, Indigenous research practices, restorative justice and the AES First Nations Cultural Safety Framework provide useful framing. Although focused on the Victorian treaty context, the paper may be transferable to other jurisdictions and evaluations involving or impacting Aboriginal and Torres Strait Islander peoples in support of their sovereignty and self-determination.
ion trainers and facilitators.
Chair
avatar for Martina Donkers

Martina Donkers

Independent Evaluator
I'm an independent freelance evaluator with a background in program design, grants, and science communication. I have a Master of Evaluation, and I'm finding my sweet spot in qualitative and mixed methods evaluation with a complexity and systems lens. I like rubrics, semi-structured... Read More →
Speakers
avatar for Kate Nichols

Kate Nichols

Senior Evaluator, Department of Economic Development, Jobs, Transport & Resources
I've been a practising evaluator since Missy Elliot released 'Work it' which a) reveals a bit too much about my age, but b) gives you a sense of how much I'm into this stuff. I've recently returned to an evaluation role in the Victorian public sector after working in a private sector... Read More →
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

2:30pm AEST

When "parachuting in" is not an option: Exploring value with integrity across languages, continents and time zones
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
106
Authors: Julian King (Julian King & Associates), Adrian Field (Dovetail)

Ethics in Pacific Evaluation: Voices of the Rebbilib is a research project under the Strengthening Pacific MEL initiative of the Pacific Community. The ethics of evaluation practice are some of the most consistent norms documented by evaluation societies and associations internationally. This is because a clear set of ethical guidelines provides evaluators of all skill levels an understanding of their roles and responsibilities in the way they conduct themselves in practice. Similarly, codifying ethical principles of evaluation within a guideline or manual assumes that this is an effective way of ensuring evaluators adhere to a high standard of ethical practice. This presentation presents a provocation of the assumed norms of ethical evaluation practice through the sharing of experiences from the Pacific.
The panel will consider the question of whether normative ethics actually improve evaluation practice in the Pacific. A literature review of evaluation principles guidelines and manuals identified the normative ethics including the principle of best judgement in the application of ethical principles and values. This principle asks evaluators to confront their positionality within an evaluatative context and confront their own power in evaluation. The presentation will illuminate what this looks like for experienced evaluators working in the Pacific.
This provocation entices a new ethics-oriented entry point to help us seek solutions to accelerate action towards localising evaluation practice in the Pacific. This panel discussion will include key findings from a broader study on this topic which includes several leading Pacific evaluators. The findings suggests that the principle of exercising one's best judgement requires greater appreciation of culture and context with recommendations about how to achieve this. Building on the Pacific Monitoring, Evaluation and Learning Capacity Strengthening wayfinding tool the Rebbilib, this research calls on evaluators and evaluation commissioners to demand more from evaluative principles and values.

Chair
avatar for Melinda Mann

Melinda Mann

Academic Lead Jilbay First Nations RHD Academy, CQUniversity
Melinda Mann is a Darumbal and South Sea Islander woman based in Rockhampton, Qld. Her work focuses on Indigenous Nation building, Pacific sovereignties, and regional and rural communities. Melinda has a background in student services, learning design, school and tertiary education... Read More →
Speakers
avatar for Julian King

Julian King

Director, Julian King & Associates
I’m an independent public policy consultant based in Auckland. I specialise in evaluation and Value for Investment. I’m affiliated with the Kinnect Group, Oxford Policy Management, the University of Melbourne and the Northern Institute. Subscribe to my weekly blog at https:/... Read More →
avatar for Adrian Field

Adrian Field

Director, Dovetail
Adrian is the director of Dovetail, an Auckland-based evaluation consultancy, and a member of the Kinnect Group. Adrian has worked in evaluation in different capacities for some 20 years and doesn't really like how old that makes him feel. Adrian's experience traverses health, social... Read More →
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
106 102 Convention Centre Pl, South Wharf VIC 3006, Australia

2:30pm AEST

Elevating evaluation: practical insights for supporting systems transformation
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
Authors: Kathryn Erskine (Cube Group), Michael Maher (Cube Group, AU)

The session is intended to provide a practical example of how a traditional program evaluation was re-orientated to allow the application of findings to inform broader system transformation. This echoes current discourse in evaluation - particularly since the global pandemic - whereby traditional notions about how the field of evaluation is viewed and developed are being challenged (Ofir, Z. 2021). Specifically, there are calls to rethink and elevate evaluation practice to actively contribute to and support systems transformation (see Dart 2023; Norman 2021), beyond a narrow programmatic focus.

This session will illuminate this discussion by examining a mental health program evaluation in the context of significant service reform across the Victorian mental health system. The presentation will outline insights and techniques about how to lift and reconfigure a tightly defined program evaluation into one which can have broader application to the system ecosphere. It outlines how and why the pivot was made; changes we made to the methodology and the key benefits that arose from taking an expansive view of the sector in which the program operated within.

The design of the session will be a presentation format supported by a PowerPoint slide deck, comprising:
•    Introduction and purpose of session
•    Overview of the program we evaluated
•    Key challenges which required an evaluation 'pivot' - and how we worked with our client
•    Key changes made to the methodology
•    Key benefits from elevating from a programmatic to systems focus.


Chair
avatar for Nick Field

Nick Field

Director (Public Sector), Urbis
Nick has twenty years of public sector consulting experience, backed more recently by six years as a Chief Operating Officer in the Victorian Public Sector. A specialist generalist in a broad range of professional advisory services, Nick has expertise in the implementation of state-wide... Read More →
Speakers
avatar for Kathryn Erskine

Kathryn Erskine

Director, Cube Group
Combining academic rigour with a practical ‘can-do’ approach, Kathryn is committed to delivering evidence-based change that improves the lives of Australians.Kathryn brings a depth and breadth of experience in the public, private and not-for-profit sectors, specialising in program... Read More →
avatar for Michael Maher

Michael Maher

Partner & Evaluation Lead, Cube Group
Leading Cube Group’s Evaluation and Review practice, Michael brings over 30 years of experience in the public, private and not-for-profit sectors. Michael’s work spans all areas of social policy with particular expertise in early childhood, education, justice, human services... Read More →
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

2:30pm AEST

When speed is of the essence: How to make sure the rubber hits the road
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
103
Authors: Kristy Hornby (Grosvenor)

There is a lot of interest in rapid M&E planning and rapid evaluations at present; partially borne out of rapid contexts in a COVID-19 policy context; and partially borne out of constricting appetites for time and money spent on evaluations. It is unlikely this trend will reverse in the short-term, so what do we do about it to acquit our responsibilities as evaluators, ethically and appropriately, in a rapid context? This session sets out a step by step approach to conducting a rapid evaluation, inviting attendees to follow along with their own program in mind, to come away from the session with a pathway for conducting their own rapid evaluation. The session uses a fictional case study as the construct to move the rapid evaluation approach forward, describing throughout the session how you can use literature reviews, qualitative and quantitative data collection and analysis techniques, and report writing approaches innovatively to save you time while not compromising rigour.

We contend it is possible to do a rapid evaluation ethically and appropriately, but the backbone of doing so is good planning and execution. This session shares practical tips and approaches for doing so through each key phase of an evaluation, so attendees are well-equipped for their next rapid evaluation.

To consolidate the learning, attendees will be provided a framework to come away from the session with a high level plan of how to conduct their own rapid evaluation, to increase their chance of success.

Chair Speakers
avatar for Kristy Hornby

Kristy Hornby

Associate Director, Victorian Evaluation Lead, Grosvenor
Kristy has over ten years of evaluation experience, with expertise spanning the Victorian state government, federal government, local government and not-for-profit sectors. She has particular expertise in social services, employment, primary health, agriculture and environment and... Read More →
Wednesday September 18, 2024 2:30pm - 3:00pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

3:30pm AEST

Plenary: James Copestake "What next? From evaluating to anticipating"
Wednesday September 18, 2024 3:30pm - 4:30pm AEST
Professor, International Development, University of Bath, UK

As evaluators, we are often asked to find out ‘what caused what’ in the past, but also resist being labelled solely as hired evidence collectors. I argue for further effort to move beyond ‘what worked?’ to ‘what next?’- from outcome-activity links in the past to scenario-action possibilities and options in the future. Anticipatory evaluation can enhance the relevance and usefulness of our work, but also accentuates the challenges of appropriate framing, causal analysis, normative deliberation, and influencing. It increases uncertainty, recasts stakeholder relationships, and requires use of a wider range of complexity-informed methods. I explore these issues by reflecting on the scope for anticipatory evaluative practice in four diverse fields, each at a different level: planning doctoral research (interpersonal), appraising impact investment (organisational), mainstreaming social policy initiatives (national), and rethinking development (global). Becoming more future-oriented makes new demands on each of us as evaluation professionals, but also requires collective action to build stronger bridges with communities of practice in anticipatory action, appraisal, foresight, and future thinking.
Chair
avatar for Donna Loveridge

Donna Loveridge

Impact strategy and evaluation consultant
I work with public sector and not for profit organisations and businesses to design and conduct evaluations and embed evaluative thinking in management systems and processes to strengthen learning and decision-making. Most of my work focuses on inclusive economic growth through impact... Read More →
Speakers
avatar for James Copestake

James Copestake

Professor, International Development, University of Bath, UK
James Copestake is Professor of International Development at the University of Bath in the UK, where he is also Director of Studies for the Doctorate in Policy Research and Practice at the Institute of Policy Research.His publications range broadly across international development... Read More →
Wednesday September 18, 2024 3:30pm - 4:30pm AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Wayfinding for integrating social justice and culturally responsive and equitable evaluation practices in meta-evaluation: Learning from the UN evaluation quality assessments.
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
Authors: Sandra Ayoo (University of North Carolina Greensboro)

With quality in improving interventions to improve societal and planetary wellbeing being the desired destination of evaluation, it is imperative that evaluators reflect on the meaning of quality and methods to assess if evaluation is achieving it. Meta-evaluation, coined by Michael Scriven in 1969, evaluates evaluations and aids in understanding how evaluations contribute to addressing structural and systemic problems in interventions and evaluation practice. Meta-evaluation has evolved over the past five decades and is included in the program standards by major professional associations. While the field of evaluation is confronted with major concerns regarding the centrality of social justice, there is currently there are no one-size-fits-all guidelines for meta-evaluation and for addressing social justice in evaluations. To address this, we reviewed meta-evaluation literature, mapped the American Evaluation Association's foundational documents with the United Nations Evaluation Group's Norms and Standards to explore their intersectionality on social justice, and analyzed 62 United Nations Population Fund evaluation reports alongside their management responses. The study findings indicated that meta-evaluation is contingent on context rather than established standards. Thus, it's crucial for evaluators to intentionally prioritize social justice in evaluation design and implementation and to select quality assurance tools that match the evaluation context and professional association guidelines to improve the quality of the intervention. I will share key characteristics of the United Nations Group's Norms and Standards on social justice to stimulate discussions on evaluators' efforts to address systemic issues. Collectively, participants will benefit from discussing and reflecting on their own practice by responding to questions like (a) what are the examples of their work in collaborative and systems-informed ways to intentionally include social justice in their evaluations, and (b) what should the field of evaluation do to ensure that evaluations add value for people and planet?anagement response.
Chair
avatar for Carlos Rodriguez

Carlos Rodriguez

Senior Manager Strategy & Evaluation, Department of Energy Environment and Climate Action
Speakers
avatar for Sandra Ayoo

Sandra Ayoo

Assistant Professor, University of North Carolina Greensboro
Dr. Ayoo is an Assistant Professor of Educational Research Methodology in the Department of Information, Library, and Research Science at the School of Education, University of North Carolina Greensboro. She teaches graduate courses in program evaluation and research methodology... Read More →
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Developing a Tool for Measuring Evaluation Maturity at a Federal Agency
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
105
Authors: Eleanor Kerdo (Attorney Generals Department ),Claudia Oke (Attorney Generals Department, AU),Michael Amon (Attorney Generals Department, AU),Anthony Alindogan (Attorney Generals Department, AU)

To embed a culture of evaluation across the Australian Public Service (Commonwealth of Australia, 2021), we must first have an accurate understanding of the current state of evaluation capability and priorities across Commonwealth agencies. This paper shares tools on how to build an effective measurement framework for evaluation culture, and discusses how to use these for evaluation capability uplift.
We explore quantitative and qualitative methods to gather and analyse data to measure an organisation's readiness to change its culture towards evaluation. This includes assessing staff attitudes towards evaluation, the level of opportunity for staff to conduct and use evaluation, and confidence in their knowledge of evaluation.
We discuss the development of a staff evaluation culture survey based on Preskill & Boyle's ROLE and how behavioural insight tools can be utilised to boost engagement. The paper discusses the utility of holding focus groups with senior leaders to understand authorising environments for evaluation and key leverage points. Also discussed, are challenges and innovative solutions that were encountered throughout the assessment process.
This paper will be valuable for those who work in, or with, any government agency with an interest in evaluation capacity building and driving an evaluation culture within organisations. This paper explains each stage of measurement design, data analysis and results, and discussing opportunities for action.
1 Preskill, H., & Boyle, S. (2008). A Multidisciplinary Model of Evaluation Capacity Building. American Journal of Evaluation, 29(4), 443-459. ://journals.sagepub.com/doi/10.1177/1098214008324182

2 Michie S, Atkins L, West R. (2014) The Behaviour Change Wheel: A Guide to Designing Interventions. London: Silverback Publishing. www.behaviourchangewheel.com.

3 Lahey, R. (2009). A Framework for Developing an Effective Monitoring and Evaluation System in the Public Sector: Key considerations from International Experience. Framework for developing an effective ME system in the public sector (studylib.net)
Chair
avatar for Marwan El Hassan

Marwan El Hassan

Director, Future Drought Fund Program Evaluation and Support, Department of Agriculture, Fisheries and Forestry
I am the director of the Program Evaluation and Support team at the Future Drought Fund (FDF). My team is responsible of supporting the FDF's program areas in their monitoring, evaluation and learning work, and to ensure alignment of our MEL work with other areas around the department... Read More →
Speakers
avatar for Michael Amon

Michael Amon

Director, Data Policy, Evaluation & Visualisation, Attorney-General’s Department
As with most of us here, I have a passion for evaluation and bringing evidence-based policy to the policy space. I've led and established a few evaluation units/teams in the federal governement. I've also lead policy development branches, learning & development teams, and have a PhD... Read More →
avatar for Anthony Alindogan

Anthony Alindogan

Evaluation Lead, Attorney-General's Department
Anthony is an experienced evaluator with a particular interest in outcomes measurement and value-for-money. He completed his Master of Evaluation degree from the University of Melbourne. Anthony is an enthusiastic writer and has publications in various journals including the Evaluation... Read More →
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Uncovering Hidden STEM Footprints: Leveraging Output Data from Questacon’s Outreach Programs
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
104
Authors: Jake Clark (Questacon - The National Science and Technology Centre), Jenny Booth (Questacon - The National Science and Technology Centre, AU), Sharon Smith (Questacon - The National Science and Technology Centre, AU), Nick Phillis (Questacon - The National Science and Technology Centre, AU)

Join our Monitoring and Evaluation team on an exciting journey as we delve into the untapped potential of output data in evaluation and how to reach beyond the 'low-hanging fruit'.

Notwithstanding to the importance of evaluating outcomes to measure program success, monitoring implementation and reach of initiatives is fundamental to good program management and evaluation. Output data on activity reach, target groups and participants often hold hidden gems of potential that are frequently overlooked. In this presentation we shine a spotlight on its significance and actionable tips to elevate monitoring data.

Our objective is to make this exploration enjoyable and enlightening, especially for foundational to intermediate level evaluators. We offer practical and universally applicable strategies for making the most of output data on to enhance program insights.

KEY MESSAGES

Using existing tools and tapping into open-source data sets you can create powerful visualisations and draw deeper inferences about your program reach and participants.

I. Understanding equity and inclusion
• A better understanding of who is and isn't involved in your initiative.
• Looking for patterns using socio-demographic variables.
• Benchmarking your initiative against relevant population data.

II. Connecting outputs to outcomes
• Analysing participant characteristics and program journey to illuminate differences in outcomes.
• Uncover program and policy questions that need further exploration.

Design of the Session: Drawing tangible examples from the education and informal learning STEM sector, we bridge the gap between theory and practice. Real-world strategies are shared to encourage active participation along with useful resource links.
Chair Speakers
avatar for Jake Clark

Jake Clark

Senior Monitoring and Evalution Officer, Questacon - National Science and Technology Centre
What value does STEM outreach bring to an individual? How does it change someone's attitude/behaviour/disposition around STEM? And how do you quantify such probing questions? These are the types of queries I'm answering in my Senior Evaluation Officer role at Australia's National... Read More →
Wednesday September 18, 2024 4:30pm - 5:00pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Evaluation Lab: Using design to solve evaluation challenges
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
Authors: Matt Healey (First Person Consulting)

The Design and Evaluation Special Interest Group (DESIG) was established in 2017. Its primary aim has been to explore the intersection of evaluation and design, and that aim has been interpreted in different ways over time. In 2024, the DESIG identified an opportunity to take the SIG model in a slightly different direction, embarking on an innovative venture with the launch of the Evaluation Lab, an initiative aimed at talk into action, and taking evaluators through a design process to address evaluation challenges.
Drawing inspiration from the concept of 'living labs,' which serve as real-world testing grounds, the Evaluation Lab created a space where evaluation professionals could come together. Employing a design-thinking process, the Lab guided participants through a structured expedition of defining, ideating, and prototyping solutions to tackle nominated challenges. Participants also learned pitch skills to communicate their solutions.
This Big Room Session provides an opportunity for the DESIG to outline the Evaluation Lab model, capped off with participants presenting their solutions through rapid-fire pitches, either live or pre-recorded, akin to explorers sharing tales of new lands discovered. The session's innovative twist lies in the audience's role, acting as both audience and judges. The audience will vote on their favourite solution, and be involved in crowing the first AES Evaluation Lab winner.
By blending lecture-style content with dynamic team presentations and active audience engagement, the Big Room Session not only highlights the critical role of design in navigating evaluation challenges but also demonstrates the practical application of these methodologies in charting a course through real-world problems.

Chair
avatar for Duncan Rintoul

Duncan Rintoul

Managing Director, Rooftop Social
ECB devotee, mentor in the AES group mentoring program, used to be on the AES board, run a rad consulting firm that specialises in evaluation, lifelong learner. Keeping busy doing research and evaluation and facilitation work in education and justice and sustainability and health... Read More →
Speakers
avatar for Matt Healey

Matt Healey

Principal Consultant, First Person Consulting
My career in evaluation started fairly traditionally. I joined a small firm as a Research Assistant in early 2014 with no idea what evaluation was, or what I was in for! Since then I have:Co-founded and grown an organisation (First Person Consulting) to a team of 16 people working... Read More →
avatar for Shani Rajendra

Shani Rajendra

Principal Consultant & Head of Business Group (Social Impact), Clear Horizon
Shani is a Principal Consultant in Clear Horizon’s Social Impact team. Shani has extensive experience in community-led initiatives, organisational strategy, and social enterprise. She specialises in incorporating design thinking into evaluative practice. Having completed a Master... Read More →
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Navigating the choppy waters of the evaluation landscape in the Pacific
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
106
Authors: Allan Mua Illingworth (Mua'akia Consulting and Insight Pasifika) Fiona Fandim (Pacific Community (SPC), FJ), Eroni Wavu (MEL Officer for Pacific Women Lead at Pacific Community (SPC) and cofounder of the Fiji Monitoring, Evaluation & Learning Community), Mereani Rokotuibau (Balance of Power Program, FJ) and Chris Roche (La Trobe University),

In recent years there have been a number of Pacific driven initiatives designed to promote monitoring and evaluation practice which is culturally and contextually appropriate. These have occurred with projects and programs as well as at national and regional levels. At the same time geo-political interest in the Pacific region has resulted in an increased number of bi and multilateral donor agencies becoming present in the region and/or funding development programs, local organisations, national governments and regional bodies. This has in turn led to an evaluation landscape where notions of 'international best practice' as well as donor policies and practices and associated international researcher and consulting companies, risk crowding out emergent Pacific led evaluation initiatives.

This panel will bring together key participants who are leading four examples of these Pacific experiences: the Rebbilib process initiated by the Pacific Community (SPC ), Insight Pasifika (an emerging Pacific led and owned collective focused on evaluation in the first instance): the Fiji Monitoring, Evaluation & Learning Community and the Balance of Power program (a Pacific-led initiative, supported by the Australian Government, focused improving the political, social and economic opportunities for women and girls) each of whom are seeking to create space for processes of monitoring, evaluation and learning which are consistent with Pacific ways of knowing and being. They will share their experience, the challenges they face and ideas about what forms of support might be provided by international donors, consultants and advisors which are enabling rather than undermining.

Moderated by Prof. Chris Roche the panel and audience will also draw out the lessons from these four cases about what might contribute to more systemic change in the evaluation landscape more generally.
Chair
avatar for Charlie Tulloch

Charlie Tulloch

Director, Policy Performance
Policy Performance is a proud conference sponsor! Charlie delivers evaluation projects, capability building support and drives public sector improvement. Charlie loves to help those who are new to evaluation or transitioning from related disciplines. He is a past AES Board member... Read More →
Speakers
avatar for Allan Mua Illingworth

Allan Mua Illingworth

Adjunct Research Fellow, La Trobe University
Allan Mua Illingworth is a Monitoring and Evaluation specialist of Pacific Island heritage with a long career of international development experience and an extensive network of contacts who have worked to support development regionally and across many Pacific Island countries over... Read More →
avatar for Chris Roche

Chris Roche

Professor of Development Practice, La Trobe University
I am Professor Development Practice with the Centre for Human Security and Social Change at La Trobe University - (https://www.latrobe.edu.au/socialchange) - and former Deputy Director of the Developmental Leadership Program (www,dlprog.org) and member of the intellectual leadership... Read More →
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
106 102 Convention Centre Pl, South Wharf VIC 3006, Australia

4:30pm AEST

Rebalancing Power Dynamics: Philanthropy Through the Lens of First Nations Community-Driven MEL
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
103
Authors: Skye Trudgett (Kowa), Rachel Kerry (CAGES Foundation, AU)

The philanthropic sector has long grappled with power imbalances inherent in funding relationships, particularly in the context of support for First Nations communities. This panel session explores a groundbreaking Monitoring, Evaluation, and Learning (MEL) approach that inverts traditional power structures, placing First Nations communities in the driver's seat to assess the adherence of a leading philanthropic organisation to their stated values and principles.

Drawing from the collaborative efforts of one foundation and its MEL partner, this session showcases a MEL model that exemplifies shared power and mutual accountability. The panel will consist of thought leaders from the philanthropic sector, First Nations community representatives, and MEL experts who have been at the forefront of developing and implementing this innovative approach.

Through a facilitated discussion, panellists will delve into the process of co-creating a MEL framework that empowers communities to evaluate the performance of philanthropists against a set of mutually agreed-upon criteria. This approach ensures that philanthropic actions align with community expectations, cultural protocols, and contribute to genuine and sustainable impact.

Attendees will gain insights into the challenges and successes of operationalising this community-centric MEL method. The session aims to inspire other philanthropic entities to reflect on their practices and adopt similar approaches that truly shift power to First Nations communities.
Chair Speakers
avatar for Skye Trudgett

Skye Trudgett

CEO, Kowa
Skye is a Gamilaroi researcher who has contributed to numerous evaluations and research projects including place-based, systems change and government initiatives. Skye has a particular interest in Indigenous Data Sovereignty & Governance and seeks to support the practical application... Read More →
avatar for Rachel Kerry

Rachel Kerry

CEO, CAGES Foundation
Rachel was born and raised in Sydney on the land of the Bidgigal people and after much exploration now lives there with her husband and two teenage sons. Despite having an academic background in ecology and zoology Rachel somehow landed in the philanthropic world. She has over twenty... Read More →
avatar for Tara Leslie

Tara Leslie

CEO, Cullunghutti Aboriginal Child and Family Centre
I am a proud Aboriginal woman descendant of the Gamillarayand Yuin Nations. A mother of 4 boys, working and living on Wandi Wandian country. I am the Chief Executive Officer at Cullunghutti Aboriginal Child and Family Centre. I have worked in the Aboriginal service sector for over... Read More →
Wednesday September 18, 2024 4:30pm - 5:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

5:00pm AEST

Trigger warnings - do they just trigger people more?
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
104
Authors: Kizzy Gandy (Verian (formerly Kantar Public) )

As evaluators, one of our key ethical responsibilities is to not cause psychological harm or distress through our methods. We often start workshops or interviews with a warning that the topic may be upsetting and provide contact information for mental health services to participants under the assumption this is the most ethical practice.

Trigger warnings are used with good intentions and are often recommended in evaluation ethics guidelines. However, what do we know about their impact? Is there a risk they actually trigger people more?

This talk examines the evidence on whether trigger warnings are an effective strategy for reducing the risk of trauma and re-traumatisation when discussing topics such as sexual assault, mental health, violence, drug use, and other sensitive issues. It also touches on new evidence from neuroscience about how emotions are understood differently now compared to in the past.

This session will not provide a definitive answer on when or how to use trigger warnings but aims to challenge the audience to think critically about whether trigger warnings are useful in their own work.
Chair Speakers
avatar for Kizzy Gandy

Kizzy Gandy

National Director, Program Evaluation, Verian
Dr Kizzy Gandy is Verian's National Director of Program Evaluation. She leads a team of expert methodologists and provides quality assurance. With 20 years’ experience in consultancy, federal and state government, and academia, Kizzy has overseen the design and evaluation of over... Read More →
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

5:00pm AEST

Development and implementation of a culturally grounded evaluation Framework: Learnings from an Aboriginal and Torres Strait Islander Peak.
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
Authors: Candice Butler (Queensland Aboriginal and Torres Strait Islander Child Protection Peak ),Michelle McIntyre (Queensland Aboriginal and Torres Strait Islander Child Protection Peak, AU),John Prince (JKP Consulting, AU)

There is increasing recognition that evaluations of Aboriginal and Torres Strait Islander programs must be culturally safe and appropriate, and represent the worldviews, priorities, and perspectives of Aboriginal and Torres Strait Islander communities. Aboriginal and Torres Strait Islander peoples have the cultural knowledge and cultural authority to design appropriate evaluations that are safe, and that tell the true story of the impacts of our ways of working.

As a peak body for Aboriginal and Torres Strait Islander community-controlled organisations we wanted to ensure that the worldviews and perspectives of our members and communities are embedded in any evaluations of services delivered by our member organisations. This is a necessary step towards building an evidence base for our ways of working, developed by and for Aboriginal and Torres Strait Islander people. To that end we developed an evaluation framework to enable self-determination and data sovereignty in evaluation, and to build capacity among our member organisations to undertake and/or commission culturally grounded evaluations. Culturally grounded evaluations are led by Aboriginal and Torres Strait Islander people and guided by our worldviews and knowledge systems - our ways of knowing, being and doing.

This paper reports on the development and implementation process used in the project and describes the standards and principles which underpin the framework. An example of how the framework is being applied in practice is also outlined. Our principles for evaluation describe the core values which underpin culturally grounded and safe evaluation including self-determination; cultural authority; truth-telling; two-way learning; and holistic approaches. The evaluation standards and associated elements operationalise our principles and embed them in evaluative practice.
Chair
avatar for Carlos Rodriguez

Carlos Rodriguez

Senior Manager Strategy & Evaluation, Department of Energy Environment and Climate Action
Speakers
avatar for Candice Butler

Candice Butler

Executive Director, Centre of Excellence, Queensland Aboriginal and Torres Strait Islander Child Protection Peak
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
101-102 105 Convention Centre Pl, South Wharf VIC 3006, Australia

5:00pm AEST

Failing your way to better practice: How to tread carefully when things aren't going as planned
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
105
Authors: Stephanie White (Victoria Department of Education )

Evaluators can fail in many ways. The consequences of these failures can be relatively contained or wide ranging within the evaluation and can also flow on to program operations. But failure is a part of life and can be a useful catalyst for professional growth. What happens when you find yourself failing and can see the risks ahead? How do you keep going?

The session focuses on the experiences of an emerging evaluator who failed while leading a large-scale education evaluation. When some elements of the evaluation became untenable, they struggled to find the right path forward and could foresee the risks materialising if the situation wasn’t addressed. On the other side of it, they reflect on how they drew on tools in every evaluator’s toolkit to start remedying their previous inaction and missteps to get the evaluation back on track…and improve their practice along the way!

This session is relevant to any evaluator who grapples with the messiness of expectations and reality in their practice.


Chair
avatar for Marwan El Hassan

Marwan El Hassan

Director, Future Drought Fund Program Evaluation and Support, Department of Agriculture, Fisheries and Forestry
I am the director of the Program Evaluation and Support team at the Future Drought Fund (FDF). My team is responsible of supporting the FDF's program areas in their monitoring, evaluation and learning work, and to ensure alignment of our MEL work with other areas around the department... Read More →
Speakers
avatar for Stephanie White

Stephanie White

Victoria Department of Education
I found my way to evaluation to help me answer questions about education program quality and success. Professionally, I have diverse experiences in education and evaluation, from delivering playgroups under trees in the NT to reports on educator resources to senior education bureaucrats... Read More →
Wednesday September 18, 2024 5:00pm - 5:30pm AEST
105 109 Convention Centre Pl, South Wharf VIC 3006, Australia

5:30pm AEST

AES 2024 Annual General Meeting
Wednesday September 18, 2024 5:30pm - 6:15pm AEST
Join the Australian Evaluation Society (AES) Board as we celebrate another year’s achievements by members of the AES, introduce newly elected Board members, and launch the AES strategic priorities 2024-2028.

Notice of Annual General Meeting | Agenda | 2024 Annual Report

Chair
avatar for Kiri Parata

Kiri Parata

President, Australian Evaluation Society
Kia ora, greetings allI'm always excited to attend AES conferences and the Wayfinding theme of aes24 speaks to my heritage and culture. This year with my family I holidayed in beautiful Rarotonga in the Cook Islands. We visited the site where the waka (Māori watercraft/vessel) of... Read More →
Wednesday September 18, 2024 5:30pm - 6:15pm AEST
Plenary 1 114 Convention Centre Pl, South Wharf VIC 3006, Australia
 
Share Modal

Share this link via

Or copy link

Filter sessions
Apply filters to sessions.
Filtered by Date -