aes24 International Evaluation Conference: Full Schedule

Conference hashtag #aes24MEL

1:30pm AEST

Learning from failure at a NFP - pitfalls and pointers

Wednesday September 18, 2024 1:30pm - 2:30pm AEST

Authors: Victoria Pilbeam (WWF-Australia)

Across social and environmental movements, we are often reticent to talk about failure. But as innovation and learning gain greater emphasis across the sector, Not-for Profits are finding new ways to share and learn from their failures (eg: Engineers Without Borders failure reports, Save the Children Fail Fest, etc.). In this presentation, I will share both insights from the available research and reflect on my own journey developing failure programming at WWF-Australia. This presentation will provide practical guidance to evaluators and organisations navigating the challenging terrain of learning from failure.

Chair

Squirrel Main

Speakers

Victoria Pilbeam

MEL Adviser, The Pacific Community (SPC)

At the Pacific Community, I support MEL for fisheries, aquaculture and marine ecosystems across the region. Previously, I worked for WWF-Australia and inconsulting with a range of not-for profit, government , and philanthropic partners. I love MEL that is approachable, equitable... Read More →

Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Intermediate
Industry Environment and climate change, International development, Natural resources and energy, NFP or Philanthropy

1:30pm AEST

No help? No worries! How to find your way on your own

Wednesday September 18, 2024 1:30pm - 2:30pm AEST

103

Authors: Rachel Wilks (Grosvenor), Kristy Hornby (Grosvenor)

As we learn the practice of evaluation, we often have experienced evaluators around us. They provide us guidance and guardrails within which to learn and grow our capabilities - but what happens when we no longer have this support? How do we stop ourselves from getting lost?

This session will provide tips and tricks for evaluators going out into the world on their own. Attendees will leave with practical advice and strategies to help keep them on track while they build their autonomy.

Chair

Squirrel Main

Speakers

Rachel Wilks

Senior Consultant, Grosvenor

Rachel is a management consultant and an emerging evaluator at Grosvenor. She took her first steps into the evaluation world two years ago, and since then has been increasingly interested in how evaluation can be used in and across the public sector and not-for-profit space. Rachel... Read More →

Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational
Industry Other

1:30pm AEST

Revitalising Survey Engagement: Strategies to Tackle Low Response Rates

Wednesday September 18, 2024 1:30pm - 2:30pm AEST

103

Authors: Kizzy Gandy

Surveys are an excellent data collection tool when they reach their target response rate, but low response rates hinder the generalisability and reliability of the findings.

This Ignite presentation will discuss techniques Verian evaluators have applied to increase survey response rates while also assessing the efficacy and efficiency of these techniques. We will also explore other evidence-based strategies for boosting response rates and the value of drawing on other data sources if your response rates are still low.

Chair

Squirrel Main

Speakers

Hannah Nguyen

Economist, Verian Group

Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Community and social services, Education and training, Public administration and safety, NFP or Philanthropy

1:30pm AEST

Sign here: Supporting Deaf participation in evaluation

Wednesday September 18, 2024 1:30pm - 2:30pm AEST

103

Authors: Susie Fletcher (Australian Healthcare Associates)

Auslan is a visual, signed language that was developed by and for the Australian Deaf community. People who use Auslan as their primary or preferred language are not necessarily fluent in English. Our team was engaged to review access to interpreter services for Auslan users, a population group that is often underrepresented in evaluation. In this presentation we will highlight some of the issues evaluators need to consider when working with this marginalised community, and share practical skills and techniques for making their evaluations more accessible.

Chair

Squirrel Main

Speakers

Susie Fletcher

Senior consultant, Australian Healthcare Associates

Dr Susie Fletcher is an experienced health services researcher with over 50 peer reviewed journal articles and 3 book chapters in mental health and primary care. She is passionate about improving health outcomes through integrating services across sectors; her recent work has included... Read More →

Wednesday September 18, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Community and social services, Health and disability, Other

3:30pm AEST

Charting the Course: Measuring Organisational Evaluation Capacity Building

Thursday September 19, 2024 3:30pm - 4:30pm AEST

104

Authors: Rochelle Tobin (Curtin University)

Measuring evaluation capacity building is complex, and there are few examples of quantitative measurement tools to enable evaluators to chart progress. WAAC (WA AIDS Council) and Curtin established a five-year partnership to build evaluation capacity within WAAC. To measure progress, a validated tool (Schwarzman et al. 2019) to assess organisational evaluation capacity was modified and combined with another partnership-based tool (Tobin et al. in press). The survey was administered to WAAC staff at baseline (n = 17) and then one year after the partnership was established (n = 19). Significant improvements were seen in individual skills for evaluation tasks, tools for evaluation and evaluation systems and structures. These tools provide a rigorous approach to tracking progress towards organisational evaluation capacity.

Chair

Claire Grealy

Director, Rooftop Social

So looking forward to AES 2024! We are Silver Sponsors this year, which means we're keeping your devices charged up through the conference, and you'll find us next to the charging stations. I welcome any and all conversation about evaluation, strategy and design, research, facilitation... Read More →

Speakers

Rochelle Tobin

PhD candidate

I am a PhD candidate investigating SiREN's (Sexual Health and Blood-borne Virus Research and Evaluation Network) influence on research and evaluation practices in the Western Australian sexual health and blood-borne virus sector. I also support SiREN's knowledge translation activities... Read More →

Thursday September 19, 2024 3:30pm - 4:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Intermediate
Industry Community and social services, Education and training, NFP or Philanthropy

3:30pm AEST

Journey Mapping: Visualising Competing Needs within Evaluations

Thursday September 19, 2024 3:30pm - 4:30pm AEST

104

Authors: Jolenna Deo (Allen and Clarke Consulting)

Journey mapping acts as a GPS to grasp audience or consumer experience in evaluating policies or programs, highlighting twists, hidden gems, and pitfalls It can be a useful tool to help evaluators capture disparities and competing needs among intended demographics. This session will discuss the journey mapping method, drawing from an evaluation of a Community Capacity Building Program which used journey mapping to illustrate key consumer personas. It will explore the integration of multiple data sources to provide a comprehensive understanding of complex disparities and the cultural and historical contexts in which these arise.

Chair

Claire Grealy

Director, Rooftop Social

Speakers

Jolenna Deo

Consultant, Allen and Clarke Consulting

Jolénna is a consultant at Allen + Clarke consulting. She is a proud Mirriam Mer, Pasifika women with a background in Development studies, Pacific studies and social policy, combining her interests in indigenous methodologies and social justice. She is experienced in community and... Read More →

Thursday September 19, 2024 3:30pm - 4:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Community and social services, Education and training, Environment and climate change

3:30pm AEST

Reflections by a non-analyst on the use of state-wide data sets and modelled data in evaluation

Thursday September 19, 2024 3:30pm - 4:30pm AEST

104

Authors: Gabby Lindsay-Smith

Using linked Government data sets provide an opportunity to investigate the impact of state-wide programs and policies but are often out of reach for many evaluators, and especially non-analysts. This presentation will detail a non-analyst’s experience incorporating state linked data sets into a recent evaluation of a Victorian-wide family services program evaluation. The presentation will outline tips and tricks for those who may consider incorporating government-level linked data or simulation models into large program or policy evaluations in the future. It will cover areas such as: where to begin, navigating the data and key tips for working with analysts.

Chair

Claire Grealy

Director, Rooftop Social

Speakers

Gabby Lindsay-Smith

Thursday September 19, 2024 3:30pm - 4:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Education and training, Health and disability, Public administration and safety

3:30pm AEST

The evolution of evaluation: Retracing our steps in evaluation theory to prepare for the future

Thursday September 19, 2024 3:30pm - 4:30pm AEST

104

Authors: James Ong (University of Melbourne)

As new people enter the evaluation field and as evaluation marches forward into the future, it is important to learn from evaluation theorists that have come before us. My Ignite presentation will argue that modern evaluation is built on evaluation theory, and make the call for evaluators of all levels to learn evaluation theory to:

Appreciate how evaluation has evolved;
Strengthen their evaluation practice; and
Navigate themselves around an ever-changing evaluation landscape.

Chair

Claire Grealy

Director, Rooftop Social

Speakers

James Ong

Research Assistant (Evaluations), University of Melbourne

My name is James Ong. I am an Autistic program evaluator working at the University of Melbourne, where I work on evaluation and implementation projects in various public health projects such as the AusPathoGen program and the SPARK initiative. I not only have a strong theoretical... Read More →

Thursday September 19, 2024 3:30pm - 4:30pm AEST
104 113 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Education and training, International development, Public administration and safety, NFP or Philanthropy

1:30pm AEST

Investment logic mapping or evaluation logic modelling? Similarities and differences.

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Charlie TULLOCH (Policy Performance)

Evaluation logic modelling is a frequently used technique, looking at such things as inputs, activities, outputs and outcomes.
Since the early 2000s, the Department of Treasury and Finance (Victoria) has used an adapted logic modelling format called Investment Logic Mapping (ILM). It is now used nation-wide and internationally to support resource allocation planning, along with stakeholder engagement.

ILMs and evaluation logic modelling have many similarities, but some major differences.

This ignite presentation will compare and contrast both these tools, and describe when and why to use each.

Attendees will very quickly understand the main similarities and differences, their advantages and drawbacks.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Carina has over 15 years of professional evaluation experience working at the practitioner level with grassroots community groups, working within State government policy levels and consulting for government and not-for-profit sectors. She has a good understanding of local, state and... Read More →

Speakers

Charlie Tulloch

Director, Policy Performance

Policy Performance is a proud conference sponsor! Charlie delivers evaluation projects, capability building support and drives public sector improvement. Charlie loves to help those who are new to evaluation or transitioning from related disciplines. He is a past AES Board member... Read More →

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational
Industry Public administration and safety

1:30pm AEST

Measuring Impact Through Storytelling: using Most Significant Change to evaluate the effectiveness of QHub for LGBTIQA+ young people.

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Gina Mancuso (Drummond Street Services), Arielle Donnelly (Drummond Street Services, AU)

LGBTIQA+ young people experience discrimination and marginalisation which contribute to poorer mental and physical health outcomes, compared to the general population. QHub is an initiative that creates safe spaces, offers mental health and well-being services, and provides outreach tailored for LGBTIQA+ young people in Western Victoria and the Surf Coast. QHub provides LGBTIQA+ young people and their families/carers with welcoming, inclusive and integrated support, as well as opportunities to connect with peers and older role models. This presentation will outline how the collection and selection of stories of change (Most Significant Change) is helping us evaluate the impact of QHub.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Speakers

Gina Mancuso

Arielle Donnelly

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Community and social services, Health and disability, NFP or Philanthropy

1:30pm AEST

Program Evaluation Fundamentals in the NSW Department of Planning, Housing and Infrastructure: An eLearning course on evaluation

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Anabelle (Pin-Ju) Chen (NSW Department of Planning, Housing and Infrastructure)

Introducing Program Evaluation Fundamentals (PEF) in the NSW Department of Planning, Housing and Infrastructure, an eLearning course designed to facilitate a coherent journey of learning within the department. With learning and adapting together in mind, the design of PEF empowers individuals at all levels to navigate the fundamentals of evaluation. Through interactive modules, learners will understand key evaluation concepts and cultivate best practices. PEF promotes transformative growth by emphasising foundational evaluation knowledge. By leveraging PEF, we can shift our approach, embrace innovation, and advance the field of evaluation across the public sector, fostering a supportive community of forward-thinking evaluators.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Speakers

Anabelle (Pin-Ju) Chen

Senior Analyst, Evidence and Evaluation, NSW Department of Planning, Housing and Infrastructure

Anabelle (Pin-Ju) Chen is a distinguished senior analyst hailing from Taiwan, with a global perspective on evaluation, data analysis, and project management. Having studied in Taiwan, the United Kingdom, and Australia, Anabelle brings a diverse range of experiences and insights to... Read More →

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational
Industry Economic development tourism and infrastructure, Education and training, Information media and telecommunications, Public administration and safety

1:30pm AEST

Putting values on the evaluation journey map

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Samantha Abbato (Visual Insights People)

Values guide all evaluation processes, methods and judgements. Although evaluators are often not aware of the values shaping their work and can't readily name them, they know when we are straying off their values path through the experience of conflict or unease. Reflecting on the evaluation literature and two decades of evaluation practice using a 'values' perspective, it is argued that there has never been a more important time to build values literacy. This presentation demonstrates how values literacy can guide conversations with yourself, your team and others and provide signposting and illumination of a more rigorous and ethical evaluation journey.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Speakers

Samantha Abbato

Director, Visual Insights People

My twenty-plus years of evaluation experience are built on academic training in qualitative and quantitative disciplines, including mathematics, health science, epidemiology, biostatistics, and medical anthropology. I am passionate about effective communication and evaluation capacity-building... Read More →

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Intermediate & Advanced
Industry Community and social services, Education and training, Health and disability

1:30pm AEST

Squaring up with rubrics

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Josh Duyker (Centre for Evaluation and Research Evidence, Victorian Department of Health)

Much like Felix the Cat, evaluators have a bag of tricks to get us out of sticky situations. But when you are staring face to face with a complex evaluand, juggling tricky stakeholders whist sat on a mountain of data, it's not always clear what 'trick' you need! One twisted potential solution is the colourful, yet humble rubric. In this reflective practice ignite presentation, I will guide you through our journey of using rubrics as a tool to way find through an evaluation, and our key takeaways in how rubrics can support evaluators to make comprehensive and balanced evaluative judgements.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Speakers

Josh Duyker

Evaluation and Research Officer, Centre for Evaluation and Research Evidence

I am an emerging evaluator, currently working at the Centre for Evaluation and Research Evidence in the Victorian Department of Health. I've completed a Master of Public Health and am embarking on a Masters of Evaluation. Through roles in the not-for-profit sector and my studies... Read More →

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Education and training, Environment and climate change, Health and disability, International development, Public administration and safety

1:30pm AEST

Voices of the Future: Elevating First Nations Leadership in the Evolution of Educational Excellence

Friday September 20, 2024 1:30pm - 2:30pm AEST

103

Authors: Skye Trudgett (Kowa ),Sharmay Brierley (Kowa, AU)

This ignite presentation will delve into the pioneering evaluation within the education sector, where a series of education initiatives were designed and implemented by Aboriginal Community Controlled Organisations (ACCO's) and mainstream Education partners to uplift and support young First Nations peoples. We will uncover how the initiative's evaluation framework was revolutionarily constructed with First Nations communities at its heart, applying the reimagining evaluation framework, utilising diverse data collection methods and producing Community Reports that reflect First Nations experiences and voices.

Attendees will be guided through the evaluative journey, showcasing the incorporation of wisdom to demonstrate the profound value of community-delivered initiatives that contribute to change. The session will highlight the success stories and learnings, emphasising how this approach not only benefits the current generation but also lays the groundwork for the prosperity of future generations.

Chair

Carina Calzoni

Managing Director, Clear Horizon Consulting

Speakers

Sharmay Brierley

Consultant, Kowa Collaboration

Sharmay is a proud Yuin woman and project lead at Kowa with prior experience supporting First Nations peoples across human services sectors.As a proud First Nations woman, and through lived experience, Sharmay has a strong understanding of the many challenges faced by First Nations... Read More →

Skye Trudgett

CEO, Kowa

Skye is a Gamilaroi researcher who has contributed to numerous evaluations and research projects including place-based, systems change and government initiatives. Skye has a particular interest in Indigenous Data Sovereignty & Governance and seeks to support the practical application... Read More →

Friday September 20, 2024 1:30pm - 2:30pm AEST
103 110 Convention Centre Pl, South Wharf VIC 3006, Australia

Ignite session

Format Ignite presentation
Level Foundational & Intermediate
Industry Education and training, Other