<Program name> evaluation work plan

Document title	<Program name> evaluation work plan
Contact details
Approved by
Date approved
Document review	<Eg: Annually>
TRM number	<NA if not required>

Version	Date	Author	Changes made
<X.X>	<Date Month Year>	<Firstname Lastname>	<Eg: First version>

Acronyms	Full form
<Eg: NT>	<Eg: Northern Territory>

This template has been adapted from the Evaluation Strategy Template developed by the Australian Government Department of Industry, Science, Energy and Resources, available at https://www.policyhub.gov.au/resources/getting-evaluation-ready-process-diis

Contents

1. Introduction	4

2. Program overview	4

3. Evaluation overview	5

4. Evaluation roles and responsibilities	5

5. Evaluation methodology	7

5.1. Types of evaluation	7

Process	7

Outcome	7

Impact	7

5.2. Program logic	8

5.3. Key evaluation questions	8

5.4. Data matrix	9

6. Budget and resourcing	9

7. Stakeholder engagement	9

8. Ethical considerations	10

9. Evaluation risks	10

10. Reviewing the evaluation work plan	10

Appendix A: Program logic	11

Appendix B: Data matrix	12

Introduction

[Section 2 of the Program Evaluation Toolkit is structured to mirror this template to give additional guidance, section by section. Delete this text and other guidance text in red.]

This evaluation work plan outlines future evaluation activity for <Program name> evaluation work plan over the next [x] years.

The work plan proposes a [insert summary of planned evaluations e.g. process evaluation in x year and an outcome evaluation in x year].The work plan details the reasons behind the evaluation activity planned, and outlines the scope of each evaluation, the evaluation questions, and the data that is available or will need to be collected to answer these questions.

[Delete if there is only one evaluation planned]. This evaluation work plan shows how different evaluation questions will be addressed over the next [x] years. It will inform the terms of reference for each evaluation.

This evaluation work plan has been prepared by taking into account the strategic importance of the program and the expected level of resourcing for evaluation. It has been developed by the [insert policy/program team names] in collaboration with the [insert name additional teams if applicable, otherwise delete]. The strategy has been endorsed by the [insert title and name of approver] and will be submitted to the Program Evaluation Unit at the Department of Treasury and Finance.

As programs may change over time, this should be considered a ‘living document’. This work plan will be reviewed periodically or in response to significant program events by the program manager. The Program Evaluation Unit will be provided with updated versions as they are developed. [Please ensure the version control box on page 2 of this template is updated as changes are made].

Program overview

[In this section explain why the program is needed, what it is aiming to achieve and how it is expected to impact demand on future government services. Describe how the program operates, its funding, governance etc. Some example text follows. Adjust this section as best fits your program. For example, you may add current sensitivities, such as whether the program has ceased or has a finite life, proposed changes to the program etc. Aim for approximately half to one page in length and put additional information (such as a needs assessment) into a brief appendix if required.]

<Program name> evaluation work plan was an initiative of the Northern Territory Government in [insert year]. It was implemented as part of the [insert Agenda/Strategy if relevant] with the intention of [insert program objectives]. The program came into effect on [insert date]. Approved funding for the program will cease [insert year/date].

The program has been subject to [insert number] of previous reviews and evaluations. Most recently [list recent review titles and dates if applicable]. These reviews led to key changes to the program, most notably [outline key changes to program objectives, design, eligibility etc.]. For a full list of previous evaluations and reviews please see Appendix [X].

Evaluation overview

[If an evaluation overview has been completed as part of a Cabinet submission, it can be referred to in this section and attached as an appendix.]

Evaluation roles and responsibilities

[Use this table to outline the broad roles and responsibilities of different areas involved in your evaluation(s). You might include the evaluation manager, an evaluation steering committee, a consultant and the program evaluation unit. Contributors to involve in the decision-making process may include:

The program manager within the agency

An evaluation steering committee

A technical advisory group or a number of individual technical advisors (including service providers)

A community consultation committee or key informants from the community.

Further guidance is in the program evaluation toolkit.

Table 1 Evaluation roles and responsibilities [adjust this table to ensure it is appropriate for the size and complexity of the evaluation plan]

Area/Committee	Responsibility
The program manager	Educate the external evaluator(s) about: the program's objectives, operations and intended beneficiaries; the expectations about the evaluation and any relevant organisational background. Provide input and/or collate feedback on the evaluation plan. Specify the reporting requirements in terms of progress in the implementation of the evaluation (including reporting of important challenges and their resolution or which potential issues need to be raised for decision making elsewhere) Specify what is expected to be included in the formal evaluation report(s). Keep the evaluator(s) appraised of any changes in the program's operations or evaluation context. Provide regular updates on the evaluation process to all staff. Monitor the implementation of the evaluation including completion of milestones/deliverables. Facilitate program staff involvement in the evaluation, where relevant and agreed. Serve as the trouble-shooter, resolving problems or locating help to resolve them
[Program Name] Evaluation Steering Committee	Endorse the Terms of Reference and evaluation work plan Provide feedback on draft findings and recommendations and the draft evaluation report Chair of the Steering Committee to sign off on the final evaluation report Draft the evaluation Terms of Reference and evaluation plan for the evaluation Conduct, manage, or advise on evaluation activity as required
Evaluator (internal or external)	Develop an evaluation plan, in conjunction with the program manager. Provide monthly or quarterly progress reports on the implementation of the evaluation (written or in person). Attend evaluation meetings. Train data collectors on participant/case selection for sampling purposes, using data collection instruments, data quality assurance Ensure adherence to ethical standards adherence (e.g. confidentiality of data) during all phases of the evaluation. Oversee implementation of data collection such as: interviewing program staff and participants, conducting focus groups, observing service delivery activities, reviewing participant case records, developing data management procedures and tools (e.g. database), coding and cleaning data, analysing data. Write interim (quarterly, biannual, yearly) evaluation reports and the final evaluation report. Present findings.
Program Evaluation Unit	Review the final evaluation plan

[Briefly outline the decision-making process. This may include:

•	Decisions made based on support from the majority. In contentious decisions, it is important to be clear who is eligible to vote and whether proxy votes are allowed.

•	Decisions made based on reaching a consensus. In practical terms, that can mean giving all decision makers the right to veto.

•	Decisions made based on hierarchy (formal positions of authority).

Evaluation methodology

Types of evaluation

[Delete unnecessary evaluation sections and this text. While there are a number of different approaches to evaluation, the Program Evaluation Framework is based on three types, linked to the program lifecycle:

process evaluation:	considers program design and initial implementation (≤18 months)

outcome evaluation: considers program implementation (>2 years) and short to medium term outcomes

impact evaluation:	considers medium to long term outcomes (>3 years), whether the program contributed to the outcomes and represented value for money.

These three evaluation types address different questions at various stages of the program lifecycle, with each evaluation building on the evidence from the previous evaluation. Not all programs will require all three evaluation types. The evaluation overview, completed as part of the Cabinet submission process, will specify which evaluation types are necessary for each program.

As a general guide, outcome or impact evaluations of high tier programs should be externally evaluated. Further guidance is in the program evaluation toolkit.]

Process [Add year/s]

<Program name> evaluation work plan will have a process evaluation in [insert year]. [Add reasoning]. This evaluation will focus on the initial implementation of the program to allow decision makers to identify early issues regarding program administration and delivery, and take corrective action if necessary.

It is proposed that this evaluation be conducted by [select one: internal evaluators/ external evaluators/ a combination of internal and external evaluators]. [Explain reasoning for this decision.]

Outcome [Add year/s] [Delete if this evaluation is not required]

<Program name> evaluation work plan will have an outcome evaluation in [insert year]. [Add reasoning]. This evaluation will focus on testing the program’s performance framework, including the data sources, in order to assess whether they are providing the information required for both the ongoing management of the program and future impact evaluations. In doing so, this evaluation may be able to assess the performance of the program’s short term outcomes.

It is proposed that this evaluation be conducted by [select one: internal evaluators/ external evaluators/ a combination of internal and external evaluators]. [Explain reasoning for this decision.]

Impact [Add year/s] [Delete if this evaluation is not required]

<Program name> evaluation work plan will have an impact evaluation in [insert year]. [Add reasoning]. This evaluation will assess the impact of the program. It will seek to compare program outcomes with a prediction of what would have happened in absence of the program, it may include a counterfactual or cost benefit analysis [only include this after you have assessed the real possibility of this type of analysis taking place]. It is proposed that this evaluation be conducted by [select one: internal evaluators/ external evaluators/ a combination of internal and external evaluators]. [Explain reasoning for this decision.]

Program logic

[At the heart of each program is a ‘theory of change’ by which policy planners determine the outcomes sought and how that change can be achieved. A program logic visually represents the theory of change and describes how an intervention contributes to a chain of results flowing from the inputs and outputs to short, medium and long outcomes.]

A program logic (Appendix A) has been developed for <Program name> evaluation work plan to:

clarify and communicate program intentions and outcomes

demonstrate alignment between activities and objectives

explain causal assumptions and test if they are supported by evidence

identify relevant external factors that could influence outcomes (either positively or negatively)

identify key performance indicators for monitoring

identify gaps in available data and prioritise additional data collection

clarify the timing of outcomes and identify early indicators of progress or lack of progress in achieving results

focus evaluation questions

Key evaluation questions

[Across the program cycle, evaluations need to include a range of key questions that promote accountability for public funding and learning from program experiences. These questions need to align with the program logic and will form the basis of the Terms of Reference. Note that each evaluation does not need to address all the key evaluation questions – they can be spread out across the process, outcome and impact evaluations.

Questions should be tailored to the program – contact the Program Evaluation Unit for further guidance]

The key evaluation questions for the process evaluation are:

[Delete if outcome evaluation not planned] The key evaluation questions for the outcome evaluation are:

[Delete if impact evaluation not planned] The key evaluation questions for the impact evaluation are:

Key evaluation questions may be added to or amended closer to the individual evaluations to account for changes in the policy context, key stakeholders, or performance indicators.

Data matrix

The data matrix at Appendix B shows the data that will need to be collected to monitor and evaluate this program.

A mix of methods, both quantitative and qualitative, will be used to gather evidence to answer the key evaluation questions. [Change if this is not the case.]

Specific methodologies for the forthcoming evaluation[s] will be agreed by the [Evaluation Steering Committee] prior to the commencement of [each/the] evaluation.

Methodologies from the evaluations are expected to include: [add/change as necessary]:

Desktop research: a systematic review of program documents that may include program guidelines, executed grant agreements, program logic, policy papers, and program reporting and procedure manuals. This may also include a review of relevant reports and existing data.

Literature review: a systematic review of similar programs run in other jurisdictions, reviews or evaluations of similar programs, relevant journal research articles or media reports (with caution), and [other policy specific sources of information].

Semi-structured interviews with a range of stakeholders that may include face-to-face, telephone, or video-conferencing. With permission, interviews will be audio recorded to enable transcription and improve the accuracy of analysis. Stakeholders to be interviewed will be identified when refining the methodology of each evaluation in consultation with the Reference Group.

Surveys.

[inset name] data set.

Case studies of selected projects.

The evaluation will draw on data from [a sample of / all] [grant recipients/program participants].

[Add further information on methodology as required]

Budget and resourcing

The [Add evaluation type] Evaluation

It is estimated this evaluation will cost [insert amount], however it is difficult to accurately forecast costs for the evaluation before finalising the scope and approaching the market. In addition, [insert number] FTE are expected to be required across the department for [insert duration].

[Repeat above section for each expected evaluation]

Stakeholder engagement

[Briefly outline stakeholders who have an interest in the evaluation (including those who will be impacted by the evaluation) and consider the appropriate way to engage these stakeholders during evaluation planning, as part of the evaluation and after the evaluation is complete.]

Ethical considerations

We intend to ensure that the participants engaged throughout the evaluation process will be respected and protected by [add/change as necessary: adhering to the relevant ethical and cultural safety guidelines and seeking external ethics review, if required].

Evaluation risks

[This should articulate risks or limitations that the evaluation faces, not the risks of the program in general]

[If significant mitigatable risks are identified, the following risk assessment table below may be helpful to provide descriptions and controls; otherwise please delete]

Description	Consequence	Analysis	Current control
Likelihood	Consequence	Risk Rating
Example: Stakeholders may be unwilling to participate in research	Example: The evaluation would lack descriptive information about perceptions	Example: Likely	Example: Moderate	Example: Medium	Example: A variety of intercept surveys, focus groups, telephone, internet based surveys and information conversation methods will be used to improve access for a broad range of participants.

Use the likelihood and risk rating to determine the overall risk rating. Those that are high or extreme are likely to require closer monitoring than those that are moderate or low.

Consequence
Likelihood	Negligible	Low	Moderate	High	Extreme
Almost certain	Moderate	Moderate	High	Extreme	Extreme
Likely	Low	Moderate	High	High	Extreme
Possible	Low	Moderate	Moderate	High	Extreme
Unlikely	Low	Low	Moderate	High	High
Rare	Low	Low	Moderate	Moderate	High

[If appropriate after mitigation:] All risks are rated as either medium or minor and on this basis, they are accepted. The risks will be managed by routine procedures and regular monitoring to ensure that the risks are managed effectively and controls are reliable.)

[Further guidance is in the program evaluation toolkit].

Reviewing the evaluation work plan

This evaluation work plan has been reviewed by [inset relevant reviewer such as the Program Steering Committee and/or the Program Evaluation Unit].

Appendix A: Program logic

[Add inputs, outputs and outcomes and alter arrows as appropriate.]

Medium-term outcomes (<timeframe>) Assumptions Identify any assumptions that are being made about casual links between the program activities and outcomes. For example, people who attend a training session will change their behaviour as a result of the training session. External factors What other programs or strategies are targeting similar outcomes/impacts for similar target groups? For programs in remote communities, a good starting point might be to check the Bushtel website and consider other NTG, commonwealth or NGO interventions. Any other factors that may impact the economic, political, cultural or social context? Short-term outcomes (<timeframe>) The intended results of program outputs Long-term outcomes (Impacts) (<timeframe>) <Short-term outcome 1> < Short-term outcome 2> < Short-term outcome 3> <Long-term outcome 1> <Long-term outcome 2> Inputs Resources (funds, expertise, time) Outputs Products or services generated by program activities <Input 1> <Output 1> <Output 2> <Output 3> <Output 4> <Output 5> <Input 2> <Input 3> <Input 4> <Medium-term outcome 1> <Medium-term outcome 2> <Medium-term outcome 3>

Appendix B: Data matrix

[Add/remove rows to align with the program logic]

Key Evaluation Questions	Program logic alignment	Indicator Based on SMART principles – specific, measurable, attainable, relevant, time-bound	Baseline Baseline values prior to the intervention or values of comparator	Target Consider historical trends, current government priorities, agency capacity and demand	Data source Database, survey, interviews, administrative file etc. Include the agency/unit responsible for the data
Was [the program] implemented in accordance with the initial program design? If not, why not? Are there any adjustment to the implementation approach the needs to be made?	Outputs
<Output 1>
<Output 2>
<Output 3>
<Output 4>
<Output 5>
[TBD – program specific]	Short-term outcomes
<Short-term outcome 1>
<Short-term outcome 2>
<Short-term outcome 3>
[TBD – program specific]	Medium-term outcomes
<Medium-term outcome 1>
<Medium-term outcome 2>
<Medium-term outcome 3>
[TBD – program specific]	Long-term outcomes (Impacts)
<Long-term outcome 1>
<Long-term outcome 2>
What external factors have influenced the outcomes?	External factors	N/A
Have there been any unintended consequences, positive or negative?	Assumptions	N/A

For information about other evaluation types, please see the Better Evaluation website

Further information on these three evaluation types are in sections 3.2.1 to 3.2.3