[Originally published on the Australian Government Public Sector Innovation Network under a Creative Commons 3.0 BY AU licence]
In this post, we take a look at another difficult aspect of developing a business case[1. This post follows on from two other business case posts – one on dealing with risk and another on identifying resources required.] – identifying the outcomes and being able to say what success (or failure) will look like in implementing the idea. You should know what you hope will happen, but how will you assess whether it has worked, and whether its implementation should continue? In APS jargon, what are the key performance indicators or the outcome measures?
It may be helpful to consider the business case for the proposal as testing an experimental hypothesis – what is it you are trying to test and how will you measure the results of the experiment? What problem will this idea address and how will you know it has worked?
For some innovations, it will be quite clear what success will look like. There may be a clear problem and the proposed action/initiative will either fix the problem or it won’t. Even better, that assessment may be easily done in a specific (and short) time frame. If it fails, you can move on and try another approach. But other problems will be more complicated and there won’t necessarily be a clear relationship between the idea and the outcomes.
Some of the difficult questions that can arise include:
-
- The innovation may have a number of parts and be happening at the same time as other developments (if a problem is big enough, there are often many things tried at the same time). How will you be able to tell whether it was this particular innovation that made the difference? (One way to address this may be to conduct a randomised policy trial where a random assignment of participants to control and treatment groups provides a counterfactual of whether the intervention worked or not, as is often done in medicine.)
-
- Is there agreement/consensus from relevant stakeholders about what a positive outcome will be or look like? Is it possible the results of the experiment could be interpreted in a different way? If yes, work may need to be done on that before you apply your idea.
-
- Many innovations will have poor results when first tried as they are new and different. It will likely take time and investment to get better at the innovation, rather than being implemented perfectly straight away (e.g. see discussion of the S-Curve). Has this been factored into the consideration of the idea? What if the outcomes are at first worse than what is currently in place – how will you know if the initiative will get better, and how can that be communicated to stakeholders?
Another point is that you may not know what success will really look like. The idea may be very experimental and you cannot predict how it will fix the problem, but you suspect it will contribute to fixing it or you think it will lead to beneficial outcomes but they may be a bit nebulous. In these instances you may want to think about the following:
-
- What are the reasons for thinking that the experiment is worth trying?
-
- How can the idea be implemented in a controlled or limited fashion (you may not want the experiment to be too big)?
-
- What will failure look like? If you cannot be sure what success will look like, can you be certain what failure will be? How will you know the experiment has not worked?
The Australian National Audit Office provide some tips on evaluation for innovative ideas as well. In Innovation in the Public Sector: Enabling Better Performance, Driving New Directions it notes “Performance information and its availability to agency managers, politicians and Australian citizens, contribute to learning, innovation and improvement. An appropriately tailored evaluation strategy includes the collection and analysis of performance information that provides:
-
- an early indication of policy/delivery effectiveness; and
-
- longer-term evaluation of outcomes.
Early indicators can be used to detect any significant problems and enable corrective action to be taken. Longer-term evaluation can be used to better understand the details of the impact of policy, service delivery and regulatory changes.” The better practice guide goes on to provide some advice on these steps that you may want to consider in your business case for your idea.
How will you know if your idea has worked? How do you communicate that?