evaluation

Evaluation of MAS Budget Planner online tool

Evidence type: Evaluation i

Description of the programme
The study
What are the outcomes?
Key findings
Points to consider

Description of the programme

The Money Advice Service is an independent service set up by the UK government to help people optimise their money by making financially capable decisions. A range of information and tools are available from their website, including an online Budget Planner. This tool is accessible to all and allows consumers to enter their income and expenditure details. It provides a snapshot of their finances and breaks down spending into categories such as household bills and leisure. Users are then directed to tips and advice on how to manage their budget and make the most of the money they have, or to deal with a budget deficit.

The study

The main objective of the evaluation is to understand the extent to which financial capability outcomes are improved through the use of the Budget Planner, and if these outcomes could be further improved. Winning Moves were commissioned to conduct the evaluation, which was carried out between September 2017 and May 2018.

There were two key stages to the research:

Qualitative interviews exploring how people use the Budget Planner and what outcomes they experience. A sample of ‘squeezed parents’ (as defined in MAS’ market segmentation) were recruited from a consumer panel. The participants were asked to use the Budget Planner (or the associated Christmas Money Planner), after which the interviews took place. In wave one (October/November 2017) 32 participants were interviewed, while 26 of the same participants were re-interviewed in wave two (February 2018). Findings were supplemented with data captured from a ‘pop-up’ survey built into the Budget Planner (which received 448 useable responses); and from in depth interviews with five Budget Planner users recruited through these pop-up surveys.
Insights from customer data and web analytics: A wide range of data is captured from users of the Budget Planner, and part of this evaluation considers whether this information can be a useful indicator of the success of the tool.

A Theory of Change was also drafted, to summarise understanding of how the budget tool currently works, and the assumptions that lie behind it. This initial theory building was informed by a rapid evidence review as well as discussion with a cross-section of staff from the Money Advice Service at a half-day workshop.

What are the outcomes?

Budgeting and Keeping Track
Savings
Saving Mindset
Financial Confidence

Key findings

The evaluation found that in certain circumstances the Budget Planner helps people make significant changes to their income and expenditure.
Some participants were beginning to save as a result of the intervention.
People who had more money available to them after using the Budget Planner started to feel more motivated and empowered to experiment with alternative financial behaviours or set higher savings targets.
However, using the Budget Planner was typically short-term behaviour, with very few participants having updated or revisited the Budget Planner when interviewed three months later.
Due to the duration of the evaluation and the evaluation approach, it was not possible to draw any strong conclusions about the medium and longer term outcomes hypothesised in the Theory of Change. However, there was some indication that some of these outcomes were emerging. Examples include sustained reductions in spending and increased shopping around at the follow-up interview stage.
Whether users made changes seemed to depend on three underlying factors:
=The need to budget in the first place
=Whether the tool was completed accurately
=Whether users were shocked by their level of expenditure and recognised opportunities to make cutbacks
The data analysis suggested that the income profile of users was similar to that of the UK population.
Half of the visitors came from within the MAS site itself.
Approximately a third of visitors to the landing page leave at this stage. The remaining two-thirds proceed through the tool, with reducing drop-off on completion of each stage.

Recommendations to MAS emerging from the evaluation include:

Increase engagement with the Budget Planner by linking budgeting to financial goals
Ensure users understand the benefits of accurately completing the tool
Links to further advice and support should be tailored to the user and more visible
Consider collecting some basic personal data that can be used to ensure the results are more bespoke.

Points to consider

Methodological strengths/limitations:

Full details of the recruitment process for the interviews are given in the appendix, and the sampling strategy appears robust.
This study only tentatively explored causality (i.e. understanding the extent to which the changes observed were directly caused by the Budget Planner tool), by exploring this with participants and conducting a light-touch Contribution Analysis. There is no comparison group to benchmark the observed changes against (i.e. a group where there was no intervention).

Generalisability/ transferability:

The evaluation relies mostly on qualitative data, and though the methodology seems strong, caution must be taken in generalising too much from these results.

Relevance:

This report is relevant to all stakeholders and policymakers with an interest in evaluating the success of online tools, and to those interested in developing digital tools aimed at enhancing financial capability.