Evaluating the Wesley Mission: In charge of my money

1. Description of the programme
2. What are the outcomes?
3. Key findings
4. Points to consider

• Description of the programme
• What are the outcomes?
• Key findings
• Points to consider
• Full report

Description of the programme

The Wesley Mission ‘In Charge of My Money’ programme has been delivered to over 4,000 people since its inception in August 2011. It aims to improve the financial literacy of vulnerable groups with problems including addiction, homelessness, unemployment, domestic violence, low income and health issues. The program provides information on how to better manage money; it is designed to be practical and easy to understand, aiming to provide participants with relevant skills to manage their money and make effective financial decisions. It is delivered as three independent modules:

• Know it – Your income and expenses
• Do it – Managing your money – covering goals saving and budgeting
• Review it – Managing your debt – covering borrowing and debt

The key message to participants is to apply what they learn to everyday life. It is delivered by qualified training professionals, usually financial counsellors. The program can be delivered in one day but is most commonly delivered in three weekly sessions of three hours each. To allow time for questionnaire completion, the program was actually delivered in four weekly instalments for the purpose of this evaluation. Text messages affirming positive money behaviours every 3-4 weeks to half the participants to test if reminders/nudges had an impact on financial behaviours of participants. With delivery by financial counsellors, participants are encouraged to access financial counselling and support.

What are the outcomes?

• Changes in financial behaviours:
  • Change in money management behaviour
  • Change in level of saving behaviour
  • Change in degree of planning behaviour
  • Change in level of monitoring expenses
• Change in reaching financial goals:
  • How much the programme assisted in achieving goals
  • The extent of progress towards the specific goal identified during the programme
• Change in financial strain:
  • The number of problems due to lack of money
  • The degree of difficulty in paying bills
  • How much money is left over by the next payday
  • Frequency of asking family for money
  • Number of financial products i.e. financial inclusion
  • Change in level of debt
• Outcomes deuced from participants’ qualitative responses:
  • The programme elements that were most relevant to, and useful for, programme attendees
  • Programme components that were best remembered
  • How the programme can be improved

Key findings

• There is quantitative and qualitative evidence that the program content and delivery is relevant for some of the most socially and economically disadvantaged and vulnerable people – it is rated highly by participants and, several months after participation would be prepared to recommend the program as easy to understand, relevant and helpful.
• The program is evidence-based, conforming to what has been identified as best practice in literature to date including relevant tailoring, delivery style, delivery by knowledgeable and non-judgemental trainers, at an opportune time, adhering to principles of adult learning and empowering program participants.
• The program facilitates sustained improvements in financial behaviours. Statistical analyses show substantial improvement in financial behaviours from before the program to immediately after and then several months later. Notably, improvements were observed in savings behaviour, planning, monitoring of expenses and other money behaviours. There was significant improvement between behaviour before the programme and the end; there was evidence that improvements were retained over the following 5-7 months. 59% of participants had no financial goal before the start of the programme but had one at the end. 56% said the programme helped them in achieving their goal (a lot or very much).
• There is evidence of retention of knowledge and understanding several months after completion of the program. A very large statistically significant improvement was found in general financial knowledge and personal financial knowledge – even 5-7 months after the course. About two thirds (65%) remembered that the programme was about budgeting and money management generally and a quarter recalled more specific information about different types of loans, how to manage debt and the danger of payday loans. A quarter also remembered specific advice about being mindful of spending habits. Almost a quarter remembered tips about how to save. Only six participants (11%) remembered little or nothing from the programme.
• Despite some issues in maintaining text contact with a population without reliable mobile phone access, the study does offer evidence that text ‘nudges’ had a significant and positive impact on participants’ financial behaviours. These did not extend to all outcomes e.g. on financial strain, achieving goals or reducing debt. Small samples lead the authors to conclude that more research is required in this area. Almost half of participants who received nudges said they were only a little bit useful. Only 5 participants (20%) said the nudges were very or extremely useful. Despite this the statistical analysis suggests that text messages did have some sustained positive impacts on money behaviour, saving behaviour, monitoring expenses and emotional spending. No positive correlations were detected between text messaging and planning behaviour, number of problems associated with lack of money, difficulty paying bills, frequency of having money left over or frequency of asking family for money.
• In addition to the key areas measured, the study noted some other program benefits in improvements in attitudes, self-efficacy, emotional spending and financial strain. It notes, however, that the links between these improvements and other key findings are not necessarily causal relationships.
• No demographic factors were noted associated with improvements in financial behaviours – so program impact is not related to gender, age, nationality, education level or income. The conclusion is therefore that the program may be relevant to a broad range of people within vulnerable populations.

Points to consider

• Methodological limitations:
  • The research area and the environment studied raise challenges relating to sample sizes and reliability.
  • The subject matter is difficult for some and responses in some measures required incentivisation.
  • There is significant attrition of the sample when it comes to follow up research.
  • The research on the efficacy of text message nudging suffered even more from these deficiencies because of the unreliable mobile phone access of some respondents.
  • The fact that the study was based on real world rather than experimental approaches may also be challenged by some commentators as there was no control group or comparison group against which to measure outcomes.
  • The study did not collect information on changes in participants’ non-financial circumstances over time; it is therefore not possible to isolate changes resulting from the program from effects of changes in personal circumstances.
• Generalisability/transferability:
  • The study relates to individuals in vulnerable populations and findings would not be generalizable to the population as a whole.
• Applicability:
  • Notwithstanding the methodological challenges, the evaluation suggests that the programme itself and the approach are capable of wider application among vulnerable populations.

Full report

Evaluating the Wesley Mission: In charge of my money - full report