This is an example of how the Toolkit would be used to evaluate a reading group for adults. The evaluation question was 'What impact does taking part in the reading group have on group members?' The evaluation has not actually been carried out, but the example is based on experience.
Step 1Identify your outcomes
Chosen outcomes: Reading frequency, attitudes to reading, empathy, mental health
Step 2Design the methods
A before and after survey with reading group members supplemented by interviews
Step 3Choose your measures
Questions chosen from relevant section which are designed for use with adults and the right length
Step 4Design your materials and collect the data
Survey constructed with additional demographic/experience questions, tested internally then sent out to participants online
Step 5Analyse and learn
Greater reading enjoyment and decreased levels of anxiety found, but no increase in reading frequency. Findings used to make the case for targeted reading groups
Step 1: Identify your outcomes
The things that the reading group aims to change for participants were listed (the outcomes). This included: reading behaviour, attitudes to reading, mental health, empathy, self-esteem and communication skills. The Reading Outcomes Framework was used as a guide to the possible outcomes of reading and to clarify the definitions of these terms. Possible unintended outcomes were considered as well, such as the possibility of a reduction in reading frequency and enjoyment if the specific books being read were too challenging. It wasn't possible to collect data on all potential outcomes so the list was refined by:
- Focusing on the most important intended outcomes of the programme.
- Focusing on the outcomes that it was most important to have evidence about to make the case for investment and to inform programme development.
- Considering existing evidence about whether each outcome was likely to result from the reading group, for example, using anecdotal feedback and research conducted by others.
The agreed outcomes to measure were: reading frequency, attitudes to reading, empathy and mental health.
Step 2: Design the methods
For this research, a 'before' and an 'after' survey were used: reading group members were asked a number of questions when they joined the group with the same questions repeated after they had attended 12 reading group sessions, to see if there had been any changes.
A small number of participants were then interviewed to explore their experiences in more depth and find out if there had been other impacts which had been valuable to them but not picked up by the survey.
Other options considered were a one-off survey, but it was felt this would not give enough objective evidence about the change that members had experienced over time (impact). Observations of the reading group sessions were also considered but there was not a staff member available to do this.
Step 3: Choose your measures
Using the 'questions' section, the questions for the relevant outcomes were reviewed, and those that were felt to be the right length and style and for the target audience (adults) were selected. The following questions were used:
How much do you enjoy reading? Source: Bookbuzz student survey developed by BookTrust and Chatterbooks survey developed by NLT and The Reading Agency.
How often do you read? Source: Chatterbooks survey (2015) developed by the NLT and The Reading Agency.
Overall, how satisfied are you with your life nowadays? Source: Labour Force Survey (2015).
Basic Empathy Scale e.g. I can often understand how people are feeling even before they tell me. Source: Basic Empathy Scale (2013) developed by Carré et al, University of Reims Champagne-Ardenne.
Step 4: Design your materials and collect the data
The questions were structured into a survey. Questions were added about demographics (age, gender, employment status) and about the experience of taking part in the reading group (number of sessions attended, level of enjoyment and suggestions for improvements).
The wording was reviewed to help the respondent navigate the survey and to ensure consistency. Introductory text was included stating how the data would be used, that responses would be anonymous and confidential, no personal data would be collected or stored and that participation was voluntary. The survey was checked for length to ensure it was not off-putting.
Another member of staff (who was not involved in the group or the evaluation) tested the survey to make sure that the instructions and flow of questions worked and the meaning of each question was clear, unambiguous and not open to different interpretation.
SurveyMonkey was used to collect the data; with paper surveys also available for those without internet access (a staff member then inputted the data). Clear deadlines for responses were given and an incentive of entering a prize draw for a £10 high street voucher was used to encourage responses.
Step 5: Analyse and learn
The results were analysed in SurveyMonkey and via Excel, comparing the responses from the 'before' survey to the 'after' survey to see if things had changed either positively or negatively.
The results from the survey showed that for the majority of members their enjoyment of reading had increased and levels of anxiety had decreased for members. The findings from the interviews suggested that being able to talk about feelings and experiences in relation to the reading in a supportive environment had been an important factor in reducing anxiety. The frequency of reading had not changed for group members and analysis suggested that this was because group members were already reading frequently before they joined the reading group.
A short report was written highlighting the outcomes of the reading group and making the case for continuing to allocate staff time to support reading groups. The report referenced other research from the Toolkit about the benefits of reading for pleasure, to put the findings in context.
The research findings were used to improve the reading groups: it was found that people would prefer to have more say in the titles that they read so this was changed for future meetings. Group members tended to already be frequent readers so the promotion of the group was adapted to attract less frequent readers as well.
The report was also used to approach a local charity and a grant was secured to run new reading groups targeted specifically for people diagnosed with anxiety and/or depression referred through local GP and health services.