Reading Outcomes Framework Toolkit

Practical example

This is an example of how the Toolkit would be used to evaluate a reading group for adults. The evaluation question was 'What impact does taking part in the reading group have on group members?' The evaluation has not actually been carried out, but the example is based on experience.

  1. Step 1

    Identify your outcomes
    Chosen outcomes: Reading frequency, attitudes to reading, empathy, mental health

  2. Step 2

    Design the methods
    A before and after survey with reading group members supplemented by interviews

  3. Step 3

    Choose your measures
    Questions chosen from relevant section which are designed for use with adults and the right length

  4. Step 4

    Design your materials and collect the data
    Survey constructed with additional demographic/experience questions, tested internally then sent out to participants online

  5. Step 5

    Analyse and learn
    Greater reading enjoyment and decreased levels of anxiety found, but no increase in reading frequency. Findings used to make the case for targeted reading groups

Step 1: Identify your outcomes

The things that the reading group aims to change for participants were listed (the outcomes). This included: reading behaviour, attitudes to reading, mental health, empathy, self-esteem and communication skills. The Reading Outcomes Framework was used as a guide to the possible outcomes of reading and to clarify the definitions of these terms. Possible unintended outcomes were considered as well, such as the possibility of a reduction in reading frequency and enjoyment if the specific books being read were too challenging. It wasn't possible to collect data on all potential outcomes so the list was refined by:

The agreed outcomes to measure were: reading frequency, attitudes to reading, empathy and mental health.

Step 2: Design the methods

For this research, a 'before' and an 'after' survey were used: reading group members were asked a number of questions when they joined the group with the same questions repeated after they had attended 12 reading group sessions, to see if there had been any changes.

A small number of participants were then interviewed to explore their experiences in more depth and find out if there had been other impacts which had been valuable to them but not picked up by the survey.

Other options considered were a one-off survey, but it was felt this would not give enough objective evidence about the change that members had experienced over time (impact). Observations of the reading group sessions were also considered but there was not a staff member available to do this.

Step 3: Choose your measures

Using the 'questions' section, the questions for the relevant outcomes were reviewed, and those that were felt to be the right length and style and for the target audience (adults) were selected. The following questions were used:

Step 4: Design your materials and collect the data

The questions were structured into a survey. Questions were added about demographics (age, gender, employment status) and about the experience of taking part in the reading group (number of sessions attended, level of enjoyment and suggestions for improvements).

The wording was reviewed to help the respondent navigate the survey and to ensure consistency. Introductory text was included stating how the data would be used, that responses would be anonymous and confidential, no personal data would be collected or stored and that participation was voluntary. The survey was checked for length to ensure it was not off-putting.

Another member of staff (who was not involved in the group or the evaluation) tested the survey to make sure that the instructions and flow of questions worked and the meaning of each question was clear, unambiguous and not open to different interpretation.

SurveyMonkey was used to collect the data; with paper surveys also available for those without internet access (a staff member then inputted the data). Clear deadlines for responses were given and an incentive of entering a prize draw for a £10 high street voucher was used to encourage responses.

Step 5: Analyse and learn

The results were analysed in SurveyMonkey and via Excel, comparing the responses from the 'before' survey to the 'after' survey to see if things had changed either positively or negatively.

The results from the survey showed that for the majority of members their enjoyment of reading had increased and levels of anxiety had decreased for members. The findings from the interviews suggested that being able to talk about feelings and experiences in relation to the reading in a supportive environment had been an important factor in reducing anxiety. The frequency of reading had not changed for group members and analysis suggested that this was because group members were already reading frequently before they joined the reading group.

A short report was written highlighting the outcomes of the reading group and making the case for continuing to allocate staff time to support reading groups. The report referenced other research from the Toolkit about the benefits of reading for pleasure, to put the findings in context.

The research findings were used to improve the reading groups: it was found that people would prefer to have more say in the titles that they read so this was changed for future meetings. Group members tended to already be frequent readers so the promotion of the group was adapted to attract less frequent readers as well.

The report was also used to approach a local charity and a grant was secured to run new reading groups targeted specifically for people diagnosed with anxiety and/or depression referred through local GP and health services.