Tünde Varga-Atkins, Jaye McIsaac and Ian Willis from the Educational Development Division present their findings from trying a new approach to focus groups.
If you want to involve students in curriculum development then here is a method that seems to work for us. We have explored combining a focus group with Nominal Group Technique (a process for problem identification and group decision making) in one student evaluation session. We named our approach the ‘Nominal Focus Group’.
This combination gives the benefits of both: the in-depth discussion of a Focus Group and the prioritising of results of Nominal Group Technique. In our work as Educational Developers working with academic staff on curriculum enhancements, results from a Nominal Focus Group provided us with rich data and actionable outcomes that were used to make informed curriculum enhancements for the programme teams. The process also increased students’ feelings of ownership. In a Higher Education era focused on the student voice, these benefits seem particularly appealing.
A bit about focus groups
Sometimes, certain focus group members can dominate the discussion and this may skew the recommendations made by students. This is the sort of bias that the process of the Nominal Group Technique is meant to design out.
- Advantages: ability to cover a number of questions in the session (up to 5-6 main topics), peer suggestions can help jog memory of others;
- Disadvantages: potential bias of vocal members, facilitator makes judgement about importance of responses and suggestions.
A bit about the Nominal Group Technique
The Nominal Group Technique has been developed by Delbecq and Van de Ven (1971). In a Nominal Group Technique session used for curriculum enhancement, students are usually asked one or two questions. For instance, “How could we improve this module?” Every participant is asked to write down their response individually on a post-it note (see Figure 2, stage 1). Then they read it out aloud and all the responses are placed on a board for all to see.
The group then discuss each response and merge those that are the same (stage 2). In the final stage, each participant is asked to rank the merged responses in order of importance (stage 3). Responses are then prioritised. This makes it a democratic process as the group’s decision is based on the majority of votes. The outcome of the session is a ranked list of items.
- Advantages: everyone has an equal input, responses are prioritised in the form of an actionable list;
- Disadvantages: ability to ask only up to 1-2 questions, students make suggestions starting ‘cold’ without peer discussion.
What does a combined Nominal + Focus Group look like?
The Nominal Focus Group is our way of maximising the benefits of both the Focus Group and the Nominal Group Technique. All we do is conduct a session, starting with a focus group discussion along the specified topics (stage 1). In the final part of the session, we conduct a mini-Nominal Group Technique focusing on one or two questions (stage 2). A typical question could be, “what needs to change in X?”
This additional result of a Nominal Focus Group is what makes the combination extremely powerful. It shows the group decision on aspects of a module or programme that students like to change in the form of an actionable list. This is in addition of the transcript and the focus group report.
- Advantages: reduces potential participant bias, everyone has an equal input, responses are prioritised in the form of an actionable list, possible to cover a range of questions (more than two), students can help each other remember experiences
- Disadvantages: the only disadvantage we can think of concerns time management: if only restricted time is available (e.g. 60 minutes), the facilitator needs to have good time management skills to cover both stages in full.
What have we found out when we evaluated the combined Nominal Focus Group approach?
The student perspective:
- Students had plenty to say. They clearly enjoyed the Nominal Focus Group evaluation sessions and some of them asked “will we have more like these?”. They found that they were better able to formulate their ideas with the help of others; “good to hear other students’ opinions to help expand my own.”
- The fact that students had an open discussion and produced a tangible outcome and an actionable list of priorities clearly led to a sense of ownership by the students. We found this to be a key benefit of the Nominal Focus Group approach.
- Quality. The dynamic between the two stages, and so, the power of the combined technique was clearly evident to those students who said that both stages contributed to the quality of results: the relationship between the two stages was that of (1) opinion formulation and (2) making opinions count within the group members: “The open discussion helped to get me thinking of my own experience of feedback, whilst writing my opinion on the post-it note [voting] helped get my opinion across”.
- Others commented in the post-session survey: “[the session was] Very good. Was nice to see if the University cared about the problems we are all having.” This latter comment signalled a significant unintended outcome. Students appeared to feel that the university cared about them and valued their opinions based on their being invited to the sessions and on the nature of the process itself
The programme team perspective:
- The combined Nominal Focus Group sessions were reported to have given the teaching team a rich picture of student experiences, not usually possible with traditional module or programme evaluation surveys.
- Things came up, and then in the end they came all together again. That was very helpful. It kind of made it easier for us to identify what the students thought was the most important thing (interview).
- As a result of the quantitative element in the nominal, ranking stage, the combined approach brings the potential for scalability. Finally, there was some indication that staff from science disciplines found the quantitative output of the Nominal Focus Group more convincing.
Facilitator time involved
The resource needs in the data collection stage were similar, whether in Focus Groups, Nominal Group Technique, or the Nominal Focus Group. We found that there were time gains in the analysis stage since the ranking stage in the Nominal Group Technique began the data analysis for us.
Final words: our take-away message
We hope that the student and programme team feedback signal evidence that our combined Nominal Focus Group process produced data that was both ‘owned’ by students and was readily actionable by programme teams. One notable benefit was that it generated both qualitative and quantitative data. Quantitative data is likely to indicate the priorities as seen by students, and perhaps be more convincing for many disciplines. Qualitative data provide greater insight and understanding for programme teams engaged in curriculum review.
In this blog post, we hope to have demonstrated that combining two different ingredients into a two-course meal, Focus Group and Nominal Group Technique, is more pleasing to the palate of student evaluation than serving either one alone! If you would like to read more about our study, please check out our journal article entitled ‘Focus Group meets Nominal Group Technique: an effective combination for student evaluation?’ http://dx.doi.org/10.1080/14703297.2015.1058721
Tünde Varga-Atkins: as a Learning Technologist, I support staff across the university in their e-learning development, developing a specialism in research and evaluation in technology-enhanced learning. I also co-convene a regional research group, ELESIG NW [elesig.ning.com/] with other UK NW colleagues.
Jaye McIsaac: as an Educational Developer I am interested in learning design and curriculum development. I work with programme teams and contribute to institutional initiatives such as the implementation of the Postgraduate Taught Curriculum Review.
Ian Willis: I am the Head of the Educational Development Division and teach on postgraduate programmes in learning and teaching and support pedagogic research projects. I also teach on the University of Liverpool’s online professional doctorate in Higher Education Leadership (EdD) and I am the UK Director of a collaborative project to ‘Enhance learning and teaching in medical education, in Punjab’, Pakistan.