6 A group self-evaluation protocol: assessment of member engagement

Here we describe an online application we developed to assess group member engagement: this is part of the group self-evaluation protocol. The goal is for group members to experience a high degree of autonomy. We assume that this contributes to constructive group dynamics.  

Mid-project: progress checklist

mid-project events: group self-evaluation and project-progress checklist

At mid-project (or mid-term) a “structured” progress-survey (Eddy et al., 2013) is offered during one of the project tutorials. It comprises a list of 9 questions dealing with the title and the problematic of the project, whether the group has reached the level of task-delegation and whether they have found external experts for their project. They were also asked to provide examples of pertinent documents and websites that they had consulted for their research. These questions, of course, vary according to the nature of the project. Lastly, the groups were asked whether they felt fairly treated by the project-associated instructors. The number of positive answers was scored and the survey was rated as x/9 (see below, figure 2, dashboard)

Mid-project: first online group self-evaluation

In the self-evaluation procedure, group members rate their own compliance with the ground rules and that of other group members. Group members could tick three boxes for each ground rule: below expectation, meets expectation (satisfactory) or above expectation (exceptional) (figure 1 below). Instructors can set coefficients for these qualifications and we had chosen 0.8, 1.0 and 1.2 respectively. The individual engagement is calculated by the number of points divided by the number of ground rules. Values thus are in between 0.8 and 1.2. We thought it wise to restrict the scale to abovementioned limits because the aim was neither to punish, nor to glorify certain group members (see also Cheng & Warren, 2000; Kilic & Cakan, 2006). The auto-evaluation procedure was meant to give students a sense of agency with respect to the functioning of the group and to give each other insight in a most objective way.

Figure 1.  Example of the screen of the group self-evaluation application with seven ground rules. Group
Figure 1.  Example of the screen of the group self-evaluation application with seven ground rules. Group members can rate compliance with the rules as below-, meets- or above expectation (exceptional).

The application also calculates inter-rater differences by comparing self- with other member evaluations. We refer to this as rating coherence. The difference is 0 (same appreciation), 1 (difference of only one level of the scale) or 2 points (difference between below expectation and above expectation for the same rule). The application counts the differences for each ground rule and divides them by the number of ground rules. Individual coherence values are presented as 2 minus the delta value. The same applies for the calculation of the “group average coherence” (the sum of individual differences divided by the number of members).

Rather than correcting for inter-rater disagreement, to prevent that self-boosting students gain higher grades, we inform group members about the coherence of their evaluation and, in case of disagreement, whether this was due to over- or under appreciation of their own performance (Leik & Wyvill, 2001; Neus, 2011). We reasoned that realistic self-evaluation is a good measure of the extent to which a group member is mindful of his or her functioning in the group. Moreover, groups can only repair dysfunction if all members agree on their functioning so it is really important that members are aware of each other’s appreciations. Low rater coherence may require intervention from leaders or instructors.

Feedback after mid-project evaluation

The online application creates a report, (“dashboard-1 and -2” in figure 2 below) of the self-evaluation procedure that informs staff and students about compliance with the ground rules (“individual engagement”) and about the coherence of the respective evaluations (“individual coherence”). The report also shows an “individual engagement coefficient” which is the equivalent of an individual weighting factor (IWF). It is simply measured as the “individual engagement” divided by “average group engagement”.

Figure 2. Example of a dashboard that group members receive. On the left, dashboard-1, a group with high engagement (individual and group engagement) and coherence (individual and group coherence). On the right, dashboard-2, an overall productive group but with one important disaccord. Note the low score for individual engagement, meaning low compliance with ground rules, and the low inter-rater coherence (low individual coherence). The voting results revealed that Pierre had rated himself above expectation, whereas the other three group members found this contributions totally unsatisfactory.
Figure 2. Example of a dashboard that group members receive. On the left, dashboard-1, a group with high engagement (individual and group engagement) and coherence (individual and group coherence). On the right, dashboard-2, an overall productive group but with one important disaccord. Note the low score for individual engagement, meaning low compliance with ground rules, and the low inter-rater coherence (low individual coherence). The voting results revealed that Pierre had rated himself above expectation, whereas the other three group members found this contributions totally unsatisfactory.

The mid-term evaluation is informative. Its purpose is to familiarize group members with the application (how to access the site and how to vote), to make them aware of how their votes were translated into scores of group functioning, and to inform them about how they functioned in the eyes of their groupmates. The instructors inform the groups about the significance of the results and are open to any questions and suggestions. Besides giving students and instructors insight into group functioning, there is evidence that a mid-term evaluation with individual feedback has a positive effect on cooperation (Archer-Kath et al., 1994).

What could you do if the group indicates that it cannot correct the behavior of one of its members (a dissident)? Read more in “dealing with misaligned behaviour

End of project: second online group self-evaluation

end of project event: second group self-evaluation.

The self-evaluation is repeated at the end of the project, in a closing meeting where the product is handed in (presentation, report, blog, etc). The group members again receive a dashboard as described above. When grades need to be given (as is the custom in schools and universities) the individual effort coefficient is used to generate an individual grade (coefficient x group grade = individual grade). In case of other circumstances, the dashboard values can be reported to instructors or leaders or the group member’s appreciation can be included in a portfolio.

We foresee that when the self-evaluation procedure is applied in a number of collaborative projects, with different groups and for a number of years, personality traits could be distinguished. For instance, “group leaders”, or “extra milers” (Li et al., 2015), would be characterized as those who regularly, and with a high degree of coherence, received high marks for their compliance with the ground rules. The self-evaluations may also reveal problematic individuals, where compliance is weak and linked to incoherent inter-rater appreciation.

We note that much of the problem of over- or under-rating in our classes was resolved after the formative self-evaluation at mid project. Group members were informed about the incoherence, which member was concerned, and in almost all cases this was corrected in the second self-evaluation at the end of term. This correction may in part be the consequence that students wish not to penalize themselves or others in a summative setting (Sridharan et al., 2019). Alternatively, students may have taken on a different attitude (Archer-Kath et al., 1994; Gabelica et al., 2012). We also note that although we encourage students to vote confidentially and the procedure provides opportunity for confidential voting, some groups take a collective approach and agree to rate all members equally. We do not see this as a problem as long as there are no signs of bullying.

References

Archer-Kath, J., Johnson, D.W., & Johnson, R.T. (1994). Individual versus group feedback in cooperative groups. The Journal of Social Psychology 134(5), 681-694. https://doi.org/10.1080/00224545.1994.9922999

Cheng, W., & Warren, M. (2000). Making a difference: using peers to assess individual students’ contributions to a group project. Teaching in Higher Education 5(2), 243-255. https://doi.org/10.1080/135625100114885

Eddy, E. R., Tannenbaum, S. I., & Mathieu, J. E. (2013). Helping teams to help themselves: Comparing two team-led debriefing methods. Personnel Psychology, 66, 975–1008). https://doi.org/10.1111/peps.12041

Gabelica, C., Van den Bossche P., Segers, M., & Gijselaers, W. (2012). Feedback, a Powerful Lever in Teams: A Review. Educational Research Review, 7 (2), 123–144. https://doi.org/10.1016/j.edurev.2011.11.003

Kilic, G.B., Cakan, M. (2006). The analysis of the impact of individual weighting factor on individual scores. Assessment & Evaluation in Higher Education, 31(5), 639-654. https://doi.org/10.1080/02602930600760843

Lejk, M., & M. Wyvill. (2001b). The effect of the Inclusion of Self-assessment with Peer Assessment of Contributions to a Group Project: A Quantitative Study of Secret and Agreed Assessments. Assessment & Evaluation in Higher Education, 26(6), 551–561. https://doi.org/10.1080/02602930120093887

Li, N., Zhao, H., Walter, S.L., Zhang, X., & Yu, J. (2015). Achieving more with less: extra milers’ behavioral influences in teams. Journal of Applied Psychology 100(4), 1025-1039. http://hdl.handle.net/10722/222711

Neus, J. L. (2011). Peer Assessment Accounting for Student Agreement. Assessment & Evaluation in Higher Education, 36(3); 301–314. https://doi.org/10.1080/02602930903342315

Sridharan, B., Tai, J., & Boud, D. (2019). Does the use of summative peer assessment in collaborative group work inhibit good judgement? Higher Education, 77(5), 853-870. https://doi.org/10.1007/s10734-018-0305-7