Here we describe an online application we developed to assess group member engagement: this is part of the group self-evaluation protocol. The goal is for group members to experience a high degree of autonomy. We assume that this contributes to constructive group dynamics.
Mid-project: progress checklist
At mid-project (or mid-term) a “structured” progress-survey (Eddy et al., 2013) is offered during one of the project tutorials. It comprises a list of 9 questions dealing with the title and the problematic of the project, whether the group has reached the level of task-delegation and whether they have found external experts for their project. They were also asked to provide examples of pertinent documents and websites that they had consulted for their research. These questions, of course, vary according to the nature of the project. Lastly, the groups were asked whether they felt fairly treated by the project-associated instructors. The number of positive answers was scored and the survey was rated as x/9 (see below, figure 2, dashboard)
Mid-project: first online group self-evaluation
In the self-evaluation procedure, group members rate their own compliance with the ground rules and that of other group members. Group members could tick three boxes for each ground rule: below expectation, meets expectation (satisfactory) or above expectation (exceptional) (figure 1 below). Instructors can set coefficients for these qualifications and we had chosen 0.8, 1.0 and 1.2 respectively. The individual engagement is calculated by the number of points divided by the number of ground rules. Values thus are in between 0.8 and 1.2. We thought it wise to restrict the scale to abovementioned limits because the aim was neither to punish, nor to glorify certain group members (see also Cheng & Warren, 2000; Kilic & Cakan, 2006). The auto-evaluation procedure was meant to give students a sense of agency with respect to the functioning of the group and to give each other insight in a most objective way.
The application also calculates inter-rater differences by comparing self- with other member evaluations. We refer to this as rating coherence. The difference is 0 (same appreciation), 1 (difference of only one level of the scale) or 2 points (difference between below expectation and above expectation for the same rule). The application counts the differences for each ground rule and divides them by the number of ground rules. Individual coherence values are presented as 2 minus the delta value. The same applies for the calculation of the “group average coherence” (the sum of individual differences divided by the number of members).
Rather than correcting for inter-rater disagreement, to prevent that self-boosting students gain higher grades, we inform group members about the coherence of their evaluation and, in case of disagreement, whether this was due to over- or under appreciation of their own performance (Leik & Wyvill, 2001; Neus, 2011). We reasoned that realistic self-evaluation is a good measure of the extent to which a group member is mindful of his or her functioning in the group. Moreover, groups can only repair dysfunction if all members agree on their functioning so it is really important that members are aware of each other’s appreciations. Low rater coherence may require intervention from leaders or instructors.
Feedback after mid-project evaluation
The online application creates a report, (“dashboard-1 and -2” in figure 2 below) of the self-evaluation procedure that informs staff and students about compliance with the ground rules (“individual engagement”) and about the coherence of the respective evaluations (“individual coherence”). The report also shows an “individual engagement coefficient” which is the equivalent of an individual weighting factor (IWF). It is simply measured as the “individual engagement” divided by “average group engagement”.
The mid-term evaluation is informative. Its purpose is to familiarize group members with the application (how to access the site and how to vote), to make them aware of how their votes were translated into scores of group functioning, and to inform them about how they functioned in the eyes of their groupmates. The instructors inform the groups about the significance of the results and are open to any questions and suggestions. Besides giving students and instructors insight into group functioning, there is evidence that a mid-term evaluation with individual feedback has a positive effect on cooperation (Archer-Kath et al., 1994).
What could you do if the group indicates that it cannot correct the behavior of one of its members (a dissident)? Read more in “dealing with misaligned behaviour“
End of project: second online group self-evaluation
The self-evaluation is repeated at the end of the project, in a closing meeting where the product is handed in (presentation, report, blog, etc). The group members again receive a dashboard as described above. When grades need to be given (as is the custom in schools and universities) the individual effort coefficient is used to generate an individual grade (coefficient x group grade = individual grade). In case of other circumstances, the dashboard values can be reported to instructors or leaders or the group member’s appreciation can be included in a portfolio.
We foresee that when the self-evaluation procedure is applied in a number of collaborative projects, with different groups and for a number of years, personality traits could be distinguished. For instance, “group leaders”, or “extra milers” (Li et al., 2015), would be characterized as those who regularly, and with a high degree of coherence, received high marks for their compliance with the ground rules. The self-evaluations may also reveal problematic individuals, where compliance is weak and linked to incoherent inter-rater appreciation.
We note that much of the problem of over- or under-rating in our classes was resolved after the formative self-evaluation at mid project. Group members were informed about the incoherence, which member was concerned, and in almost all cases this was corrected in the second self-evaluation at the end of term. This correction may in part be the consequence that students wish not to penalize themselves or others in a summative setting (Sridharan et al., 2019). Alternatively, students may have taken on a different attitude (Archer-Kath et al., 1994; Gabelica et al., 2012). We also note that although we encourage students to vote confidentially and the procedure provides opportunity for confidential voting, some groups take a collective approach and agree to rate all members equally. We do not see this as a problem as long as there are no signs of bullying.
Archer-Kath, J., Johnson, D.W., & Johnson, R.T. (1994). Individual versus group feedback in cooperative groups. The Journal of Social Psychology 134(5), 681-694. https://doi.org/10.1080/00224545.1994.9922999
Cheng, W., & Warren, M. (2000). Making a difference: using peers to assess individual students’ contributions to a group project. Teaching in Higher Education 5(2), 243-255. https://doi.org/10.1080/135625100114885
Eddy, E. R., Tannenbaum, S. I., & Mathieu, J. E. (2013). Helping teams to help themselves: Comparing two team-led debriefing methods. Personnel Psychology, 66, 975–1008). https://doi.org/10.1111/peps.12041
Gabelica, C., Van den Bossche P., Segers, M., & Gijselaers, W. (2012). Feedback, a Powerful Lever in Teams: A Review. Educational Research Review, 7 (2), 123–144. https://doi.org/10.1016/j.edurev.2011.11.003
Kilic, G.B., Cakan, M. (2006). The analysis of the impact of individual weighting factor on individual scores. Assessment & Evaluation in Higher Education, 31(5), 639-654. https://doi.org/10.1080/02602930600760843
Lejk, M., & M. Wyvill. (2001b). The effect of the Inclusion of Self-assessment with Peer Assessment of Contributions to a Group Project: A Quantitative Study of Secret and Agreed Assessments. Assessment & Evaluation in Higher Education, 26(6), 551–561. https://doi.org/10.1080/02602930120093887
Li, N., Zhao, H., Walter, S.L., Zhang, X., & Yu, J. (2015). Achieving more with less: extra milers’ behavioral influences in teams. Journal of Applied Psychology 100(4), 1025-1039. http://hdl.handle.net/10722/222711
Neus, J. L. (2011). Peer Assessment Accounting for Student Agreement. Assessment & Evaluation in Higher Education, 36(3); 301–314. https://doi.org/10.1080/02602930903342315
Sridharan, B., Tai, J., & Boud, D. (2019). Does the use of summative peer assessment in collaborative group work inhibit good judgement? Higher Education, 77(5), 853-870. https://doi.org/10.1007/s10734-018-0305-7