Mechanics and Workflow of Grading Group Work, Notes from the Collaborative Classroom pt 3

Mechanics and Workflow of Grading Group Work

With any group work, and in particular with how I’ve set up the assignments and the amount, there are a lot of moving parts for any one project.

Grading and Infrastructure

There are two things to consider when setting up a system: the classroom management system (assignment uploads, etc.) and individual grading apparatus. To see some of the types of topics that would be graded, here’s a bird’s-eye view of the general grade breakdown from the course syllabus:

Grade Weight Overview

These days, many schools will have something like Moodle or Blackboard in place for communicating with students, attendance tracking, turning in assignments, grade books, quizzes, polls, etc. My university uses Moodle (I believe at the time, it was version 2.0) to help teachers manage their classrooms. Here is an snapshot of what a typical class of mine looks like on Moodle (apologies for needing to zoom in quite a bit for details):

Sample Class Moodle Page

My approach has been to begin the semester by covering many of the document elements and principles up front, and then applying those concepts to actual projects. I do this to avoid “teaching by mode” because, like freshman composition essays, rarely are documents created using one single type of element or rhetorical tool; that is, successful documents and essays usually make use of many tools and elements. However, the project topics tend to themselves create a more nuanced focus of the material previously discussed. This also helps with “learning” one topic then forgetting about it to move on to the next concept.

As you can see from this snapshot, there are a number of assignments/tasks co-occuring at any one time:

Mutliple Assignments Circus

There is a lot to do in this class and I find that by juggling this many tasks, we cover more assignments and projects than we otherwise might.  It’s also necessary if you want to evaluate all parts of any particular project’s processes. And of course you do! The volume of tasks for students also seems to balance out equitably, (my perception of) the lack of difficulty in the material itself, avoiding the “Easy ‘A’ Class” label. Although this organization seems a little…unorganized…,once you dive into it, it begins to make sense. Really, it’s just sheer textual volume that is masked as a seeming lack of organization (students and I both have to pay attention to the schedule!). Admittedly, I do keep an eye open for other tools that I could integrate with Moodle that might help alleviate that sense of overload.

Electronic versions of projects/assignments and the ability to store them without physical copies has been one of the most useful tools for managing my classes. However, even though most projects and assignments do have a digital birth, I also have students turn in hard-copies. A physical manifestation of their work gives them yet another perspective of their work, which is really most assignment’s larger goal. This works for non-digitally created work as well; that is, if it the project was born non-digitally, then students turn in digital photos or scans, again giving them another opportunity to see their work from a different perspective.

I was using electronic gradebooks in the form of spreadsheets even before my university set up that feature for our Moodle implementation. And though Moodle’s grading system is mostly a clunky system (and limited), I really do like it—especially for students’ ability to track their grades on their own and grade notifications (online and email). While most professors typically use a total point system of grading, I prefer the weighted percentage method, though the point system really is the same thing–if you know how many quizzes or other assignments you are going to use ahead of time for the entire semester. I never know how many quizzes I’m giving because I use them primarily as a check on whether or not students are doing the reading. Also, the number of projects sometimes changes depending on the speed of the class.  For details on how to set this up weighted grading on Moodle, open up this pdf copy of instructions I created for a particular professor’s American literature class: Sample Weighted Grades Setup.

One thing I really like about using Moodle or Blackboard for turning in assignments is the ability to set due dates AND times. I used to set them for 5 minutes after the start of class on the day a project was due, but later changed it to a 5 minutes before–too many groups were busy uploading projects at the last minute from the computer labs, arriving late to class. After the typical rodeo of turning in their first assignments, I’m able to address individual problems so that they become used to the upload process. After that initial run, I maintain a strict no-late-work-accepted policy. That electronic time-keeper removes the burden on me placed by students who are constantly late as well as maintains a sense of fairness to other groups who do get their work in on time.

Crowdsourced Grading

The first time I taught this course, my own comments and grades were the only written feedback about their projects that the students received. I originally thought groups taking notes of their peers’ comments during the workshop would help them be more engaged. But I found this was a missed-opportunity for another chance of engaging with their material and design choices. Also, only a few students consistently volunteered feedback during these workshops.

For my 2013 classes, I decided to try a crowdsourced grading approach advocated by Cathy Davidson in her post, “How to Crowdsource Grading” that would hopefully encourage more students to participate in giving more feedback and again, learning from their own observations. For my own brand of crowdsourced grading, I based the project grades on the average of their peers’ evaluations, folding my own rankings into the mix. While I typically find rubrics a very limiting way to grade, it quickly became obvious that I needed some sort of descriptive ranking form in order to get everyone on the same page in terms of what and how to evaluate their peers’ projects. It also ended up having the benefit of constantly reminding students of the concepts and question about which we were discussing throughout the semester. I created a 0-5 point scale rubric based on Steven Gerson’s evaluation form found within his Writing That Works A Teacher’s Guide to Technical Writing. Although my own evaluation scores were averaged along with their peers’ scores, I did reserve the right to adjust a project’s final grade if the class had for some reason missed the complexities of a particular project (which only happened a few times). For the Spring semester, I had thought to use the evaluation forms themselves as a pedagogical exercise and have the students develop their own form using the topic/ratings from my borrowed form. Instead, it was a logistical nightmare.

Grading and Technology

There are two things to consider when setting up a system: the classroom management (assignment uploads, etc.) and individual grading apparatus. To see some of the types of topics that would be graded, here’s a bird’s-eye view of the general grade breakdown from the course syllabus:

Grade Weight Overview

Notice that there are two sets of evaluations for each project: one for individuals grading group projects, and the other, for grading their group co-members’ contributions to those projects.

The generic workflow for project evaluations went like this:

  • Individual students evaluated each project and presentation.
  • I compiled this information into an Excel spreadsheet.
  • After all the tabulations, I formatted that information into a report within a copy of those worksheets.
  • I then copy the different sections of those worksheets into Word and adjust any formatting, then print that file into a PDF file (for user compatibility), which I then uploaded to each group within the grading of the assignment listed on Moodle. (I used Word because it was easier to arrange the report sections into a format that’s Page-Reading Friendly rather than trying to work with Excel’s print options.)

This meant I would start off with roughly 35 sets of evaluations for each group’s project that ranged from text files html files, in as many different layouts as there were students. For that first semester last year, I had to manually go through each form copying/pasting or retyping in many instances, the different information to my Excel spreadsheets. This also included written comments on what worked well (and why) as well as what worked less well (and why) and how it might be improved. The many different formats I received as well as number of which I could not simply copy and paste the information took entirely too many hours to compile for each project.

That Fall semester, I decided this process needed to be automated. One option was to build my own website using a SQL backend. However, while storyboarding my tables and forms, a friend gently reminded me of how much testing and time it would take (class had already begun). So instead he convinced me to look for an out-of-the-box solution.

SurveyMonkey

I’ve had some experience implementing SurveyMonkey for a former employer, so I decided to try it out for our projects. It may seem odd at first, but if you think about it, a rubric or evaluation form is really just a survey form. The free version only allow for 10 questions, far fewer than the number of topics I needed, so I upgraded to a paid one. While it wasn’t exactly cheap, it didn’t break my bank account either ($25/month per semester seemed within reason). Unfortunately, the screenshots I made last year disappeared (probably while cleaning up my folder structure—a learning moment for making actual use of my backup system!), so please be patient with an evaluation using screenshots from an outdated survey as well as from my imperfect memory.

The webform for creating the survey had some flexibility but trying to create a ratings system using option buttons lacked the precision I would like it to have and it also took a while build out. Although the individual evaluation areas (using a scale rating) were tedious to build, I was able to easily adapt the topics from that pdf form. Editing the different sections was simple. Adding, deleting, and moving areas worked as you would expect. The form was also browser based, so I didn’t have to worry about computer compatibility. I could also download all the responses to an Excel (or csv) file. The problem was, that a couple of students swore up and down that they submitted responses but nothing showed up on the backend. This was important to confirm because their responses count as part of their grades. But SurveyMonkey does not provide an email confirmation to say if their information was successfully submitted. Students also complained about not being able to edit their responses: I set it up so that there was one generic form for projects within a class for a particular assignment; part of that form required them to fill in their own name, project title, and class number in addition to evaluating the project. This type of setup required then that the student take the same “survey” 5 or 6 times (depending on the number of groups). Although this makes sense for a one-off actual survey, it really was tedious and error prone.

Here are a few screenshots that should illustrate that although it is a functionally usable product, it really is simplistic in design and usability (but it does get the job done):

  • The main dashboard, My Surveys:

MySurveys

  • Sample Empty Survey form (page 1 and 2):

 Sample Survey Page 1                   Sample Survey Page 2

  • The general editing of a survey:

Editing a Survey

  • Editing a specific question’s options (I broke this scrollable window into multiple clips because my screen capture software (Snagit) would not capture a sub/child-scrollable window):

Editing a Question 1 Editing a Question 2 Editing a Question 3 Editing a Question's Type

 

 

 

 

Adobe FormsCentral (and a lot of Microsoft Excel and Word)

For the next project, I tried Adobe’s FormsCentral. It was much cheaper ($15/month) and I liked this process better (mostly):

  • I could first create a form within Excel and save it as a pdf file.
  • I would then use my own copy of Adobe Acrobat Pro XI to convert the pdf into a form (using form fields in place of the cells I used within Excel).
  • I would then upload this form to FormsCentral where I had further options I could use on this form. For instance, I could insert a submission button within the form itself, so that all the student needed was Adobe Acrobat Reader installed and a web connection to upload the form upon pressing the Submit button. I set it to also email a confirmation back to the student which I customized to also include their own responses to the form (that is, they had a copy of their data as proof of submission in case something went wrong).
  • Distributing the form was also easy. Although I could give students an url, I decided to download and post the newly create form from FormsCentral to the class’ Moodle page.
  • Like SurveyMonkey, I could download the results as an Excel (or csv) file. This was the part for which I really needed automation! I could sort and separate out this information into individual sheets for each group.

The one thing I found frustrating  was that if I needed to change the form, I couldn’t edit it online because I created it locally. If I had created the form online, I could—but the online creator wasn’t as robust as it was to arrange the form using Excel first). This usually meant going back to Excel and begin the creation process again, though in some instances, it only meant adjusting a form field property (for instance, “required”) and then re-saving that form and re-uploading it to FormsCentral. One potential show-stopper for this product is that it is platform dependent. That is, if the student used a Windows PC with Adobe Acrobat Reader installed, the process worked fine. But that requires that students knows if they are using Acrobat on their computer, which you cannot take for granted. In some cases, they were unknowingly using other PDF software which broke all of the form field features of my Adobe document. But that was a simple problem to solve; just download and install Adobe Acrobat Reader. However, this was a different story for students who used an Apple computer. FormsCentral’s online help about that issue was only so helpful… Most students were able to find a work around which I then distributed to the rest of the class, but sometimes even those solutions inexplicably would not work. For those cases, the students would email me their pdf file and I would manually copy the information over to my spreadsheet.

Although I like the features of FormsCentral more than SurveyMonkey, the cross platform compatibility of SurveyMonkey  mitigates a large problem, making the idea of a browser solution more appealing than a pdf-bound one. Again, the programmer in me wants to develop my own survey application. I think building one-off webforms (that is, a particular survey page for a particular project) should be fairly easy, but like anything code related, it requires time and testing to ensure you won’t have problems that would get in the way of class time. In the end, it seems that the out-of-the-box solutions are quicker and much easier (involving only a few workarounds) but are still limited (well, from this one sampling–I did look at other solutions such as email-based forms, but those were also lacking).

Note:  my apologies for the lack of FormsCentral screenshots from the actual site; however, their site does provide good documentation and the interface is fairly intuitive.

Sample of the Workflow

Here is an example of a Project evaluation form students would submit: SAMPLE PROJECT Evaluation Form

Here’s an example of what the raw data (well, with formatted titles and column sizes) in a worksheet looked like once I imported the results from FormsCentral:

Partial Capture of Compiled Data

 

Here is a screenshot of the worksheets tabs within an Excel workbook I used to keep track of and create reports for a class’ particular project:

Sample Worksheet listing within a Workbook

I used the corresponding worksheets to

  1. compile the FormsCentral submitted data,
  2. create a template I would copy for each group to later paste a particular group’s results from the compiled worksheet (normally, it would contain the students’ names, but here are removed),
  3. use just as a quick reference after all grades were turned in.
  4. compile final individual grades/responses for all the groups (worksheet name based on Group name),
  5. clean up the previous worksheet (worksheet named with “(2)”) by first sorting the responses differently for each project in order to hide the identity of comment creators, afterwards removing those names all together)
  6. to create a pdf report as feedback to the students. This report was created by copying the different sections of the “(2)” version worksheet into a Word document, and then formatting it into something readable. It would then be saved as a PDF file and uploaded to Moodle:

In order to help make a little more sense of what I’m doing with all of these worksheets, I’ve included an actual workbook (removing student names, of course):

Sample Workbook used for Project Evaluations and Reports

It should be noted that in some of the worksheets, you will occasionally see two types of highlighted rows. They were for accounting purposes. The yellow ones indicate people who never turned in evaluations for that group’s particular project and so would receive no points for that particular evaluation. The off-green highlighted rows indicate that the individual in question was a member of the group for which the current worksheet was about–they were presenting their own project, so no evaluations from them were needed.

Here are samples of the Reports I created from the worksheets that students would ultimately see. The Word document was created first from the worksheets marked with a “(2)” on their tabs. I added this intervening step because formatting in Word gives me much more fine precision than I would otherwise have in Excel. The PDF file is just the same Word document; PDF files remove any compatibility issues I might otherwise have (these were what students saw within the grading section of their assignment pages in Moodle ):

Sample Project Report (Word version)

Sample Project Report (PDF, student version)

Some Thoughts about my Project Grading Efforts

No matter the technology used, the topics themselves as well as ranking scales are ultimately what’s important in terms of student evaluations. I was embarrassed by my earlier attempts and so engaged the students themselves to evaluate their evaluation forms. Again, pedagogically, I thought it would be a useful exercise. In practice, most student feedback was not very considered. However, they did point out a few inconsistencies.

In the end, the 0ne thing I was trying to avoid was creating specific evaluation forms for specific projects–that is, I wanted to use one form for all projects. I really wanted the lower maintenance of a generic form, but after scouring the web for examples of such forms, what I did find was just as problematic as my own. Not only are grade descriptions too generic there just weren’t that many samples available, which is why I made it a point to include my own in this posting–not that I think they are perfect (far from it) but they might help serve someone else as a starting point.

I’m also wondering if it might be more useful in future courses to decrease the point value range (and therefore the grade descriptions) from 6 to 3. I do like the idea of having more nuanced evaluations but a simplified system might lead more to bottom and top heavy grades as opposed to a middle range. One of the things I’ve noticed when it came to students evaluating each others work, was that some students always gave high, positive feedback. There were also some students who consistently gave low, harsh feedback–either out of laziness or vindictiveness (I’ve heard a remark on more than one occasion that went something like, “So and so’s a know-it-all and I can’t stand her!”)

Grading the Grader

Typically, most students began workshop comments as well as written comments at a very generic level and with little support.

At first, I would give specific in-text feedback to their form (see the first posting of this series):

in-evaluation commentswhich would be followed by some general reminder about detail:

generic comments

On that first round, I didn’t count off; they received full points. After all, they were learning about details and I was trying to be positive.

But being this detailed for 35 sets of evaluations for each group (4-6) proved to be overwhelming. In the end, I opted for a simplified approach, hoping to illustrate by using my own comments on their work as an example of the details they needed to be forming within their evaluations. I created a Word document with my comments for each person, which I would then copy/paste into Moodle‘s grading form. I would then use this file as the template for the next project, making my notes underneath the previous comments. This way, I could easily tell if the students were making use of my feedback or just plain ignoring it, thus, helping me write the new set of comments:

combined comments

 

On the next project, I would remind them of comments and suggestions I made on their first set of evaluations and take off a few points. I would do this until later, they would receive very little points.

comments improvedBut even with my streamed down style, providing this kind of feedback is important. I would even say it was one of more important learning moments for students. If you read through the sample Report pdf above, you’ll get a better idea for the range of comments. I freely admit that this is an area I need to spend more time experimenting with but even so, I would never cut this part out of the class.

Grading their Peers

Originally, I had students writing a one to two page essay evaluating both their peers’ as well as their own contributions to their projects. However, given that students had a lot to work on for class at any given moment, and to help me cut down on the amount of grading, I decided to streamline this into a form, though free-form text one. I looked around quite a while until I stumbled upon this Collaborative Project Evaluation Form on Purdue’s website. I altered it only by removing the form fields. Like project evaluations, efforts were mixed and required coaching throughout the semester. Though this student tended to give full marks every time, it is still an example of a fairly thorough evaluation: peer evaluation sample

 

As this post is already well-beyond a comfortable reading length, I’ll just end with a graphic illustrating my folder organization as far as projects are concerned. Of course, even for non-collaborative classes, file management ought to be a concern; however, I hope this image demonstrates that with many moving parts task-wise, in terms of actual file management, you’ll need more than the one folder per class method I so often see other teachers using:

Folder Organization

Next: Pt 4, A Sample Project

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *