Peer Review and Digital Badges paper published

Our paper on the project has now been published. This gives a good overall summary of the project, the rationale for the underpinning design and the analysis of the evaluation project. The paper is published in Chemistry Education Research and Practice, which is free to read (although you may need to register). The paper should be open access in the next few weeks (i.e. no registration necessary). Enjoy!

All resources developed are freely available to reuse as described in the previous post.

developing-laboratory-skills-by-incorporating-peer-review-and-digital-badges

Experiment Done & Resources Available

This week was D-Day…Week; we had our first years completing their lab skills lab. If you want to try what we did, I’ve linked everything below.

Each student had to:

  • prepare in advance for the lab by:
    • completing a pre-lab survey (this is for the associated research project)
    • watch the pre-lab videos;
  • complete the techniques in the lab while
    • being videoed on their phone for two techniques and
    • completing peer review forms for three techniques;
  • after the lab they had to
    • Upload their video to labs afterwards to a video sharing site;
    • Submit the video links for review.

Overall it seemed to go well. There’s quite a lot to unpack, and having spent every spare moment this week reviewing the titration videos, I haven’t much more energy to say a lot beyond that. But I can say while I was expecting chaos and to be running around the lab coralling students, it just ran like any other lab. Students had something to do. They did it.

Feedback takes two forms.

  1. Students get feedback on their technique via the VLE. This involves watching their videos and giving individual feedback. It is very time consuming, but offers a very rare opportunity to provide specific technique feedback to students. I will be able to build up a good assessment checklist now having gone through lots of them, which is exactly what Marcy Towns said would happen when she described her own implementation.
  2. Students also get feedback on their peer review forms. This gives them feedback on number of decimals recorded, calculations, and significant figures. It was done en masse using mail merge and some Excel formulae under the bonnet, which again means you can give some very individualized feedback, but this time quite easily. (Love mail merge!)

Resources

I’ve put the links to the resources we used below. These are freely available to use, without permission, for anyone who wants to use them. I would ask that if you do use them to let me know how you get on – I like to hear the anecdotes. I would again like to acknowledge the IAD for funding and the project team.

Exemplar Videos.

Peer Review Sheets. I’ve added the Word file at this link so you can customize for your own institution/preferences

Digital Badges. These fabulous badges were designed for the project. If you click on them you should get the original high quality PNG file.

 

Final Badges for issuing

We are counting down now to our final days before our students complete their lab badges skills. Just over a week to go! To move things along, we have gotten the badge back from the designer and they look really great! I’m thinking of making these flyers to try to summarise to students what it is they are all about.

titration-badge-flyer

The final videos we are using this year are at:

 

Getting ready to badge and looking for interested partners

Over the summer we have been working on a lab skills badging project. Lots of detail is on this site, but briefly this is what it’s about:

  • Experimental skills are a crucial component of student laboratory learning, but we rarely assess them, or even check them, formally. For schools, there is a requirement to show that students are doing practical work.
  • By implementing a system whereby students review particular lab techniques in advance of labs, demonstrate them to a peer while being videod, reviews the technique with a peer using a checklist, and uploads the video for assessment, we intend that students will be able to learn and perform the technique to a high standard.
  • The video can form part of students electronic portfolio that they may wish to share in future (See this article for more on that).
  • The process is suitable for digital badging – awarding of an electronic badge acknowledging competency in a particular skill (think scout badges for… tying knots…).

Marcy Towns has a nice paper on this for pipetting and we are going to trial it for this and some other lab techniques.

Looking for interested parties to trial it out

I am looking for school teachers who would like to try this method out. It can be used to document any lab technique or procedure you like. You don’t necessarily need an exemplar video, but a core requirement is that you want to document students laboratory work formally, and acknowledge achievement in this work by a digital badge. We will provide the means to offer the badge, and exemplar videos if you need them, assuming they are within our stock. Interested teachers will be responsible for local implementation and assessment of quality (i.e. making the call on whether a badge is issued).

Yes I need help with badge design
Yes I need help with badge design

This will be part of a larger project and there will be some research on the value and impact of the digital badges, drawing from implementation case studies. This will be discussed with individuals, depending on their own local circumstances.

So if you are interested, let’s badge! You can contact me at: michael.seery@ed.ac.uk to follow up.

Badging Workshop and Titration exemplar

I attended a useful workshop given by Marcy Towns and colleagues of Purdue at the BCCE 2016 conference. They published the paper on badging for pipette techniques. A few things came out of the workshop:

  1. Student videos were less than 2 minutes long. A series of common errors were noted in pipetting.
    • This study didn’t use exemplar videos, but did use demonstrations in the lab, so it will be interesting to see if exemplars reduce common errors in the first instance.
    • This has prompted me to think about assessment – we will need to have prompts to focus students on demonstrating the actual technique and just recording that.
    • Therefore we will need to tell them not to video the washing procedures, and probably point them to what they must demonstrate. Towns used this to develop a rubric subsequently.
  2. Silent videos are scary.
    • In their first iteration, students duly demonstrated what they were meant to, but didn’t narrate. Students are now told to narrate, to ensure that it is clear that they know what they are doing in each step and why they are doing it.
  3. When to do it.
    • The Purdue implementation allowed students to make the videos at any stage of the lab course. In our case, we are dedicating an early (Week 2) lab session to it, as other labs are very busy and we feel that some students may not have the time to complete the videoing. We also want them to get the idea of good technique early.
  4. It worked!
    • By several measures, including costs of broken glassware, tutor and demonstrator observations, and students reporting themselves, the process of demonstration worked. I hope that by adding a peer-review step, we will emphasise the notion of correct technique by adding in an interim review step before videos are uploaded.

We are on the last leg now. I hope that this is the final iteration of the titration video. I decided to cut the pipetting from it, and have it as a parallel video:
Titration Exemplar: https://media.ed.ac.uk/media/Titration+Exemplar+Video/1_wmnz31b5

Enabling lifelong digital artefacts

The protocol for peer-assessment of lab skills involves peers videoing each other as they complete a particular lab activity. Each student will then upload this video to the badging platorm (internally students will do it via Learn, externally students will do it via open source host). As described previsouly, demonstrators (or teachers) can then review the video to make sure it is what it says it is, and then click to issue the badge.

The purpose of the badge is to create a label, or perhaps a shorthand, for stating that the student has competency in a particular lab skill. As this will essentially be issued by the university for internal students (or school/teacher for external) there is some sense for an external viewer (e.g. employer) that this has credit. However, the purpose of the exercise is to allow students create a digital portfolio or their skills, rather than just accreditation, or micro-accreditation.

Therefore the submission of video will not be a submission of the physical file to the platform, as this will be locked in, and not be available for anyone but the marker to view. Instead, students will submit a link to where they have hosted their video.

This means that as well as for the purposes of assessment, students will have the video available to share with anyone externally should they choose. Students can opt to share the link privately (i.e. accessible only to have it visible to those who know the link), or publicly. They can delete the video once the badge has been issued, or maintain it for showcasing in the future. In other words, they will have control over their digital footprint.

How will this be managed? Obviously YouTube allows for uploading and sharing or restricted sharing of videos. For Edinburgh students, the university is about to launch Media Hopper. This will allow students upload (and edit) academic related videos to their own account, which will be available to link both for the purposes of assessment and for their future CVs.

mediahopper

 

What it looks like to issue and receive a badge

The second prototype of the badging platform built on experience described last week; namely that peer-review will need to take place in advance of assignment submission, and that a tutor should look at submissions to make sure they are what they say they are before a badge is issued.

With that in mind, a new prototype involves:

  1. Students watching and clicking to confirm that they have watched the exemplar video.
  2. Students uploading their video to obtain a particular badge.
  3. Tutors reviewing video and awarding a score. It is possible to set a minimum score necessary before badge is issued.

So what does this look like? We will look at the tutor view as well as what the student sees.

As a tutor, I can add the exemplar videos and create an assignment dropbox for each technique. Then, as outlined in the last post, I can use the achievement tool to create triggers that (1) require that students review a video (and click to say they have reviewed); (2) to upload an assignment file; and (3) to have the file reviewed by tutor before badge is issued. In the case of the latter, a minimum mark can be set which the student has to receive before the badge issue is triggered. As a rough outline, the screen below shows the rudiments of this from the student view:

info sheet 1

Trialling the process

I logged in as a student, watched the video (or at least clicked to say I watched it) and uploaded a file. The good news is that a video file upload is possible, although whether this is what is done is going to be the subject of another post. As a student, I have done all that is required of me.

Back in tutor view, I see in the gradebook that there is an assignment that needs marking.

assignment

I can view this and award a grade (100/100 – I am a great student!). This was the final trigger for badge issue, and so the badge is issued for this student. We can see the number of recipients for this achievement increases to ‘1’ in the Achievement control area (number on the right).

achievements with one recipient

The student view, and exploring Mozilla Backpack

Back as a student, I now see that I have one new achievement in the Achievements area.

Badge Issue

There is an option now to publish to Mozilla Backpack, so that I can share my badge with the world. Clicking on the option connects me with Mozilla – this first time I need to sign up. After confirming, my “Badge Issue Trial” badge is now in my Backpack. This tells me the name of the issuer, and details of what was required to get the badge. I can add this badge to my “collections”, and after making it public, share via social media and other channels.

Mozilla