The use of Rubrics in Assessment

One of the ways we have found in improving student assessment choice, increasing transparency in the assessment process and encouraging students to engage with their feedback was to use rubrics in the assessment process. I know rubrics are contested by some, and are not right for every discipline and in every context, but we had some really positive results by using them. It isn’t easy to create a rubric, especially a scored one where it works out the grade, but most academic members of staff have a marking scheme of some sort, or even just tacit knowledge to know what a piece of work is ‘worth’. It takes a while to get it right and working in the way it should. I worked with another member of staff at the University on a project which tried rubrics out within assessment. The steps were:

1. The student completed the assignment, they could hand in their assignment in any format they wanted as long as it could be submitted directly to the University VLE or linked to. So they could write an essay, do a PowerPoint presentation, do a short video or a podcast, or use whatever format they chose as long as it met the learning criteria and could be accessed.
2. Whilst the member of staff was marking the assignment using the rubric, we asked the students to self-assess themselves against the rubric, and submit the rubric to us.
3. When the work was handed back we asked the students to compare the marked rubric against their own.
4. If the students felt that they had been unfairly marked on any of the criteria, they could appeal. They had to write no more than 500 words referring to the rubric and their original piece of work submitted.
5. We compared the results of the self-assessment with that of the tutors.

 

We found the following:

1. That using the rubric created complete transparency of where the marks came from. The students really liked this instead of not being sure of how a particular mark was arrived at.
2. Using the rubric allowed students complete choice on choice of format, which played to their strengths rather than just favouring those students who were good at writing or exams.
3. Breaking the mark down into the different criteria was also useful, as it gave the students some really clear information on how they could improve their piece of work or future assignments.
4. Giving the student the chance to appeal gave them agency over the process. Only two students took the tutor up on this option, one convinced the tutor they should have been graded slightly higher, the other admitted they were trying it on and accepted the given grade.
5. The self-assessment task gave the tutor some really interesting data:
   1. Firstly it gave the tutor some really rich diagnostic data as it broke down the information into individual criteria, the tutor could see if any of the criteria had been misunderstood. In this case the students rated themselves much higher than the tutor on one of the criteria, which was to do with secondary resources. They had obviously misunderstood what this involved, so intervention could take place, and some training/explanation of that could immediately be put in place to correct that.
   2. Most students saw themselves as in the 2.1 category (60-70%). Not sure if this was wishful thinking by some but those that were 2.2 or 3^rd students rated themselves higher, and maybe through modesty the 1^st students (over 70%) also rated themselves as 2.1s.
   3. The students who rated themselves as 2.1s and were graded by the tutor as 2.1s were not scoring consistent 2.1s across all criteria. This was interesting as I think that those students who expected a 2.1 and then subsequently got it, would probably not have engaged with their feedback much if it had not been for the rubric. The rubric showed them that they were perhaps scoring a 1^st in some criteria and lower, even down to a 3^rd in other criteria but gaining an overall average of a 2.1. Having that in front of them in the rubric showed them clearly what they needed to do to improve their work and gave them the aspirations of getting a first (perhaps they had never thought this possible).

 

This activity was improved in subsequent years to have the students engage with their criteria before submitting their work, and involving peer assessment before having a chance to improve their own piece of work. The tutor was also able to compare years and cohorts to see how the students have improved. It was very interesting to be involved with this, and the students really liked the rubrics being used as it was crystal clear in how the marks had been arrived at and exactly what they needed to do to improve their grade.

 

Some further references about using rubrics (including our conference presentations about the above project)

Campbell, A. (2005). Application of ICT and rubrics to the assessment process where professional judgement is involved: the features of an e-marking tool. Assessment & Evaluation in Higher Education, 30(5), 529-537.

Ellis, C., & Folley, S. (2009). Improving student assessment choice using Blackboard’s e-assessment tools. Paper presented at BbWorld Europe 2009, April 6–8, in Barcelona, Spain.

Ellis, C., & S. Folley, S (2009). The use of scoring rubrics to assist in the management of increased student assessment.

Hafner, J., & Hafner, P. (2003). Quantitative analysis of the rubric as an assessment tool: an empirical study of student peer-group rating. International Journal of Science Education, 25(12), 1509-1528.

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144.

Meier, S., Rich, B., & Cady, J. (2006). Teachers’ use of rubrics to score non-traditional tasks: factors related to discrepancies in scoring. Assessment in Education: Principles, Policy & Practice, 13(1), 69-95.

Creating Virtual Learning Sessions (webinars)

One of the discussion threads this week on otTEL is about running online synchronous sessions using platforms like Blackboard Collaborate or Adobe Connect. I have used both these tools for delivering and attending webinars, but I am always looking to improve my skills in this area. I attended a free webinar run by Citrix in March this year on how to create virtual learning sessions, which I found really useful, so thought I would share the notes I made from the session here:

1. How to keep participants engaged and active:

• Call a session by its rightful name; terming a session meeting, presentation, webinar, learning event all create different expectations in terms of participation etc – so make sure you label your session correctly.
• Every 3-5 minutes have the audience do something different to keep attention and prevent multitasking. You have plenty of tools at your disposal including: asking questions; using the whiteboard; using chat; polling; giving a break; thinking time; asking for people to speak; read something; show a video; etc.
• For a small group, a tip to make sure everyone is engaging is to write everyone’s name down on a piece of paper and mark down when they contribute, that way you can invite those who are quieter for their opinion.

2. 3-step instructional design technique:

• Identify goal performance and objectives:  the instructional goal is the session’s mission statement; the performance objectives are what the learners will be able to do when they leave the session.
• Determine the assessment needs: how do we know each of the objectives have been achieved? Take each objective separately for this to determine if it suitable for online instruction. A general rule of thumb is ‘if it can be tested online, it can be taught online’.
• Determine collaboration needs: think about if you need to bring people together to learning this thing. What does collaboration bring? Will it make the experience richer? A general rule is that if it does not need collaboration, then it can be taught via self-paced online materials rather than in a collaborative session e.g. learning road signs.

3. Determine when and how to design interaction and collaboration online

• Differentiate between interaction and collaboration. Interaction involves participation including polling and adding ideas to the chat space; collaboration involves working as a group to come up with a solution to a shared problem.
• Two main reasons to include interaction and collaboration: firstly to support participant engagement and secondly to support learning outcomes.
• Interactions are usually at the lower end of Bloom’s taxonomy and promotes communication between the tutor and participants and between the participants.
• Collaborations usually target the higher levels of Bloom’s taxonomy like evaluation, analysis and synthesis  and achieves deeper learning.
• The goal with collaboration is to help participants to achieve better results than they would individually.
• Interactions can be serial or concurrent. Serial means that people take it in turns, concurrent means that everyone can participate at the same time. A unique feature of online platforms means that concurrent interactions can take place. If serial interactions take place online, people will quickly start to lose interest and multi-task.

Summary:

1. Set realistic expectations for your students.
2. Create opportunities for learners to engage.
3. Follow the 3-step process to make sure the content is suitable for an online webinar session.
4. Ensure that the learners are interacting and collaborating with you, the other participants and the technology.

Digital Methodologies in Educational Research Conference

This week unfortunately I have not had a great deal of time for ocTEL but I did attend the webinar and played with some of the OER resource websites. One of the reasons I have been busy is that I attended and presented at the Digital Methodologies in Educational Research Conference which was held in Preston on 10th May. As the theme of the conference very much fits in with the ocTEL themes, I thought I would write a blog post about some of the presentations, while it is all still fresh in my mind.

The conference was held in the Brockholes Nature Reserve near Preston, which was a lovely location for a conference. The conference centre was located in a building surrounded by water, with a lot of bird life around, it is just a shame that the weather was dreadful.

First up was Paul Seedhouse from Newcastle University talking about the French Digital Kitchen and European Digital Kitchen. These projects were set up to teach language skills with the additional advantage of the students acquiring cooking skills. The idea was to teach language skill in a more authentic setting than a classroom. The kitchen was fully equipped with technology to help the students who had to follow recipes in the language they were learning. They were paired up to support peer learning, and could use the technology provided to translate things they did not understand. The utensils they used were all fitted with technology that detected movement in a similar way to wii controllers, so it can be checked that the students were carrying out the correction movements. It is not a technology intervention that others could repeat elsewhere due to the heavy investment needed but was interesting none the less.

Second up was Jeff Bezemer taking about using multimodal framework in educational research. This is something I hadn’t come across before but looks really useful. Its focuses on capturing at all the different modes in the research context, such as audio, speech, images, dress, gesture, gaze, etc and very importantly how these inter-relate. The situation being researched is usually videoed so that all the modes can be captured and later analysed in minute detail. He had applied the framework to research in primary school teaching and operating theatre contexts. He used a software package called Elan to analyse the data, which is time-line based so all the different modes can be examined in relation to each other and when they occurred. The rationale behind using this framework is to make visible/explicit the unspoken, so to explore and identify the tacit/embodied knowledge in the given context.

Next up was Stephen Bax from Bedford University speaking about a project which used eye-tracking software to track how language students read some text to answer questions in an exam. This was fascinating. The students had to locate the answers to questions in the text which was set in their second language. The software could show the order of the elements on the screen that the students focussed on, and the length of time their eyes were fixed on certain places on the screen. The object of the exercise was to compare how successful students read compared to those less successful. The results showed that successful students did have a reading strategy but that the strategies used were very varied, so they all read the text in different ways e.g. some would read the whole text first then look at the questions, other would read the questions through first and then look for answers in the text. The less successful students seemed to have less of a strategy and looked around the screen a great deal, moving forwards and backwards. It was really interesting, and you could see the application to other research projects.

Next were myself and Liz Bennett from the University of Huddersfield and our presentation was about the use of digital tools to support the doctoral process. We particularly focused on the use of social media and Web 2.0 tools and their impact on identity. We argued that use of these tools can be both enhancing and exposing, and these feelings are amplified because of the widespread audiences they reach. We theorised our experiences in terms of the notions of liminality and hybridised identity.

I had to leave the conference after our paper to travel back, but I was sorry to miss Cedric Sarre speak about social network analysis and also the closing keynote which was Stephen Downs via video conference from Canada.

So overall a good conference with some really interesting presentations

Notpron – Far too frustrating

I chose the Notpron from the list for activity 3.2 because the word puzzle was next to the name, and I normally like puzzle type games. I say normally. I hated this one, it was extremely frustrating. If left on my own, I would not have got further than the first level, which was easy as you just clicked on the door. This ‘puzzle’ was far too hard for me. You needed a degree in computing/programming and a very creative imagination to work through this, I have neither of these. Hints were provided, but these were insufficient to help me. When thinking of this in terms of learning, it reminded me of Vygotsky’s (1978) Zone of Proximate Development (ZPD). This ZPD stretches students to learn new things, but within their capabilities with guidance and providing suitable scaffolding for this to happen. This was way out of my ZPD, and the hints provided were not sufficient scaffolding for me to guess what to even try to do to get to the next level. I suppose this is a lesson learned in making sure that that enough scaffolding is provided for all level of student, and it provided me with the insight of what it felt like to feel way out of my comfort zone. In my case, I could just give up on the game (or cheat – see below) but what if it was someone’s degree course and they felt like that?

I didn’t give up straight away, as I don’t like to be beaten. I asked my son, who enjoys that type of thing, and together (mainly him) we worked out a few more levels. I also Googled the game and found a walkthrough, which provided the step by step answers up to level 10. Then we gave up – as it was way too hard and the hints and walkthough stopped at that point (no scaffolding at all). I didn’t feel that I built on my knowledge from one level sufficiently to get to the next one, it was all really hard and far-fetched. May be others with more creative minds found it easier but I just found it frustrating.

So lessons learnt were to build learning activities that build up knowledge (slowly if necessary), provide enough support and guidance to assist students to work within their ZPD, and that if students feel that the tasks are way out of their comfort zone, they are likely to give up.

Feeling Overwhelmed or Not Keeping Up?

A few people I have spoken to about the ocTEL course are feeling a bit overwhelmed and behind already, (me included). I think there are a few things worth remembering:

1. Everyone will engage differently. Don’t compare yourself with some of the more prolific bloggers or contributors. Some people will just read the materials and listen in to some of the webinars, and that is totally fine, they will still learn a great deal. Others will not blog at all but may part in the discussion in the forums or on Twitter, – that is ok too. Some people will do a lot one week and nothing the next: still absolutely fine. There are no strict rules with MOOCs, no right or wrong ways to engage and no one will chase you up.
2. No one can engage with everything. Not even the course leaders and tutors. It is impossible. Do what you can, when you can. Some engagement is great. Don’t feel guilty about not doing everything.
3. If you miss a week for any reason or don’t do much one week, don’t worry, just move on. If you wait until you catch-up to move on, you never will. You can always revisit the earlier materials later or after the course has finished. If you move on when the next batch of resources come out you will keep up with the discussions etc.
4. Don’t give up even if you have done nothing or very little yet, it is not too late to join in. Perhaps do the ‘if you only do one thing this week’ in the sections you haven’t engaged with yet then move on.

Online Learning Readiness

I completed all the questionnaires, just because I am a bit of a geek and like taking questionnaires :). Luckily they confirmed I was ready to take an online course, (phew seen as I am signed up for this course!). I did think they were useful tools in helping potential online students consider what they were about to take on, but also raised a couple of issues:

Firstly that it is important to manage expectation of students and this is not limited to online courses. We are focusing this on online courses as the course is about TEL, and because online courses are probably new to some people, so we have to manage their expectations. However each course, no matter what the delivery format should have its expectations clear in terms of how long it will take; the delivery format; what the tutor responsibilities are: what the student responsibilities are; how much self-directed study the students will be expected to do; whether group work and collaborative work are involved, and so on. This will make it much easier for a potential student to decide if a course is for them or not and will prevent wasting everyone’s time of a student enrolling on something that is clearly unsuitable.

I think the questionnaires did not make clear the importance of some of the issues. For example if you lack access to a pc with internet connection, that pretty much rules you out of doing an online course. However if you answered no to being able to attach a file to an email – that could be shown to you very quickly and should not act as a deterrent to participating. Some of the skills the students could develop as they participate. I am sure that not everyone will have had have the skills to participate in a MOOC at the start of this course, but it does not mean they won’t be successful in developing them during participation.

Finally on procrastination and time management/self organisation. Who isn’t guilty of procrastination from time to time, especially when you are using social networking tools to help support you take a course? In addition who could claim to have perfect time management and self-organisation? Often taking part in a course like this does drop down the priority list, especially when it is voluntary and not an official part of our day-jobs. Again these skills can be developed in time. Motivation is an important aspect to consider, as if you are highly motivated, you will find time to participate and develop the skills you need to succeed.

The Teaching Machine, Socratic Method and Social Constructivism

The Teaching Machine was learning by reinforcement by of right/wrong. It used small steps to build up knowledge about something. It had the advantages of self-paced, individualised learning but could not test higher levels of thinking and did not involve collaboration, discussion or other social aspects. It had the advantage of learning independently and privately, so not exposing to weaker students. The disadvantage is that it assumes there is one right answer to questions, so is limited to its applicability to certain disciplines. The learning also must have been very linear, so does not cater for different approaches to learning.

The Socratic method was about drawing out knowledge rather than cramming it in, which differed from the teaching machine, which was definitely about cramming information into the students. The Socratic method is about questioning and thinking not just reproducing answers, which presumably how the teaching machine worked, as the technology could not have been very sophisticated at that time. The Socratic method is about learning socially through dialogue where as no discussion or social activity took place using the teaching machine. The Socratic method may not suit all students though as those less confident would not feel good about their ideas being pulled apart or questioned. They would feel very exposed. It would also not work well with topics that were totally new to students, as they would have no experience on which to draw or build on.

Social constructivism put the emphasis on the social aspects of learning, so learning with others via dialogue and collaboration, and also on creating things to assist the learning process. The teaching machine does not do either of these things as the learning was very individual, and students work in the same room but no collaboration appears to take place. In addition the students are not using what they have learnt to apply to other contexts or learning higher order skills of critique and evaluation.

In summary the teaching machine probably offered an improvement to education at the time, so rather than learning all together by reciting times-tables etc, the teaching machine offered the students a more individualised learning experience, allowed them to work at their own pace, and I am sure was a novelty at the time, so learning perhaps was more fun than students were used to. The reward of getting something right and moving on is similarly to the badge system which is popular at the moment. Some students who were perhaps slower to learn and not confident would have preferred the teaching machine to the Socratic method, even though questioning techniques can be very effective in some contexts to achieve higher order skills. The teaching machine however does not fit very well under the social constructivist way of learning, as there was not social activity and the students were not really constructing anything new based on the knowledge that they have learnt

Week 1 – Do Mitra’s and Mazur’s methods transfer to other contexts?

Week 1 – For the first task this week, I have chosen Sugata Mitra’s keynote, in fact I think I may have been at that presentation, I have heard him speak and he is a very engaging and inspiring presenter. I think what he did in his work with children worked extremely well, and demonstrated a natural curiosity and willingness to learn in children. I wonder though if this is transferable to HE and even more pertinent to my role as Academic Developer, with professional adults? I run short training courses, usually about TEL. I don’t have regular classes with my participants, just a couple of hours, and I can’t really give them preparatory or follow-up work. So would Mitra’s methods work here? If I asked academic staff to huddle round a pc in groups of 4 to find out about the pedagogic affordances of blogs and/or wikis, would it have the same response? And how would they respond to being asked to do that? Also what has stopped them looking this up before if they are curious about it? Although we really try to approach our courses from a pedagogical perspective, we do find that staff really want the ‘how’ not the ‘why’. Similar with Mazur’s peer instruction, if I used that method with academic staff on a short training course, would that work? My guess is that it wouldn’t go down well. Has anyone tried these methods in short training course contexts, particularly with academic staff? Have they worked? I would be very interested in hearing other’s experience on this.

Week 1 – and already playing catch up!!

Well it is week 1 and already I feel I am playing catch up. I was on holiday last week, so missed out on the introductory stuff, and am having enough trouble catching up with my normal work, so I am struggling to engage with the amount of material and resources for week 1. I imagine many of you feel the same, as it can be very daunting starting a new course, making new friends (not easy with 800+ participants and not meeting face-to-face), and engaging perhaps in new ways and using new communication methods.

I have taken part in a couple of MOOCs before, so have some idea what to expect, however, it is still difficult to try and set time aside and accept that you are not going read/engage with everything, and that is ok. The more you do engage, the more you will get out of it, BUT no one has the time to read all the discussion posts, blogs, resources etc, so you have to accept that fact. The best thing is to find a strategy that you know you can stick to.

1. Decide how you are going to engage – many have set up blogs and some use Twitter, others may just contribute to the discussion forums. Decide what you want to do and set up the accounts if necessary.
2. Decide when you are going to engage – again not easy, as this type of self-directed professional development often falls to become lower priority to other work. See when you can fit it in – it may be evenings, it may be an hour each morning, or if you are lucky and dedicated day/morning/afternoon each week. Put it in your diary/calendar and stick to it. Treat it like a face-to-face course that you have signed up for and keep that time free and use it for that purpose. Go somewhere else if necessary away from distractions.
3. Decide how much time you realistically have to engage – and make sure you adapt your expectations accordingly. Accept the limitations, so if you only have an hour a week, you are not going to have time to read loads of blogs etc, you may only have time to do one thing and tweet or discuss that one item. Don’t be put off by the amount of possible work you could do on this. No one will manage all of it all of the time.
4. Accept you won’t have enough time to engage with even a fraction of the content that is around. That is perfectly fine and normal. Someone doing this full time would still not be able to do that. See what interests you and learn something and discuss with others.
5. If you miss a week or don’t have time to engage much, move on when then next materials come out – don’t try and catch up later as it is very unlikely to happen – unless you have a week booked off to dedicate to it. You are likely to fall even further behind and then give up.
6. There is no right or wrong way to engage with a MOOC and no one will be chasing you up if you don’t engage.
7. Try and find some others you can network with – via Twitter and blog posts. Many will be feeling the same as you, and feeling equally daunted and overwhelmed.
8. Don’t give up and good luck!

Hello!

Hi and welcome to my ocTEL blog! I have set this blog up purely for engagement with the ALT ocTEL course running April-June 2013. I am looking forward to the course, both from the content point of view and for the experience in taking part in a MOOC. I have taken part in one MOOC before – the Digital Cultures one run by Coursera, which was a really interesting course, so I am now looking forward to this one. I work as an Academic Developer supporting staff with learning technologies and so this course is very relevant to my role. I have recently completed an EdD which explored tutors’ early experiences of teaching online, so have a particular interest in online education.

I am also providing some support for other staff from the University of Huddersfield who are participating in this course, so providing support with both the technical side of setting up blogs and using Twitter in addition to providing some resources and space for discussions about the course. So if you are from the University of Huddersfield and have not yet let me know you are taking the course, please let me know – thanks!

Other than that, looking forward to the course content and engaging with the wider educational community about technology enhanced learning.