Yesterday, I reported on the results of an experiment into the impact of Powerpoint custom animation on learning. I’ve now been given permission by the authors (Dr Stephen Mahar and colleagues) to publish samples of the screencasts used in the research.
Summary of the experiment
The purpose of the experiment was to test the hypothesis that:
incrementally introducing information on PowerPoint slides via custom animation decreases student learning over having all information shown on the slide at the same time.
Two screencasts were prepared – one animated and one non-animated. The screencasts are designed to teach basic technical information about internet security.
93 students were split into two groups. One group saw the animated version and one group saw the non-animated version. The students were tested on their knowledge before seeing the slides (five weeks beforehand) and just after seeing the slides. The test was composed of multichoice questions.
The samples published below are just under 3 minutes long whereas the originals were 17 minutes 30 seconds.
Non-animated version
Animated version
The results
Jan Schultink asked in the comments of yesterday’s post how the researchers calculated how much more the students had learnt. Here’s more detail on that:
Cognitive load
As I said in yesterday’s post, I think that the animation in this slideshow is generally simple. I don’t think the results of this research can be dismissed on the basis that the animation was flashy and distracting (with the exception of the zooming Dialog box).
However, I think the researchers didn’t adequately take into account the cognitive load (load on working memory) imposed by other features of the screencasts. Specifically:
- Having to read bullet-points at the same time as listening to a narration
- The lack of silence during the animations.
Guidelines for use of custom animation
I’m not ready to throw out custom animation on the basis of this research. But I think that animation has to be used carefully. In the next few weeks, I’ll review other relevant research and write a post with suggested guidelines on custom animation.
Olivia, thanks for posting the samples!
I agree that the animated slide show violated principles established by prior research into cognitive load. For example, here are some research-based principles from Efficiency in Learning (Clark, Nguyen, Sweller) that the animated version violates, and sometimes the non-animated version as well:
–”Give learners control over pacing.” The slides were presented to a class that had no control over them.
–”Present information in as few modes as needed to make it understandable” because “multiple content expressions actually overload working memory.” While we’re processing the audio in the slides, we’re also seeing redundant text, pictures, and animation, and some bullet points are inexplicably in different colors.
–”Audio explanations aided learning only when the tasks were more complex and only for visuals that were not self-explanatory.” The only time audio seems useful to me is when the presentation explains the screen shot.
–”Instructors should remain silent when presenting textual information to learners.”
–”Sequence on-screen text after audio to minimize redundancy.”
In addition, there seemed to be timing issues, with some items appearing at odd moments during the narration.
As a result, the researchers’ findings aren’t surprising.
I think their argument is that the only difference between the slideshows was the animation, and therefore the animation alone is responsible for the decrease in comprehension. However, I think it’s also helpful to look at how media choices could affect motivation, especially when we’re presenting simple facts rather than involving people with stories, discussion, and other techniques.
I dealt with the static show by skimming the bullets and tuning out the narrator. In the animated version, it was harder to do that, so I got squirmy and annoyed.
So the question might not be simply, “How do media choices affect our ability to process info?” I think we should also consider, “How do media choices affect our *willingness* to process info?”
Hi Cathy
Thank you for posting these research-based principles here (For readers who want to know more about this, head over to Cathy’s blog http://blog.cathy-moore.com/2009/06/could-animations-hurt-learning/).
We have two methods of transferring information colliding here: e-learning and live presenting. There’s a lot more research on e-learning and in recent years those interested in having a research basis for their presenting decisions have borrowed from the e-learning research.
But although the e-learning and presenting disciplines can borrow from each other – they are different. So not all of these e-learning principles will apply to live presenting.
Olivia
Note: It’s not clear to me whether the authors of this paper were more interested in e-learning or presenting. I think they used an audio recording to ensure that the audio was exactly the same for both the animated and the non-animated version.
I wonder if part of it has to do with how the students were taught to learn. When I was in university I, much like the comment above, would not listen to the professor until I had read the entire slide.
Perhaps then it was more to the way they are taught as students to learn and so when the information wasn’t right in front of them they were forced to listen to what is a very boring presentation.
Hi Phil
Yes, I notice that the generation that have gone through University during the PowerPoint era have developed coping mechanisms for death by bullet-point! Olivia
Hmmm, I noticed that in the non-animated version the information spends more time on the screen. If you compared 2 presentations both with the information on the screen for the same amount of time and only use animation to highlight which aspect is being talked about, then I wonder what the results would be? Perhaps better for animation.
Hi ITIL expert
The researchers did speculate that one of the reasons the non-animated version was more effective was because of the longer exposure time. I think your suggestion is an excellent one. Olivia
This definitely is a fascinating study that causes me to stop and rethink what I thought I knew.
Does anyone else think that the subtle differences in design should also be taken into account? (Cathy mentioned this in the first comment above.)
In the animated version, on slides 2 and 3, the titles and bulleted text were not all the same color as in the non-animated version. This may seem like a very small detail, but it had me confused for a few minutes trying to figure out why the title was red, one line blue, the other black. When you’re talking about cognitive load, that can be important!
Hi Nathan
Yes, it seems a simple thing, but you spend a couple of seconds wondering about the different colors – and meanwhile you’ve lost what the presenter said. Olivia
I’ll start with my reactions to the two versions of the presentation. There were aspects of both versions that I appreciated.
I preferred the animated version on the first section highlighting the three things to watch out for. I agree with Olivia’s assessment yesterday that the animation where the certificate explodes in is excessive. I liked that it sequentially pointed out things to look for.
I had trouble tuning out the narrator for the second slide with the bullet points. I was frustrated when just the heading came up during the animated version; I wanted MORE information NOW!
I thought the final slide on Access Control was horribly designed for either approach. There was too much text, and I gave up trying to read it. I felt it would have been more effective if the one slide, one concept rule had been applied here.
If the point of the research was to determine whether simply adding animations affected ability to retain information, I say that it’s an intriguing beginning. More experiments using different groups of people and different topics are needed to establish a statistical degree of confidence in the assertions. If this isn’t done properly, this study could morph into a new version of the Mehrabian Myth.
Let me give a big thank you to Cathy! Her comments about the effective learning were very helpful, and I’m adding that book to my to-read list!
Thanks Todd for that analysis. I agree that there’s definitely an issue of too much text, which is a major reason why this research won’t immediately have me changing my approach to animation. Olivia
I also wonder if the listener becomes more passive when the slides are animated. In the static slides, I found myself mentally engaged trying to understand what the speaker was saying and trying to paint a mental picture in my head.
For the animated slides, I found myself disengaging, waiting for the slide to “do something” that made it all make sense. Like watching TV rather than reading a book.
This is an interesting point – would be useful to design research to test this. And I think it’s definitely useful to design and present slides in such a way that the audience does have to work to understand – rather than just be a passive consumer. Olivia
Hi Olivia
Thanks for your consistently excellent posts.
I’m finding it interesting to watch people’s reaction to this research, which actually seems to make a lot of sense to me. Let me explain.
We humans are very good at detecting changes, especially movement, in our visual field. As I understand it the theory is that this skill helped us spot tigers in the jungle way back when. In modern times, it’s why a television that’s on in a bar so easily distracts us from our converation with our date.
When we notice such movement, it seems natural that we’re then inclined to process it, to work out what it is we are looking at. In the digital certificate example in the screencast above, it certainly took me a good few seconds to work out what this thing was that was moving into the screen.
Because we are physically unable to do two tasks simultaneously*, we simply cannot listen to the narration and unpick the animation at the same time. Hence this learning opportunity is compromised.
I would suggest that for the best outcome, any time something new is brought onto the screen, the viewer is given time to see and uncode it, without any other information being directed at their senses (e.g. verbally).
Thanks again,
Jessica
* While I don’t have a reference for this at hand, I am fairly confident that I have seen research from psychology that verifies that at best we can switch very quickly between two tasks, but cannot do them at /exactly/ the same time.
Hi Jessica
I think that your explanation makes a lot of sense. The main thing we ought to do is stay silent during an animation. Olivia
Interesting post and comments. Great work, Olivia! Like at least one other commenter, I felt antsy during the animated version and annoyed at having to wait for what came next. This definitely interfered with listening to the speaker. I think having the whole text on the slide allows students to see the wholeness and make connections better. On the other hand, they don’t listen very much to the speaker. I’d be interested to know if students would do better if there was less talking or, on the other hand, less text. Both at once don’t seem to work very well. Richard E. Mayer’s Redundancy Principle states that “Students learn better from animation and narration than from animation, narration, and on-screen text.” Note that his animation was showing a process like how lightning works — very different from the animation used here which didn’t add to the conceptual meaning at all.
There are some in-between techniques which might also be good to test, such as showing all the text but in a lighter color and “undimming” it and a similar idea of displaying all the text at once in a dark color but changing its color slightly as you focus on a point (I have tips of these techniques on my site.) By now, we’ve come up with enough additional research ideas for the rest of the authors’ careers!
One more point I’d like to make is that how much people learn isn’t always the main goal. In a sales presentation, for example, the goal is how much people buy, and that might or might not vary along with how much people learn. Academic research isn’t always fully applicable in the business world. Something more to test!
Ellen
Thanks Ellen, for your thoughts – and adding to all the value in the comments of this post. Great ideas on other ways the animations could have been done. Olivia
Wow! At last I got a webpage from where I can actually get helpful information regarding
my study and knowledge.