e-Learning Quality Assessment “Lite”

Want a clean, lite, solid assessment of elearning courses, for yourself or your team? Read on…

Much has been written about quality in e-learning, and numerous e-learning quality assessment schemes have been developed over the years. A fairly recent “Guide to Quality in Online Learning” by Neil Butcher and Merridy Wilson-Strydom is a good review of the question and provides links to many QA systems, rubrics and tools.

But no quality scheme has risen to the top to become the de facto standard for measuring e-learning quality. Buyers and users of e-learning are pretty much left on their own to figure out what’s good and what’s not so good.

In a great little project initiatied by the National Collaborating Centre for Determinants of Health last year, I was asked to do a “scan” of online learning courses available in the health equity area. The project team also wished to assess the quality of found courses, so that the best could be recommended to public health practitioners.

NCCDH nicely agreed to share our simple but quite effective approach. If you’d like to re-use it but aren’t quite sure how, feel free to get in touch!

FYI, our scan and quality assessment was directed at self-paced e-learning courses/modules of any length, as well as facilitated online courses, and to some degree blended courses.

Best practice quality indicators

image of NCCDH best practice reviewThe “best practice quality indicators” we used in our tool were derived and adapted from several sources, including Cathy Moore’s Checklist for strong e-learning, Clayton R. Wright’s detailed Criteria for Evaluating the Quality of Online Courses, and my own work over the years in e-learning quality assessment and evaluation. (Click the image to enlarge it, and click it again to maximize it.)

The 12 best practice dimensions address what we considered to be the most critical aspects of e-learning design and implementation. A 7-point scale was provided to rate each course on each dimension, with each “end” of the scale being expressed in words. When actually reviewing a course using this “form”, the reviewer also entered one to three comments on each dimension, providing evidence from the course review to support the rating.

Course Review “Protocol”

The Best Practices review was part of a simple course review protocol that also included doing a bit of online research on the course being reviewed, filling out some basic Course Summary Information (on content, objectives, audience, course cost/access and so on), and gathering up representative screen grabs from the course. The effort involved was in the order of 2 to 4 hours per course. We created a “micro web site” for the project, with a WordPress form being used to input course information and best practice reviews. This allowed members of our expert advisory group to access course reviews online, and to provide their comments and rankings online as well. This is how we came up with a list of the most highly recommended courses, which NCCDH subsequently posted on their web site.

You can download the simple Course Review Protocol HERE, and a sample course assessment HERE. If you’d like to know more about implementing this approach online, again, just get in touch!


Of course a process like this has issues and limitations:

  • Subjectivity:  First, the choice of assessment criteria is subjective; we addressed that by basing our choice of best practice dimensions on good sources of expertise in the area, but others may have chosen to highlight different dimensions. Secondly, the course assessment itself is subjective, reflecting the evaluator’s biases (in this case, mine). We addressed this by striving to back up the best practice reviews with evidence from each course, and we asked our expert advisory group to “review the reviews”, so we could have several opinions on each course.
  • Usefulness:  Quality assessment of anything must have a purpose. In our case, our goal was to help public health practitioners choose or prepare for learning experiences. We also hoped that our quality reviews might help developers improve the next version of their course. On both of these counts, NCCDH staff report that we have succeeded to some degree.
  • “Doability”:  The assessment has to be doable with “reasonable” effort for each course. We assessed dozens of courses, and with 2-4 hours per course, this was judged reasonable. Other e-learning quality assessment schemes out there claim to be able to “guarantee quality”, but when I looked one of these up, they had only managed to assess about a half-dozen courses in each of the first 4 years of operation. That process may be a bit too onerous…
  • Coverage: There are numerous types of e-learning out there, and more approaches and delivery modes are being invented all the time. Quality assessment criteria must be relevant for different types of online learning, e.g., self-paced modules, blended courses, etc. This is quite challenging. For our part, our criteria were mostly relevant for self-paced e-learning, so our revew of facilitated or blended courses wasn’t as thorough.

E-learning and Wine: Who Cares if They’re Good?

Having worked many years in training and e-learning, and a few years in the wine business as well, I realized recently that both these industries face somewhat related issues with quality. Sadly, way too many consumers of both these products are satisfied with, well let’s say it, crap.

This is not that different from the situation with many types of products and services, but with e-learning and wine, there are some interesting parallels. For instance, with both of these, “quality” has both a technical and a more “creative”, less easily quantifiable component. Technical measures of wine quality include the quality of the processes used to make it, but also quite objective sensory measures, including the absence of known faults (such as too much sulfur, the presence of various bacteria and so on) and the presence of some key characteristics that are generally accepted to make for good or great wine. These are things like a wine’s complexity (the variety of aromas and flavours in a wine), its balance (the absence of overpowering aromas and flavours), and its finish (how long the flavours linger).

Wine cheaper than gasAn interesting thing is that personal preference doesn’t really have that much influence on how someone will assess the quality of a wine. When a group of people have been trained on wine tasting and appreciation, and have a common framework for assessing wine quality (check out the Wine & Spirit Education Trust for this kind of training, you winos), they will still vary widely on the specific flavours and aromas they will detect. These are dependent on how finely tuned one’s nose and palate are, and on personal memories and experiences too. They will also disagree about how much they like the wine, because people vary greatly in their appreciation of a wine’s  sweetness, acidity, body and tannins. But in my experience, even with very different types of wine drinkers in the room, there is surprising consensus on the basic elements of quality, when everyone is asked to independently rate a wine’s balance, complexity, finish and so on.

Now if everyone had to get a little training on wine quality before they bought and drank their first bottle, I dare say the wine industry would be healthier and a fairer business as well for hard-working grapegrowers and passionate winemakers. Millions of bottles of awful wine (at all price points) would stay on the shelves, and wine marketers would turn their attention to producing better wine rather than spending oodles of cash on cool labels and catchy commercials. I know I know, a pipe dream.

On the e-learning side, the situation is similar. Principles of high-quality instructional design have been well known since the middle of the last century, and quite clear evidence-based best practices have emerged for applying these principles to training on a computer-based platform. For a number of years, I helped out with and reported on a well-regarded “Excellence in E-learning” awards program. Like in my wine courses, when our e-learning judges were provided with a clear framework for assessing the quality of submitted e-learning courses, there were all kinds of dfferences of opinion on how much each judge LIKED the program or would want to take it. But yet again, there was only little disagreement about how GOOD each program was, measured by things like: how well learnng strategies matched the skills or knowledge being taught; how appropriate was the use of media to support learning; how expertly did the program use strategies to overcome motivation issues with e-learning; and so on.

So why then, after thousands of years of making wine, and many decades of educational technology, do we still drink bad wine while suffering through hours of tedious page-turning e-learning courses?

If I had the answer, I would have made more money in both these businesses! But perhaps the answer lies partly in our culture of constant change. With so many new devices and platforms to deliver learning on, our attention is drawn to the mechanics of responding to these innovations rather than the noble drive to continuously improve the quality and effectiveness of what we’ve been doing for years and years. Perhaps if we soon hit the “resource wall” that sustainability experts say is coming and we stop being obsessed with growth, perhaps then we will return to a passion for craft and for doing things well rather than just fast and cheap.

On the wine side of things, however, it’s more or less hopeless. By the time wine industry experts get to drafting creative proposals for ridding the world of bad wine, everyone at the table is on their fifth glass of a sublime Bordeaux and the day’s great ideas have long been forgotten.

p.s. I like Jim Shamlin’s take on “Why e-learning sucks”. Check out his blog post on the subject HERE.

Three Kinds of E-learning – Still Drinkable

When you’ve been in the e-learning biz (or any industry for that matter) for a long time, you experience many developments, trends, fads, fashions, and of course numerous changes in jargon and terminology. Years ago (I’ll let you try to pinpoint the date from clues in the piece), I wrote a brief blurb on “The Three Main Types of E-learning” that found its way into a couple Brandon Hall Research reports we were working on (including the one on E-learning in the Non-Profit Sector, which is why the examples are from NGO e-learning initiatives).

The three categories suggested were:

  • Self-paced e-learning
  • Live e-learning
  • Online collaborative learning

JacquesThis is pretty basic stuff, but even today it is surprising how much confusion remains about just what e-learning is, and isn’t. The above simple categorization hasn’t spoiled in the bottle yet because it still seems useful and relevant, and the types suggested are mutually exclusive, and still cover pretty well what we do in e-learning. The little piece also makes reference to performance support and blended learning, two concepts that are still widely in use in the industry.

The WordPress Pig Farm

Some months ago, as I was hustling work contracts, I submitted what I thought was a pretty good quote for a compliance training effort. Online scenario-based training seemed to be a good strategy for the context, and I thought I had pretty deep and relevant experience to apply to the project. The response I got back surprised me a bit.

There was acknowledgement that my experience was good, the quote reasonable in price, and my skills likely up to snuff, but the end client had a look at my little web site, and thought it was, well, too ugly and old-looking. They decided an instructional designer with a web site like this couldn’t be trusted to work in a modern corporate training development context.


lipstickJust a few years ago I had received a number of compliments on my little WordPress site based on the “Dark Essence” theme. And already, I was a dinosaur and my web site was a pig.

It took me a while to get over my righteous indignation at having my instructional design and e-learning abilities questioned because I had the wrong WordPress theme. It didn’t seem very relevant to the creation and implementation of effective and impactful training scenarios. And of course, it’s not.

But I gave in. The kind colleague who had provided me with this project lead was nice enough to suggest some updates to my site, including perhaps one of those new parallax scrolling WordPress themes. This week, at long last, I took another plunge into the WordPress theme quagmire, and as several times before, it was not fun. I knew that putting some needed lipstick on my little web site pig would take way longer than it should, and of course, it did.

The result, as you can see, is very basic. First off, I decided to stay away from parallax scrolling themes. Simply put, they make me sick. Not figuratively. Literally, like those amusement park rides that I used to relish as a kid, but that now make me toss my cookies (not the web kind). Then there were many single page themes, another current rage related in part to the parallax scrolling craze. I knew that if I chose one of those, I’d probably be spending many hours re-jigging the site content, etc., and I couldn’t face that. So I searched for a simple, modern, attractive, reliable, responsive (hey, I even know what that means!) theme that puts the focus on the content. I ended up choosing the Highwind theme. My little pig make-up adventure was not over, however.

placeholder howMy first tiny task with my new theme was to find out how to put my little headshot photo into the obvious placeholder for that photo in the centre of the theme’s top banner/header. In many made-for-regular-humans software tools, this would be simple – click on the placeholder in edit mode, and a dialog comes up, asking you to browse to the image file, maybe providing guidelines on image size, and Bob’s your uncle. But this is WordPress, which is only partly made for regular humans, and in too large part still made for PHP and CSS and who-knows-what coders.

My nice new theme had no accompanying help or directions on this issue, but a Google search quickly revealed that many theme users were wondering the same bloody thing. Why do you have a clear placeholder for the blogger’s photo, yet no instruction anywhere for how to freakin put it in?? About an hour later, the stupid solution was revealed – I had to either try to set up a generic “gravatar” account somewhere (didn’t work), or else find a plugin that would allow me to use my profile picture from Facebook, Google+ or something like that. Highwind theme developer, shame on you. As far as theme design AND customer service are concerned, you got a ways to go.

With my mug now firmly implanted in the banner, my next wish was to reduce the height of the banner, because it’s just too big. Again, you would think this would be a slam-dunk. But this is WordPress. In 2014, “flexible header” themes are just beginning to hit the streets, so with most other themes, changing the height of the header is another exercise in search and frustration. WordPress and theme forums threw out all kinds of code snippets, code changes and possible locations for these edits and additions, and then maybe if 3 planets aligned, the banner would change size. No such luck for me. Another hour of searching and swearing, and my banner stubbornly remained at 600 goddamn pixels high.

Glutton for punishment that I am, I turned to my third task, which was to stop the page titles from being displayed, which is unnecessary and annoying. Guess what. Yet again, something that should be so bloody simple took another 45 minutes, and I was only partly successful! I had to get yet another plugin that provides an option of removing the title from a page. But some of my pages are actually lists of posts of certain categories, so that led to more searching about alternative ways to display categories on pages, and there are still raging debates among WordPress coders about how to do that. After so many years of WordPress supposed evolution and development, with a massive community of coders at work, there is still no basic procedure nor agreement on how to display categories on a page!! WTF!!

For coders, this is all fun of course. Just more challenges to play with. But for regular humans trying to USE WordPress rather than TINKER with it, this is continuous, protracted, lonely psychological warfare. When oh when will the needs of us folks be actually considered by WordPress platform and theme developers and the rest of this community? I’m perfectly happy to pay for themes or other WordPress stuff, but that hasn’t made much difference. Paid themes seem to have similar issues of lousy documentation and needlessly complex functions.

Are we destined to only farm pigs on the web, with or without lipstick? Who will design the first platform to raise racehorses?

How Much CO2 Is Too Much?

Being one of those analytical, left-brain, anal, dorky types, I hate it when people talk about the need to reduce greenhouse gas emissions without putting real numbers on the problem. Oh sure, there’s beginning to be some sort of wishy-washy jello-like sort of consensus on numbers like “80% reductions by 2030″ but that says very little about just how much CO2 and other greenhouse gases our battered atmosphere can take.

The number that many really smart people want us to know more about is 350. This, in climate scientists’ considered opinion, is the atmospheric concentration of CO2, in parts per million (ppm) that we need to be shooting for now if we’d like to avoid some pretty awful consequences.

co2 simulation

Click on image for CO2 Simulation

One problem with this number is that, well, we’re at 390 ppm right now! And this concentration is growing at about 3% per year, so next year it will be about 400 ppm, and so on and so on.

So here’s the thing – and this is a thing that many many people truly don’t realize:  We’re already past the danger level as far as CO2 concentration is concerned and EVEN IF WE STABILIZE  CO2 EMISSIONS NOW THE CONCENTRATION WILL KEEP RISING.

There’s a good animation you can check out that does a good job of making this notion more obvious and clear. Click on the image in this post to see it. (The “danger” level that this animation uses is 450 ppm, a more politically palatable number…) For more on this animation, its designers and the science behind it, click HERE.

So why the more aggressive target of 350 ppm? This is how James Hansen puts it in his famous paper Target Atmospheric CO2: Where Should Humanity Aim? (he’s head of the NASA Goddard Institute for Space Studies and was the guy who famously warned the US Congress about climate change in 1988):

“Alpine glaciers are in near-global retreat [refs]. After a one-time added flush of fresh water, glacier demise will yield summers and autumns of frequently dry rivers, including rivers originating in the Himalayas, Andes and Rocky Mountains that now supply water to hundreds of millions of people. Present glacier retreat, and warming in the pipeline, indicate that 385 ppm CO2 is already a threat.

Equilibrium sea level rise for today’s 385 ppm CO2 is at least several meters, judging from paleoclimate history [refs]. Accelerating mass losses from Greenland [refs] and West Antarctica [refs] heighten concerns about ice sheet stability.  An initial CO2 target of 350 ppm, to be reassessed as effects on ice sheet mass balance are observed, is suggested.”

Now these are just the 2 paragraphs where he sets this target of 350, but there’s a whole lot of science behind WHY this is so important. You can trust me on that, or you can read his paper. Come on, go read it. A half-hour well spent!

And then you’ll begin to understand who the crazy fools are:   the tree-hugging greenies screaming about the need to DECARBONIZE our economy fast, or those business-as-usual types warning us to be cautious in our emissions goals in order to not adversely affect jobs…

Jacques LeCavalier & Associates Inc.
Sustainability Learning that Sticks!
Kelowna, BC

What Stops Personal Action on Climate Change?

Everyone involved in some way in sustainability or environmental work can’t help but get pissed off  and frustrated at the bozos who still idle their monster pick-ups in July while they’re in the liquor store, or the penile implant warehouse outlet.

How bloody much more information, research, cajoling, scary facts, threats from David Suzuki, heat waves and forest fires do we all need before we’ll accept the urgency of climate change and actually DO something significant to stop gushing so much CO2 into the neighbourhood?

Well, according to the American Psychological Association and a huge new report on this very topic, it’s going to take a lot more. The 230-page report (whichi ncludes a 50-page bibliography!), with the nice user-friendly title of Psychology and Global Climate Change: Addressing a Multi-faceted Phenomenon and Set of Challenges – A Report by the American Psychological Association’s Task Force on the Interface Between Psychology and Global Climate Change, and written by no less than 8 PhD’s, lays out the picture in detail… and it’s not that pretty.

The authors spend a fair bit of time outlining the many psychological barriers to individual action and climate change, and the list is long, including things like:

  • Discounting the future and the remote
  • Numbness or apathy
  • Ignorance
  • Uncertainty
  • Mistrust and reactance
  • Denial
  • Judgmental discounting
  • Place attachment
  • Habit
  • Perceived behavioral control
  • Perceived risks from behavioral change
  • Tokenism and the rebound effect
  • Social comparison, norms, conformity, and perceived equity
  • Conflicting goals and aspirations
  • Belief in solutions outside of human control

Not surprisingly, a big part of the solution for the authors is, well, more research.

Wonderful. All we got is time.

Jacques LeCavalier & Associates Inc.
Sustainability Learning that Sticks!
Kelowna, BC

Simple is Good (Sometimes)

Sometimes, keeping things simple is just…. wrong!

findxBut then, sometimes it’s really helpful. As in the following graphic, that we used in a very short e-learning course for Nissan Japan employees. Even if you don’ t read Japanese, you can figure out the jist of the Greenhouse Effect pretty quickly.

greenhouse effect

The graphic seems to be all fun and games, with Mr. Sun being happy or really sad, but it does use some great information visualization techniques really well:

  • the relative sizes of the “energy” arrows make clear what’s escaping from the earth and what’s bouncing back
  • the relative thickness of the atmosphere, even if it’s not really representing reality, shows clearly the effect of all those human-made emissions

With so much video happening, we’ve lost some of the art and craft of scientific and technical illustration, and I really hope we get some of this back.

Jacques LeCavalier & Associates Inc.
Sustainability Learning that Sticks!
Kelowna, BC

Like, Everything’s Connected, Man

parachute catsOne of the difficult realities about sustainability is that it’s so huge and all-encompassing. So one key mindset we need if we’re learning about and working on sustainability issues is systems thinking.

There is a lot to this, but one basic idea is the notion that taking action in one part of a system can often have unintended consequences in another part of the system. Having good examples of this in the ecological realm is a great way to help people to be systems thinkers.

One of the best-known examples is the story of Cats in Borneo, which was retold by Amory Lovins of the Rocky Mountain Institute during RMI’s 25th anniversary gala in 2007. It goes something like this:

Cats in Borneo
A Cautionary Systems Tale related by Amory Lovins

In the 1950s, the Dyak people of Borneo were suffering from an outbreak of malaria, so they called the World Health Organization for help.  The World Health Organization had a ready-made solution, which was to spray copious amounts of DDT around the island. With the application of DDT, the mosquitoes that carried the malaria were knocked down, and so was the malaria.

There were, though, some interesting consequences.  The first was that the roofs of peoples’ houses began to collapse on their heads.  It seems the DDT not only killed off the mosquitoes, but it also killed off a species of parasitic wasp that had theretofore kept in check a population of thatch-eating caterpillars.  Without the wasps, the caterpillars multiplied and flourished, and began munching their way through the villagers’ roofs.

That was just the beginning. The DDT affected a lot of the island’s insects, which were eaten by the resident population of little lizardy creatures call ginkos. Overtime, the ginkos begin to accumulate pretty high loads of DDT, and while they tolerated the DDT fairly well, the island’s resident cats, which dined on the ginkos, didn’t.  The cats ate the ginkos and the DDT in the ginkos killed the cats. Without any cats,  the island’s population of rats multiplied and flourished, and we all know what happens when rats multiply and flourish.  Pretty soon the Dyak people were back on the phone to the World Health Organization, only this time it wasn’t malaria they were complaining about. It was septisemic plague, which, being universally fatal and untreatable, was way worse than the malaria.

This time, though, the World Health Organization didn’t have a ready made solution and had to invent one. The result, believe it or not, was to parachute live cats into Borneo.  “Operation Cat Drop,” courtesy of the Royal Air Force.

The moral of the story is that if you aren’t thinking about the system you’re working in and the relationships among its parts, your solutions are likely going to be the cause of  even bigger problems.”

NOTE:  My apologies to the owner of the great graphic. I’ve misplaced the info and can’t seem to find it again.

Jacques LeCavalier & Associates Inc.
Sustainability Learning that Sticks!
Kelowna, BC