Learning Blog

Random despatches from places where L&D meets software and systems

How to write qorking quiz questions

ATH is a UK-based pharmaceutical sales and marketing manager who commissions his own sales training content. He writes “My biggest frustration with our company’s LMS is that each course contains two links, one to the course content proper, the other to the associated testing. These two links appear side by side in people’s online learning plans. Becoming accredited is mandatory for our salespeople but because they’re really busy, they just keep on doing the multiple choice tests until they pass, don’t access the content, don’t learn very much and therefore don’t get any better at selling. Our LMS people are in the US and big change requests seem to take an eternity and cost a fortune. What can I do?”

The intransigence of LMS systems, and by extension their vendors and sometimes even their proponents in your own organisation, can often be quite dispiriting and I fear it may be a little while yet before LMSs no longer feature in the top 3 hassles of “federated” knowledge workers at the coalface such as yourself.

Strategically, you will be able to bring to bear more political pressure on your LMS guys if you can produce some evidence to suggest that the salespeople who go through your content (assuming you can measure that!) sell more product. Failing that, I’d suggest a combination of the following tactics. I’ve ordered these so that the work involved becomes less “them” and more “you” as you go through.

  1. Partner with your LMS people to come up with a better learning design. Try embedding more of the testing into the learning proper and make the integrated stuff part of the accreditation
  2. Ask your LMS colleagues if they can “fine-tune” the software such that users are obliged to do the content if their first attempt at the testing falls short of whatever pass mark you think appropriate. This won’t disadvantage the candidate who’s genuinely clued up and doesn’t need to go through the content but it should help minimise the unwanted behaviours happening at the moment
  3. Randomise the presentation order of both questions and answers
  4. Write as big a pool of questions as is practicable
  5. Ensure that all your multiple choice questions are good ones

This last point deserves a little more attention. How many times have we seen examples where it’s possible to guess your way through a load of poorly written questions, have no idea of the subject matter but still do “really well”?

Please try to ask questions which have answers worth knowing from a commercial point of view. I can’t really help you with this bit but I can offer some advice on how to ask your questions in the right way, or, rather, how to avoid making them too guessable.

To write a good “stem”, i.e. the question itself:

  • Make sure you get the level right, not too easy, not too difficult
  • Write your stem in such a way as to minimise duplicated words in the candidate answers
  • Use clear and concise language, avoiding negatives wherever possible

Writing good distractors is the trickiest bit. All your distractors should be plausible; and the more similar your distractors are to the correct response and to each other, the less guessable the item will be. The £100 questions from “Who Wants to be a Millionaire” are often quite funny because they demonstrate how a perfectly good question can become a dead giveaway when implausible distractors are revealed.

Aside from plausibility, other things to watch out for include:

  • The correct response contains a word repeated from the stem but the distractors don't
  • The use of words such as "all", "never" and "always" in distractors and “some”, “sometimes” "usually", "often" etc. in correct answers – another giveaway
  • Correct answers are very often longer than distractors
  • Watch out that you don’t inadvertently plant any grammatical clues; if your stem expects a simple noun (i.e. it ends in “… because of:”), for example, then don’t start a distractor with anything else (“B. the sky is blue”)

When you write your feedback, don’t just say right or wrong. Try to add some value even if it’s just some extra piece of information to keep your users interested.

To finish off, here’s a little example to show how you can include numbered options in your stem to make your people have to think that little bit harder:

 

Question

Which of the following are towns or cities in Romania?: 1) Odessa; 2) Oradea; 3) Arad; 4) Nesebâr; and 5) Timişoara

Correct answer:

2, 3 and 5

Distractor1:

2, 3 and 4

Distractor2:

all but 1

Distractor3:

1, 2, 3 and 4

Feedback:

All but Odessa, which is in the Ukraine and Nesebâr, which is in Bulgaria

 

Functionally, this fails the commercially worth knowing criterion for a pharma salesperson, of course, but formally, at least, its construction is OK:

  • it’s not easily guessable (notice I didn’t include Bucharest, Sofia or Kiev)
  • all options contain more than 1 candidate city (which … are, not is)
  • and it provides a little bit of extra interest in the feedback

Lastly, if you are in any doubt, you should ask other people’s advice as to the “guessability” of your MCQs before you publish. Good luck!


TOP