How to prepare Multiple Choice Questions for use with TML

Paul May, 2/5/96

As you may be aware, from next year (October 96 onwards) the Physical/Theoretical Chemistry section will be introducing a scheme to monitor and assess the progress of PhD students using a WWW-based format using multiple choice questions (MCQs). This scheme is called TML (Tutorial Markup Language) and was developed by the Bristol ETS.

Each member of staff that lectures a course that is recommended to PhD students (such as the special graduate lectures, the MSc colloid lectures, and some of the optional final year undergraduate lectures) will be asked to provide a set of 15-20 MCQs, which will then be converted to TML format and placed on the web. The students will then be required to do these MCQs on completion of the appropriate courses. This will allow us to determine, in a relatively painless and simple way, if (a) they've attended the course, and (b) they've learnt anything from it.

I'd suggest that before continuing with this, every member of staff involved should have a look at the example MCQs that I've written to illustrate the range and flexibility of TML. At this stage, we're probably best to limit ourselves to text-only questions, but as you can see in this link, the facility is there for more complicated question types if you wish them.

The level of the questions

The difficulty of the questions is obviously up to you, but since all we're trying to do at present is to check whether they've attended the lectures and learnt something, the level needn't be too high. Suggestions could be asking them to define certain key words which they'd been introduced to in the course, for example:

Q: Which of the following best describes a smectic phase liquid crystal?

  1. no long range or short range order
  2. short range order but no long range order
  3. some degree of short range order. Molecules are aligned in a parallel fashion, forming a layered structure.
  4. molecules retain parallel alignment, but do not possess a layered structure.

Since there is no time limit to the questions, it is possible to ask the students to do complex calculations, however, do not forget the current limitations of displaying mathematical symbols and equations on the web. An example would be:

Q: Use the Arrhenius equation to calculate the first order rate constant for a reaction at 300K, if the reaction has an activation energy of 1 kJ mol-1 and the pre-exponential factor is 50 s-1.

The Format for the Questions

The questions will all be multiple choice type, and I'd suggest a suitable format would be one correct answer out of 5 possibles. TML allows a much wider range of question types (pattern matching, multiple correct answers, etc), and these can be incorporated later if you want (see my example question set).

To start with it will be much easier and less time consuming to write simple text-based questions. These questions can be just given to me on a floppy disk (Word files or simple text files), or even emailed to me. I will then convert them to TML format and set them up in the appropriate directory on the web.

Marking scheme

This can be varied, but I'd suggest that to begin with we standardise it. A suitable scheme would be that a total of 3 marks are available on every question. If the student gets the question correct first time, they get the full 3 marks. They lose 1 mark every time they choose an incorrect reply, so they'll get a maximum of 3 attempts before they are forced to move on to the next question, in that case scoring zero. Hints (see later) may also be available, but these cost 1 mark each time. So if a student got the question wrong at their first attempt (-1 mark), then asked for a hint (-1 mark), and finally got it correct on their second attempt, they'd score 1 mark. Obviously, questions with different formats might have necessarily different marking schemes - an example would be a true/false question, where there are only 2 possible answers and so the student would only get one attempt and score full marks or zero depending on their answer.

Hints

These are optional, but can be included with a question if it's felt that the students may need a bit of extra help. Since hints cost marks, on the standard marking scheme only 2 hints per question can be used. An example is:

Q: The sodium D-line has a wavelength of 589 nm. What colour is this?

  1. ultra-violet
  2. green
  3. orange
  4. blue
  5. infra-red

Hints: (these will be only visible to the student, one at a time, when they click on the Hints option)

  1. Think of street lights
  2. If it's used for lighting, it must be visible light.

Responses

When the student gets the question correct, there will be a standard response, something like 'That is correct - now go to the next question'. This can be modified on a question-by-question basis if necessary. For example if a student gets a particularly difficult question correct first time, the response could be 'Well done! That is correct.'

Similarly, if the student gets the question wrong, the response can be altered accordingly. The default response is 'Sorry, that is incorrect'. But the system knows which of the incorrect answers the student gave, so it could give a specific reason why that answer was wrong. For example 'Sorry, that answer is incorrect. You forgot to change the units of temperature from Celsius to Kelvin'. These responses can be as brief, or as detailed as you wish, or alternatively, you can just use the defaults each time.

Images

Use of images in the questions or answers will certainly make the page more interesting, and maybe make the session more enjoyable for the student. But this will require more work from the question setters and so unless you are feeling particularly keen you may want to restrict yourself to text only at present. Having said that, images can be incorporated into TML questions with a minimum of effort, so long as they are provided in standard web-friendly formats. This means .gif or .jpg files only. Also it should be remembered that images should be kept fairly small, i.e. less than a quarter of a screen, otherwise the student will have to keep paging up and down to see what's going on.
Also, bear in mind that if you want to use complex mathematical equations or expressions, or unusual Greek symbols, in your questions or your answers, at present there is no way to display them, except to make them into .gif images.

Deadlines

Since it will take some time to set this whole scheme up, and it's meant to be fully functioning by October, I need the question sets as soon as possible. Once I have two or three such sets I can begin testing the system. I realise that some of you won't have even written the lectures yet, but can I ask that as soon as you've done so, to give me the required MCQs as well.

What exactly do I require?

For each course you are responsible for, I will need a set of around 15-20 questions. Each question should consist of:

  1. The Question itself
  2. A set of up to 5 possible answers -with the correct one indicated!
  3. The marking scheme, if different from the standard one described above
  4. (Optional) up to 2 hints
  5. A response for a correct answer, if different from the default.
  6. (Optional) a set of responses for some, or all, of the possible incorrect answers.
  7. Any images or pictures that are required as separate gif or jpg files.
You can use the following templates (cut and paste them into your editor, e.g. Word) to create your own questions (later on in the coming year, I hope to have a stylesheet for Word, written by the ETS, which does this for you automatically).

Example 1- the simplest text based MCQ

Question number 1
Marking scheme: default (3 marks total, -1 for incorrect answer, -1 for hints).
Question: Which of these is a noble gas?
Answer (a) Hydrogen
Answer (b) Ar
Answer (c) HCl
Answer (d) NaCl
Answer (e) Nitrogen
-------------------------
Correct answer is (b)
Response for correct answer = Default ('That is correct')
Response for incorrect answer = Default ('Sorry, that is incorrect')
-------------------------
No Hints available

Example 2 - the same text based MCQ with additional hints and extra responses

Question number 2
Marking scheme: default (3 marks total, -1 for incorrect answer, -1 for hints).
Question: Which of these is a noble gas?
Answer (a) Hydrogen
Answer (b) Ar
Answer (c) HCl
Answer (d) NaCl
Answer (e) Nitrogen
-------------------------
Correct answer is (b)
Response for correct answer = Default ('That is correct')
Response for incorrect answer = Non standard ('Sorry, that is incorrect - perhaps you should ask for a Hint...')
Response if (e) chosen = 'Sorry, that is incorrect. Nitrogen is inert, but is not a noble gas'.
Response if (c) chosen = 'Sorry, that is incorrect. NaCl is a solid crystal, not a gas!'
-------------------------
Hint 1 = A noble gas is inert
Hint 2 = Because they are inert, noble gases are monatomic

Example 3 - a non standard true/false question

Question number 3
Marking scheme: non standard (1 mark total, -1 for incorrect answer).
Question: Using the Arrhenius equation, the activation energy for a reaction can be found from the slope of a plot of ln k versus T. True or False?
Answer (a) True
Answer (b) False
-------------------------
Correct answer is (b)
Response for correct answer = Non standard - 'That is right. The correct plot is ln k versus 1/T'.
Response for incorrect answer = 'Sorry, that is incorrect. The correct plot is ln k versus 1/T'
-------------------------
No Hints available

Example 4 - an image-based MCQ

Question number 4
Marking scheme: default (3 marks total, -1 for incorrect answer, -1 for hints).
Question: This shows a diagram of a surfactant molecule with a hydrophilic head group and a hydrophobic tail. What does hydrophilic mean?
- Associated image: file called 'surf1.gif'
Answer (a) Water liking
Answer (b) Water hating
Answer (c) Hydrogen bonded
Answer (d) Hydrogen-containing
Answer (e) Water divining
-------------------------
Correct answer is (a)
Response for correct answer = Default ('That is correct')
Response for incorrect answer = Default ('Sorry, that is incorrect')
Response if (c) or (d) chosen = 'Sorry, that is incorrect. Hydro- refers to water...'.
-------------------------
Hint 1 = Surfactants operate on the surface of a liquid
Hint 2 = Hydro- refers to water.

Example 5- MCQ with images in the answers

Question number 5
Marking scheme: default (3 marks total, -1 for incorrect answer, -1 for hints).
Question: Which of these is the structure for para-toluene?
Answer (a) Image - file = 'orthotol.gif'
Answer (b) Image - file = 'metatol.gif'
Answer (c) Image - file = 'paratol.gif'
Answer (d) Image - file = 'benzene.gif'
Answer (e) Image - file = 'phenol.gif'
-------------------------
Correct answer is (c)
Response for correct answer = Default ('That is correct')
Response for incorrect answer = Default ('Sorry, that is incorrect')
-------------------------
No Hints available