As outlined at previous staff meetings, postgraduate students are expected to attend a series of lecture courses over the 3 years of their PhD. These courses are intended to improve the student’s level of awareness of the background of general Chemistry to which his/her research will contribute. This is because it is felt that at present, Bristol Chemistry students become expert at their own very focused area of research, while only having a limited concept of how their work fits into the general scheme of things. These courses are supposedly compulsory, but since at the moment there exists no method of properly assessing the students on this work, very few students bother to attend any of the lectures.
However there is a lot of resistance to formal examinations, both from the students, who feel it would be an infringement on their freedom to organise their PhD in the way they choose, and from the staff who do not wish to spend any more of their precious time marking extra work.
What we need is a method to informally assess the students, that:
· ensures that they are attending the necessary lectures
· checks that they are learning (or at least becoming more familiar with) the background information deemed to be complimentary to their PhD studies
· is easy to administer and mark
· has minimum paperwork and is cost effective in terms of money and time.
For several years the Educational Technology Service (ETS) has been developing a system for delivering multiple choice questions (MCQs) to students using the Worldwide web. The questions are written in a sub-set of HTML (the language of the WWW) called TML, and can be easily created by simple conversion from text or Word documents. The questions reside on a host computer (e.g. sneezy) and can be accessed by anyone with a suitable web browser. (If necessary this can be made such that only people with .bris addresses can use the system).
The system can be used in several ‘modes’:
(a) Teaching mode: - Various web pages (which can contain small or very large amounts of text, images, diagrams, rotating molecules, etc) of information are presented to the student, who reads them and then answers a few MCQs as he/she progresses through the document(s). The questions are really there to act as self-directed learning aids, and the student’s score is not important.
(b) Assessment mode:- The text is short, and maybe only consists of the MCQs themselves. The marks are collated automatically by the computer.
(c) A combination of the two: This may provide a template for possible future ‘distance learning’ projects.
The TML system is designed for use by undergraduates, however, at present in Chemistry, the undergraduates do not have sufficient access to the appropriate computing resources (networked PCs running Netscape) to make this feasible. Postgraduates, however, especially those in the Physical/Theoretical section, have ample access to suitable computers, and so would make prime candidates for testing the viability of this approach.
There are a number of different question types that can be used, the most simple to implement of which is ‘one correct answer out of, say, 5 possibles’. For example:
S.E.M. is a powerful technique for studying microscopic structures. What does it stand for?
(a) Secondary Electron Microscopy
(b) Secondary Elemental Microstructural analysis
(c) Standard Emission Magnification
(d) Surface Enhanced Mode
(e) Scanning Electron Microscopy
If the student clicked on the correct answer (e), he/she would move on to the next question having scored maximum marks (3). If they chose the wrong answer they lose a mark and can have another attempt, until they have zero marks remaining. They can also ask for hints (at the cost of a mark each time). The responses for a correct or false answer can be as detailed (or as brief!) as the question setter likes. The fuller the explanation, the more the system resembles teaching mode, the briefer, the more like assessment mode.
Each student will have a unique identifier (i.d. and password) and their marks will be automatically collated by the software. This makes it possible for the system administrator to see:
(i) whether each student has done the required number of tests
(ii) what they scored
(iii) which questions they got right and wrong. This could be very useful, for example if every student got, say, question 3 wrong, we’d know that that part of the course wasn’t understood properly (or taught well!).
As in any form of assessment, the main requirement is a set of questions from the members of staff teaching the appropriate courses. In the present proposal, this would mean that all lecturers who are teaching courses that are recommended to postgraduates will need to provide a set of suitable MCQs. Typically, I’d imagine that this would be around 20 or 30 questions (plus 4 wrong answers and 1 right answer), and any hints that might be useful, associated text (if any), diagrams, etc. The questions would simply be written in text or in Word format, and converted to the TML format using a utility being developed by the ETS. The MCQs needn’t be that difficult (at the moment), since all we’re attempting to do is improve the general background knowledge of the students. So, asking what the definitions of certain key terms are, would be a good example. I’d estimate that a suitable set of basic text Q&As could be written in a couple of hours. More complicated MCQs, incorporating graphics, images, reaction schemes, etc, would of course take longer to prepare, but might be a more valuable learning experience for the student. There is an example set of MCQs, showing all the various types of question that can be used with graphics and equations at:
http://www.ets.bris.ac.uk/bin/tdemo.p/ets/resource/tutorial/p.may/
chemtute.tml?Intro=%2fets%2fresource%2ftutorial%2ftutorial.htm
For every course that is recommended to PhD students in the Physical/Theoretical sector there would need to be set of MCQs. The students would be told that it would be compulsory for them to complete the Web Assessment Tests for each course they attend. The students will be monitored automatically to ensure that each one completes the minimum required number of tests, and scores a reasonable mark. Any student scoring badly or not completing the required number of tests will be dealt with appropriately.
1.
All the recommended
courses will need to be covered, otherwise the students will just claim they
only attended the courses for which there were no MCQs. This means the system would have to become
section policy in order that all staff members cooperate. But some courses are given by other Depts (e.g. Physics, IAC,...), and we’d need to
get each staff member involved there to go along with this scheme as well. They, of course, might claim they’ve got
better things to do than write MCQs to assess Chemistry students. However, some of these Depts, too, are
looking at Web-based MCQs for assessing their own students - so any questions
they provide might be useful for assessing their own students as well.
2. Since the students can do the tests unsupervised in their own time and at any location, there is obviously the possibility of cheating, e.g. students working together or rote copying correct answers. This may become an issue if the system is ever used for officially assessing undergraduates, but for postgraduates I feel it doesn’t really matter that much. Just the act of completing the test will force the student to encounter some of the new ideas, concepts and terminology that we wish them to be familiar with. After all, any form of assessment is better than the none we’re doing at present.
Obviously, this is only the forerunner of more sophisticated Computer Based Learning (CBL) projects which may be run in the University (and the Chemistry Dept?) in the near future. At present we are proposing that only the Theoretical/Physical section adopt this method of assessment, unilaterally, as a way of efficiently and informally assessing only its postgraduates. This will address an immediate need in the sector. But as suitable computers become more widely available, other sectors, undergraduate courses, lab sessions, and some tutorials may be linked in.
This is already happening in other Depts in the University (e.g. Geology), where CBL approaches have been shown to be remarkably successful (in terms of improving teaching standards) and popular with students. In fact Paul Browning from Geology is currently co-ordinating a bid to the HEFCE to fund a University-wide CBL scheme based on TML or a similar system. If successful, the proposal will be worth around £250k to the University, of which most will go towards paying for a programmer and administration, but each Dept involved (presently around 6 of them) will get around £8k to cover ‘administrative costs’. The commitment on each collaborating Department’s side (including Chemistry) will be to provide a certain number of MCQs per year for the duration of the scheme (3 years), which will form part of an expanding database of questions and answers. Other Universities will also be involved, and we can envisage a time in the near future when the database has grown sufficiently large to become a useful resource. New tests or tutorials could be created very rapidly by searching the database for keywords (e.g. polymer, lithium chemistry, etc), choosing any questions that are relevant to the course, writing any extra ones that may be needed (and obviously, adding these to the database as well!), and assembling them all into a suitable sequence. Voilá - instant CBL tutorial!
So if the bid is successful, our initial contribution to this will be the MCQs that we’ll be using to assess the postgrads, and maybe later, ones we might be using to assess undergrads. We’ll also have access to the database with similar questions from other Universities which will become more useful as time goes on. Note: This scheme will be completely compatible with the one that Dr Maher is currently working on with people in Engineering Maths, and it is envisaged that question sets from one will be easily convertible to the format of the other.
Whether or not this bid is successful, the TML system offers a method to address the immediate problem of postgraduate assessment in our sector, in a cost effective and efficient manner. It may also have added benefits too, in that students will become more computer literate and internet aware (necessary transferable skills in today’s society) and may help the case for improving computer resources in the Dept for postgrads and undergrads alike.