Darwinism in Adelaide

I had an interesting conversation recently with my Smaller Half's uncle.  He's a big-wig hot-shot top-banana doctor and scoots all over the world examinating other doctors for entry into whatever country it is's Medical College of Bananaology.  (I don't mean to be disrespectful to him but I'm trying to disguise his identity.)

We were talking about exams and I was having my customary moan about the quality of exam questions that I've been getting at this cut-rate bottom-tier two-bit university that I attend.  (I don't mean to be disrespectful to it - just trying to disguise its identity.)

What really tees me off in exams is ambiguity.  A good example would be the true/false question: "PTR is awesome".  Does this mean "PTR is always awesome" (false) or does it mean "PTR is sometimes awesome" (true)?  It depends who wrote the exam.  Similarly, I tend to get into trouble in exams because I interpret the question very literally (I once had a psychologist explain to me that this is very common in engineering/computer types).  For example, in our recent OSCE I was asked to take a patient history to determine the cause of anaemia.  So that's what I did.  And I got pissed off when I lost a heap of marks for not asking about the various symptoms of anaemia.  Well excuse me.


Anyway, I've gotten a little off track here.  The point is, he (the top-banana doctor uncle) agreed with me that questions need to be very carefully written.  Then he said something that was very interesting.  At first it seemed to make sense but the more I think about it the stranger it seems.


What they do is they have a small number of "trial" questions on each exam which don't contribute to your final mark.  They are used by the examination board to try out questions for future years.  A successful question will be done well by the people who do well on the test and poorly by the people who do poorly on the test.  Otherwise, it's a bad question and they'll throw it out or try to rewrite it.


At first, this seems eminently sensible.  You want to select questions which discriminate between good doctors and bad doctors.  But here's the problem: what is a good doctor and a bad doctor?  A good doctor is one who does well on good questions and badly on bad questions.  A bad doctor is one who does badly on good questions and well on bad questions.  The best students are those who are best at answering the questions that are best answered by the best students who answer the best questions best. 

The whole thing is a horrible rats-nest of circular definitions and recursive functions.  If we're not careful, not only will we end up with dodgy doctors, we'll also cause a stack overflow error and the universe will crash.