Find solution
How do I set my questions to private?

How are the questions for major standardised examinations set?

  • I would like to know how the questions for major standardised examinations are set. What are some resources which I should be looking at? Could you recommend some articles or books or blogs which talk about how examination questions for a large-scale exam are written? (I prefer the written format, but if there is a particularly informative podcast or documentary I am also open to listening or watching it.) I am not very picky about which large-scale standardised examinations they are, or which time frame this refers to. I'm fine with things like SATs, gao kao, O-levels, A-levels, Korea's college scholastic ability test, and Chinese imperial exams as long as the papers are fairly standardised and there is extant information about how the questions were set. Here are some examples of the things I would like to know, if it helps. Who and what entities are involved in this process (how are they picked?) How the whole process goes - from thinking of questions to finished paper Timeframe and time limits Considerations while setting questions, and review process if any Exam format (and maybe a bit of history related to the format?) Weird facts which aren't apparent when considering the issue on a surface level Thank you!

  • Answer:

    I served on the committee the wrote the exam for a large, international professional certification. Here is how that worked: - There were exam writing sessions in multiple cities (I think the cities were: Bejing, Mexico City, Chicago, San Diego.) - To be selected to write the exam you had to hold the certification and practice in the field. There was an application process that made sure we had a diversity of experience. - The sessions went on for 3 days - 8 AM to 6 PM. The professional association paid a large part of your travel expenses (hotel and food). - The sessions were moderated by professionals who had a ton of experience in testing. Day one was training in how to write test questions. - To make sure questions were clear and and correct. The questions were written in one city, then validated in two cities. (In other words, the San Diego group wrote questions which Chicago and Mexico City checked.) - We worked in groups of 4 people and were assigned a very specific topic for questions. - Every question went directly to one of the 20 books that were suggested for test preparation. (As in, we took the answers from Chapter 10 of XYZ book. Answer A is correct and shown on page 100, Answer B is incorrect (page 104)...That specific) - All the questions were then reviewed by the certification group and field tested in actual tests. (If the certification exam had 500 questions, 20 questions were "test" questions.) Overall, I was impressed by the thoughfulness of how the test was created. It was a process of practitioners writing questions, education/assessment professionals managing the process and the rigorous testing to make sure the questions were clear and fair.

35minutes at Ask.Metafilter.Com Visit the source

Was this solution helpful to you?

Other answers

Speaking very generally, the discipline of http://en.wikipedia.org/wiki/Psychometrics is something you will be interested in looking into. In short, there are psychologists whose job it is to set the kinds of questions to be asked and the kinds of responses, and to do studies to ensure that a test type is measuring what they want it to measure, and that it measures consistently across different populations. (Jump down to "key concepts" for more on this.) Once you've set the type of questions and answers that are called for, you have people who write the questions, and then another set of people who review the questions and answers to ensure they fit the requirements, that they don't include any forbidden materials (for example literary passages that are about controversial subject matter, or that require outside knowledge to interpret, that sort of thing), and that the correct answer is unambiguously correct. In the US, the company http://en.wikipedia.org/wiki/Educational_Testing_Service is the dominant maker of standardized tests (SAT, GRE, AP tests, etc). LSAS makes the LSAT, etc. Companies and other organizations may have their own in-house psychometric staff to develop hiring and promotion tests. From my experience, again in the US, people can get hired to write questions ("items") on a contract basis (i.e. without being a permanent employee of a company like LSAS). Writing questions, the question-writer is generally assigned several types of question (roughly, features a passage and the answers should have), and asked to write a specified number of each type. You can often choose the material for the passage - for example, you might get a stack of nonfiction books from the library and page through them looking for passages that are suitable, that allow you to write a question of the form stipulated. You write the correct answer and the "distractors" (the wrong answers). Once you have written up all your questions, you would send them in or hand-deliver them, and they would be reviewed by at least one person, sometimes by a committee or several layers of reviewers.

LobsterMitten

My knowledge is old but it used to be that the GRE multiple choice questions were normed (tested to see things like item-total correlations to determine the difficulty) the year before they are counted in the actual score. That's how the study guide companies get them

srboisvert

Also this process is conducted with a reasonably high level of security. Your materials are to go in a locked safe when you're not working on them, transmission is to be encrypted or otherwise secure, you have an obligation of complete confidentiality, once you're finished with the work you're meant to destroy all copies on your end.

LobsterMitten

This is probably much more technical than what you are looking for as an intro to psychometrics, but http://www.lsac.org/lsacresources/research/all/rr are a bunch of reports from the Law School Admission Council regarding their research on LSAT questions.

oakroom

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.