Psychometricians work to ensure the reliability, validity, and fairness of assessments. In certification testing, psychometricians also play a critical role in ensuring that assessments are legally defensible.
Having spent the past 25 years working on educational and certification assessments for various organizations, I've often been asked what my favorite tasks are as a psychometrician. It's hard to pick a favorite. So, here are my top three favorite tasks as a psychometrician:
- Standard Setting
- Exam Form Creation
- Item Review (with a little help from statistics)
In Part 1 of this series, I'll focus on standard setting, the systematic and standardized procedures by which a cut score is set for an exam that results in the assignment of a number that differentiates between a failing and a passing performance.
So, why is standard setting a favorite task of this and many other psychometricians?
First, standard setting requires that a psychometrician collaborates with subject matter experts (SMEs) on the process. We train the SMEs on the standard setting methodology and work with them over several days to follow a process that produces a number that represents the passing score, which is then presented to the Board of Trustees for approval. Psychometricians often work alone doing statistical analysis and psychometric modeling. The chance to work with SMEs on this important task grounds psychometricians, allowing us to understand the impact of our work (i.e., the numbers that we help set) on real people doing real jobs in the real world.
Second, standard setting, if done properly, following a standardized and repeatable methodology, allows for valid inferences to be made based on the pass/fail status of the candidate. If a candidate passes the exam, we can validly infer that the candidate has the requisite knowledge, skills, and abilities to perform the job. This in turn makes the assessment fair for all candidates. And lastly, because we followed a systematic standard setting procedure and did not simply set an arbitrary and capricious cut score, it makes the use of the pass/fail decision from our certification exam legally defensible.
So, standard setting is a critical task in certification testing. It's a hugely important task that ensures the validity, fairness, and legal defensibility of our assessment program.
In Part 2, I'll address why building exam forms is another favorite task among psychometricians.
