This is the first in a two-part series discussing item statistics. By understanding what makes an item good statistically, we can produce better items.
A key statistic used when evaluating items is the "p-value" or "pass rate." This is the proportion of people that get the item right, and can range from 0 (no one gets the item correct) to 1 (everyone gets the item right). When it comes to creating items, it is important to think about the ability of the entry-level technologist. What are the things entry-level technologists are likely to get right rarely, some of the time, and all the time?
We want to make sure our items fall within a certain ideal range. Just as we wouldn't ask a group of NASA engineers what 1+1 equals if we were trying to measure their mathematical ability, we want to make sure our items are not so easy that they tell us nothing that matters about our candidates. On the other hand, we don't want to ask questions that are too difficult either. We wouldn't ask a class of first-year aerospace engineering students how to build the International Space Station.
The middle of difficulty-right where the entry-level candidate is likely to get an item right some of the time-that is what we are after.
