Fast-Paced Change: With enough candidates testing in a profession experiencing fast-paced change, you can introduce new items on an expeditious schedule.
“How soon should we schedule our next item-writing workshop?” a client asked earlier today. The answer, of course, was, “It depends.”
To associate a timetable with the test-maintenance cycle, a test sponsor needs to answer two larger questions:
- How fast does the knowledge associated with the credential evolve?
- How long can an item be in use before you consider it overexposed?
If the credential is in an industry with fast-paced change, then a quarterly cycle may be indicated. A new item written and reviewed in Quarter 1 may be pilot tested in Quarter 2 and go live, as a scored item, in Quarter 3.
In an industry with a slower pace of change, the biggest driver of new content development is likely to be item exposure. It’s hard to say how much exposure is too much. I am aware of no solid guidelines on how many candidates or how many weeks is too many.
Slow-Paced Change: If you retire test items three years after first exposure, you need to introduce new items every year. If you allow items to serve longer, you can write new ones less often.
For me, a default would be to retire an item no more than three years after its first exposure. What does that mean in practice?
It means that the test needs to be refreshed at least annually. An item piloted in Year 1 could be used as a scored item in Year 2, reused in Year 3—possibly as an anchor for statistical equating—and then retired.
It would be good to use an item as a scored item for more than two cycles. That would require shorter cycles: a test refreshed every quarter or every six months.
Alternatively, of course, a more liberal maximum-exposure rule could be adopted. Considerations would include the stakes of the test, the culture of the test constituency, and the consequences of overexposure:
Bad item overexposure: You know the driver licensing test includes exactly two questions on traffic signs; one on the “Yield” sign and one on the “No U-Turn” sign. You skip learning all the other signs—and pass, doing yourself and the world no favors when you get behind a wheel.
Benign item overexposure: You know the food-safety test includes a question on the best hand-washing method. You learn the best hand-washing method. Everyone wins.
These considerations fall into the category of preventative maintenance. A periodic statistical security check can alert test sponsors, after the fact, to possible overexposure. An undesirable outcome on such a check-up would be a sign that the development cycle is too long.
This is the longish answer to the original question. The short answer: consider an annual cycle to refresh your test. If you have reason to adopt a faster cycle, do so. If you can justify a slower cycle, or must adopt one for practical reasons (most often limited financial or volunteer resources), refresh no less than once every other year.