Diagnostic Linking Pattern (not to scale)

At Grockit we believe in collaborative learning and continual improvement. This is an ideal we encourage both in our users and in ourselves. Towards this end, we continually research and develop more exact methods for estimating question difficulty and student ability to enhance the Grockit experience.

Grockit diagnostics indicate current capability level for our users by capitalizing on the mathematical tools of item response theory. These are the same methods used by major testing companies to create computer adaptive tests (CAT) such as the GMAT.

Our capacity to estimate current capability for the individual depends upon the belief that if item A is more difficult than item B, item A should be more difficult than item B consistently across people.

Getting the item difficulty estimates was one of the early keys to providing better feedback for our members. Early on, this was a fairly arduous process because data from practice questions left us with a sparse data matrix that required many levels of data filtering to ensure reliable estimates. As an example of the filtering, people were only included in these calculations if they had both correct and incorrect responses to practice questions. We could not include data from those people because the information did not help us determine which questions are more or less difficult.

In addition to our desire to get away from the intensive filtering, we also wanted a way to get difficulty estimates for the new questions that we publish on a continuous basis.

Our new approach entails making our diagnostics dual purpose. Not only do the diagnostics serve to estimate members’ current capabilities, they also increase the accuracy of our estimation process. We have implemented a shifting diagnostic schedule in which certain questions serve as links from group to group, enabling us to place all of our data on the same metric. Newly published questions are also included so that every question on Grockit gets a difficulty estimate as soon as possible.

More importantly for our members, we already have solid difficulty estimates on enough diagnostic questions to generate the ability estimates necessary for targeted quests. With the new shifting diagnostic plan, we are now able to rapidly turn out new content with scientifically-based difficulties and experience points. Take a diagnostic on Grockit and see where you rank!