Computerized Neurocognitive Testing in Sport Concussion Management: What Are the Problems?

In a recent post, I gave a brief overview of the use of computerized neurocognitive tests in the management of sports-related concussions.  I mentioned that while these tests are standard practice, they are far from perfect.  Now, I’ll expand on that statement.

As I discussed before, these tests are used to identify subtle problems that can’t be detected using standard neuroimaging techniques or medical exams.  The thinking is that if we can identify these impairments, we can decrease risk of subsequent (and potentially more serious) injuries.  However, factors relating to the tests we use, the athletes who we administer them to, and characteristics of the injuries themselves can severely impair these tests’ abilities to correctly identify individuals who are still suffering from the effects of concussions.

In terms of tests such as ImPACT, studies have generally found low-to-moderate test-retest reliability, suggesting that athletes tend not to perform consistently when they take the test multiple times (see Broglio, Ferrara, Macciocchi, Baumgartner, & Elliott, 2011; Elbin, Schatz, & Covassin, 2011).  This is a major problem, considering that the tests are designed to be administered to the same athlete multiple times.  One contributor to this issue is the presence of practice effects, whereby athletes improve their performance due simply to having taken the test before.  Another factor hampering reliability may be that the construct itself (i.e., impairment due to sports concussion) can’t be reliably measured due to its often subtle and transient nature.

The way in which the tests are used can also hurt their validity.  An advantage of these tests is their ability to be administered to groups of athletes, saving organizations time and money.  In fact, most baseline testing is done in groups.  On the other hand, post-injury testing is done on an individual basis.  Given evidence that group testing may result in poorer performance than individual testing (Moser, Schatz, Neidzwski, & Ott, 2011), comparisons of performance on baseline testing (group setting) to post-injury testing (individual setting) may not be accurate.

Athletes are generally expected to play through injury.  As a result, they may purposefully distort results by engaging in “sandbagging.” Sandbagging is when someone purposefully performs poorly on the baseline test, which resulting in a low score that is easier to equal on post-injury testing (Peyton Manning has admitted to doing this).  The computerized tests do have ways of identifying this; however, little research has been done to determine if they work.

Finally, it remains to be seen if the baseline testing model is effective in reducing the risk of either subsequent concussions or more serious long-term effects.  One study found that a symptom-free waiting period did not reduce the risk of sustaining another concussion (McCrea et al., 2009).

Although computerized neurocognitive tests are far from perfect, it is important to remember that they are just one tool in making return-to-play decisions after athletes suffer concussions.  Physical exams, self-reported symptoms, and balance testing are also used.  Furthermore, as objective measures go, these tests are the best we currently have.  Consequently, their use will continue to be widespread until something better comes along.

References

Broglio, S. P., Ferrara, M. S., Macciocchi, S. N., Baumgartner, T. A., & Elliott, R. (2007). Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42, 509-514.

Elbin, R. J., Schatz, P., & Covassin, T. (2011). One-year test-retest reliability of the online version of ImPACT in high school athletes. The American Journal of Sports Medicine, 39, 2319-2324.

McCrea, M., Guskiewicz, K., Randolph, C., Barr, W. E., Hammeke, T. A., Marshall, S. W., & Kelly, J. P. (2009). Effects of a symptom-free waiting period on clinical outcome and risk of reinjury after sport-related concussion. Neurosurgery, 65, 876-883.

Moser, R. S., Schatz, P., Neidzwski, K., & Ott, S. D. (2011). Group versus individual administration affects baseline neurocognitive test performance. American Journal of Sports Medicine, 39, 2325-2330.