Effect of Content Balancing on Measurement Precision in Computer Adaptive Testing Applications

Year 2024, Volume: 15 Issue: 4, 395 - 407, 31.12.2024

İlkay Üçgül Öcal , Nuri Doğan

https://doi.org/10.21031/epod.1438977

Abstract

This study aims to investigate the effect of content balancing, which involves equal and different weighting of content areas in dichotomous items in computerized adaptive testing (CAT), on measurement precision under different measurement conditions. Conducted as a simulation study, small sample sizes were set at 250, while large sample sizes comprised 500 individuals. The ability parameters of the individuals forming the sample were generated to display a normal distribution within the range of -3 to +3 for each sample. Using the three-parameter logistic (3PL) item response model, a pool of 750 items spanning five different content areas was developed for dichotomous items. The study considered different sample sizes, ability estimation methods (Maximum Likelihood Estimation and Expected a Posteriori), and termination rules (20 items, 60 items, and SE≤.30) as significant factors in the CAT algorithm for examining the effect of content balancing. For each CAT application, measurement precision was assessed by calculating the root mean square error (RMSE), bias, and fidelity coefficients, and these were analysed comparatively. The results showed that bias values were close to zero under all conditions. RMSE values were lowest when the test was terminated at 60 items across all conditions, while standard error termination rules and situations where the test terminated at 20 items produced similar values. Considering all conditions, the highest fidelity coefficient was observed when the test terminated at 60 items. The fidelity coefficient did not vary significantly with other variables. Implementing content balancing in conditions using different ability estimation methods increased the average number of items by approximately one item. While the average number of items in the test slightly increased with content balancing, measurement precision was maintained. Overall, the maximum item exposure rate decreased with content balancing when content areas were weighted equally, whereas it increased when they were weighted disproportionately.

Keywords

computerized adaptive testing, content balancing, measurement precision

Ethical Statement

Bu çalışma simülatif veriler kullanılarak yapıldığından etik kurul onayından muaftır.

Supporting Institution

Hacettepe Üniversitesi Eğitim Bilimleri Enstitüsü

References

• dsaewrwe werwrretre reretrytr