Comparison of several small sample equating methods under the NEAT design

Year 2016, , 96 - 118, 31.07.2016

Serdar Caglak

https://doi.org/10.19128/turje.16916

Abstract

The aim of this study is to compare the performances of Identity, Nominal Weights Mean, and Circle-Arc equating methods under the Non-Equivalent Groups Anchor-Test (NEAT) design. Synthetic equating functions (SFs) of the chosen equating methods (NWS and CAS) were also created using an equal weighting system (w = 0.5). Different sizes of small examinee samples (n = 10, 20, 50, 100) were used to equate new test forms to base test forms. Chained Equipercentile (CE) with bivariate log-linear presmoothing was used as population criterion equating function to compare the performances of the equating methods. Overall, the identity (ID) equating was the most favorable, but the NWS method produced less equating error than the ID and TL equating methods under specific simulation conditions. In the light of the findings, recommendation is given to practitioners. Limitations and directions for further research are also provided at the end of this study.

Keywords

small samples, equating, synthetic functions, NEAT design

Küçük örneklemlerde kullanılan bazı test eşitleme yöntemlerinin DOOT deseni altında karşılaştırılması

Abstract

Bu çalışmanın amacı, Identity (İD), Nominal Weights Mean (NWM) ve Circle-Arc
(CA) test eşitleme yöntemlerinin performanslarını denk olmayan gruplar ve ortak soru
içeren test (DOOT) deseni altında karşılaştırmaktır. Bu yöntemlere ait yapay test
eşitleme fonksiyonları (NWS ve CAS) eşit ağırlıklandırma sistemi (w = 0.5)
kullanılarak ayrıca oluşturulmuş ve sonrasında orijinal test eşitleme yöntemleriyle (İD,
NW ve CA) karşılaştırılmıştır. Yeni test formlarını, referans test formlarına eşitlemek
için farklı büyüklükte küçük örneklemler (n = 10, 20, 50, 100) kullanılmıştır. Chained
Equipercentile (CE) yöntemi loglinear data düzeltme tekniği ile birlikte kullanılarak,
evrendeki test formları arasında fonksiyonel bir ilişki kurulmuştur. Testler arasında
kurulan bu fonksiyonel ilişki, İD, NWM, CA, NWS ve CAS test eşitleme yöntemlerinin
performanslarını karşılaştırmada bir ölçüt olarak kullanılmıştır. Sonuç olarak, İD
eşitleme yöntemi en uygun yöntem olarak tespit edilmiştir. Ancak NWS yöntemi de
bazı durumlarda İD ve TL yöntemlerinden daha az hata üretmiştir. NWS yöntemi,
örneklem sayısının 25`ten az olduğu durumlarda İD’ye alternatif olarak
kullanılabilecek niteliktedir. Sonraki çalışmalarda ise, var olan diğer test eşitleme
yöntemlerinin yapay fonksiyonları test edilmeli ve küçük örneklemler ile
kullanılabilecek en uygun yöntem(ler) tespit edilmelidir.

Keywords

küçük örneklemler, test eşitleme, yapay fonksiyonlar, DOOT deseni

References

• Author, (2015). The use of a meta-analysis technique in equating and its comparisons with several small sample equating methods (Doctoral Dissertation).
• Babcock, B., Albano, A., & Raymond, M. (2012). Nominal Weights Mean Equating: A method for very small samples. Educational and Psychogical Measurement, 72(4), 608-628.
• Brennan, R. L., & Kolen, M. J. (1987). Some practical issues in equating. Applied Psychological Measurement, 11, 279-290.
• Cochran, W. G. (1977). Sampling techniques (3rd ed.). New York: John Wiley & Sons.
• de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
• Hanson, B. A. (1991). A comparison of bivariate smoothing methods in common-item equipercentile equating. Applied Psychological Measurement, 15, 391-408.
• Hanson, B. A., Zeng, L., & Colton, D. (1994). A comparison of presmoothing and postsmoothing methods in equipercentile equating. ACT Research Report 94-4. Iowa City, IA: ACT, Inc.
• Harris, D. J. (1993, April). Practical Issue in Equating. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.
• Harris, D. J., & Crouse, J. D. (1993). A study of criteria used in equating. Applied Measurement in Education, 6(3), 195-240.
• Heh, V. K. (2007). Equating accuracy using small samples in the random groups design. (Doctoral Dissertation). Retrieved from University of Ohio at https://etd.ohiolink.edu/!etd.sendfile?accession=ohiou1178299995&disposition=inline
• Kim, S., von Davier, A. A., & Haberman, S. (2006). An alternative to equating with small samples in the non-equivalent groups anchor tests design. ETS Research Report Series, 2, 1-40.
• Kim, S., von Davier, A. A., & Haberman, S. (2008). Small-sample equating using a synthetic equating function. Journal of Educational Measurement, 45, 325-342.
• Kim, S. & Livingston, S. A. (2010). Comparisons among Small Sample Equating Methods in a Common-Item Design. Journal of Educational Measurement, 47(3), 286-298.
• Kim, S., Livingston, S. A., & Lewis, C. (2011). Collateral information for equating in small samples: A preliminary investigation. Applied Measurement in Education, 24, 302-323.
• Kim, H. Y. (2014). A comparison of smoothing methods for the common item non-equivalent design (Doctoral Dissertation). Retrieved from the University of Iowa at http://ir.uiowa.edu/etd/1344.
• Kolen, M. J., & Brennan R. L. (2004). Test Equating, Scaling, and Linking. New York, NY: Springer-Verlag.
• Livingston, S. A. (1993). Small sample equating with log-linear smoothing. Journal of Educational Measurement, 30(1), 23-39.
• Livingston, S. A., & Kim, S. (2009). The circle-arc method for equating in small samples. Journal of Educational Measurement, 46, 330–343.
• Livingston, S. A., & Kim, S. (2008). Small sample equating by the Circle-Arc method. Princeton, NJ: ETS.
• Livingston, S. A., & Kim, S. (2010). Random groups equating with samples 50 to 400 test takers. Journal of Educational Measurement, 47(2), 15-185.
• Livingston, S. A., & Kim, S. (2011). New approaches to equating with small samples, In A. von Davier (ED.), Statistical models for test equating, scaling, and linking (1st ed., pp.109-122).
• Livingston, S. A, & Lewis, C. (2009). Small sample equating with prior information. Princeton, NJ: ETS.
• Livingston, S. A., Dorans, N. J., & Wright, N. K. (1990). What combination of sampling and equating methods works best? Applied Measurement in Education, 3(1), 73–95.
• Parshall, C. G., Du Bose, P., Houghton, P., & Kromrey, J. D. (1995). Equating error and statistical bias in small sample linear equating. Journal of Educational Measurement, 32, 37–54.
• Petersen, N. S. (2007). Equating: Best practices and challenges to best practices. In N. J. Dorans, M. Pommerich, & P. W. Hollownd (Eds.), Linking and aligning scores and scales (59-71). New York, NY: Springer Science+Business Media, LLC.
• R Core Team (2015). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from http://www.R-project.org/
• Skaggs, G. (2005). Accuracy of random groups equating with very small samples. Journal of Educational Measurement, 42, 309–330.
• Sunnassee, D. (2011). Conditions affecting the accuracy of classical equating methods for small samples under the NEAT design: A simulation study. (Doctoral dissertation). Retrieved from the University of North Carolina at Greensboro at http://libres.uncg.edu/ir/listing.aspx?id=8164
• van der Linden, W. J., & Wiberg, M. (2010). Local observed-score equating with anchor-test designs. Applied Psychological Measurement, 34(8), 620-640.