Validating the Cognitive Diagnostic Assessment and Assessing Students’ Mastery of ‘Parallel and Perpendicular Lines’ Using the Rasch Model

Year 2022, Volume: 9 Issue: 6, 436 - 452, 01.11.2022

Huan Chın , Cheng Meng Chew , Wun Yew , Muzirah Musa

https://doi.org/10.17275/per.22.147.9.6

Cited By: 1

Abstract

‘Parallel and Perpendicular Lines’ is an important topic that serves as a basis for the learning of a more advanced geometric concept in later years. Yet, this topic is hard to master by the students. To pinpoint students’ weaknesses in this topic, this study sought to develop a cognitive diagnostic assessment (CDA) to assess students’ mastery of ‘Parallel and Perpendicular Lines’. The validation of the CDA and the use of CDA in measuring students’ mastery of ‘Parallel and Perpendicular Lines’ was documented in this article. The content validation involved two subject matter experts, while the pilot test involved 154 Year Four students from Kedah state of Malaysia selected using cluster sampling. The experts' consensus on the relevancy of test items was captured by calculating the content validity index. The psychometric properties of items and reliability of assessment were analysed based on Rasch Measurement Model. The validity of the assessment content was supported with an acceptable content validity index of 1.00 (>.80). The findings of Rasch analysis span across all ranges of abilities level and hence fit students’ competence well. With an acceptable person separation index of 1.58 (> 1.50), person separation reliability of .74 (>.70), and KR-20 coefficient of .78 (>.70), the CDA developed is reliable. The findings of assessing students’ mastery level highlighted their weaknesses in defining the properties of perpendicular lines and drawing perpendicular lines. The findings of this study would encourage practitioners to utilise it in the mathematics classroom for diagnosing students’ weaknesses and hence plan for remedial instruction.

Keywords

cognitive diagnostic assessment, elementary school, geometry, parallel and perpendicular lines

References

• Akbay, L., Terzi, R., Kaplan, M., & Karaaslan, K.G. (2018). Expert-based attribute identification and validation: A cognitively diagnostic assessment application. Journal on Mathematics Education, 9(1), 103-120
• Alkhadim, G. S., Cimetta, A. D., Marx, R. W., Cutshaw, C. A., & Yaden, D. B. (2021). Validating the Research-Based Early Math Assessment (REMA) among rural children in Southwest United States. Studies in Educational Evaluation, 68, Article 100944. https://doi.org/10.1016/j.stueduc.2020.100944
• Alves, C. B. (2012). Making diagnostic inferences about student performance on the Alberta Education Diagnostic Mathematics Project: An application of the Attribute Hierarchy Method (Doctoral thesis). Available from ProQuest Dissertations and Theses database. (Publication No. 919011661)
• American Educational Research Association, American Psychological Association & National Council on Measurement in Education. [AERA, APA, NCME] (2014). Standards for educational and psychological testing. Washington, DC: AERA.
• Baghaei, P. (2008). Local dependency and Rasch measures. Rasch Measurement Transactions, 21(3) 1105-1106.
• Bardhoshi, G., & Erford, B. T. (2017). Processes and procedures for estimating score reliability and precision. Measurement and Evaluation in Counseling and Development, 50(4), 256–263
• Baykal, A., & Circi, R. (2010). Item revision to improve construct validity: A study on released science items in Turkish PISA 2006. Procedia-Social and Behavioral Sciences, 2(2), 1931-1935.
• Bond, T. G. & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum Associates.
• Boone, W. J., & Noltemeyer, A. (2017). Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1), Article 1416898. https://doi.org/10.1080/2331186X.2017.1416898
• Bradshaw, L. (2017). Diagnostic classification models. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 297–327). Wiley Blackwell.
• Brendefur, J. L., Johnson, E. S., Thiede, K. W., Strother, S., & Severson, H. H. (2018). Developing a multi-dimensional early elementary mathematics screener and diagnostic tool: the primary mathematics assessment. Early Childhood Education Journal, 46(2), 153-157.
• Broaddus, A. E. (2011). An investigation into foundational concepts related to slope: An application of the Attribute Hierarchy Method (Doctoral thesis). Available from ProQuest Dissertations and Theses Global database. (Publication No. 3487353)
• Chin, H., Chew, C. M., & Lim, H. L. (2021a). Development and validation of online cognitive diagnostic assessment with ordered multiple-choice items for ‘Multiplication of Time’. Journal of Computers in Education, 8(2), 289-316.
• Chin, H., Chew, C. M., Lim, H. L., & Thien, L. M. (2021b). Development and validation of a cognitive diagnostic assessment with ordered multiple-choice items for Addition of Time. International Journal of Science and Mathematics Education. [Advance Online Publication] http://doi.org/10.1007/s10763-021-10170-5
• Clements, D. H. (2003). Teaching and learning geometry. In J. Kilpatrick, W. G. Martin, & D. Schifter (Eds.), A research companion to principles and standards for school mathematics (pp. 151-178). Reston, VA: National Council of Teachers of Mathematics.
• Davis, A. E. (1996). Instrument development: getting started. Journal of Neuroscience Nursing, 28(3), 204-208.
• Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006–1012.
• Downing, S. M., & Haladyna, T. M. (1997). Test item development: Validity evidence from quality assurance procedures. Applied Measurement in Education, 10(1), 61-82.
• Fisher, W. P. (1992). Reliability, separation, strata statistics. Rasch Measurement Transactions, 6(3), 238.
• Fisher, W. P. (2007). Rating scale instrument quality criteria. Rasch Measurement Transactions, 21(1), 1095.
• Gay, L. R., Mills, G. E., & Airasian, P. W. (2012). Educational research: Competencies for analysis and applications (10th ed.). Saddle River NJ: Merrill.
• Gierl, M. J., Cui, Y., & Zhou, J. (2009). Reliability and attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46(3), 293–313.
• Gierl, M. J., Leighton, J. P., & Hunka, S. M. (2007). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 242–274). New York: Cambridge University Press.
• Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York: Routledge.
• Hassan, S., & Hod, R. (2017). Use of Item Analysis to Improve the Quality of Single Best Answer Multiple Choice Question in Summative Assessment of Undergraduate Medical Students in Malaysia. Education in Medicine Journal, 9(3), 33-43.
• Herrera, S. G., Murry, K. G., & Cabral, R. M. (2012). Assessment accommodations for classroom teachers of culturally and linguistically diverse students. Boston, MA: Pearson Higher Education.
• Linacre, J. M. (2002) What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878. Linacre, J. M. (2012b). A user guide to Winsteps Ministep Rasch model computer programs: Program manual 3.75.0. Retrieved from http://www.winsteps.com/a/winstepsmanual.pdf
• Mansfield, H. M. & Happs, J. C. (1992). Using grade eight students’ existing knowledge to teach about parallel lines. School Science and Mathematics, 92(8), 450-454.
• Moskal, B. M., Leydens, J. A., & Pavelich, M. J. (2002). Validity, reliability and the assessment of engineering education. Journal of Engineering Education, 91(3), 351-354.
• Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 International Results in Mathematics and Science. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/timss2019/international-results/
• Paksu, A. D., & Bayram, G. (2019). The sixth grade students’ identification and drawings of parallel and perpendicular line/line segments. Gazi University Journal of Gazi Educational Faculty, 39(1), 115-145.
• Park, J., & Kim, D. W. (2017). How can students generalize examples? Focusing on the generalizing geometric properties. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 3771-3800.
• Philipp, K. (2018). Diagnostic competences of mathematics teachers with a view to processes and knowledge resources. In T. Leuders T., K. Philipp, & J. Leuders (Eds.), Diagnostic competence of mathematics teachers. Mathematics teacher education (vol 11, pp. 109-127). Cham, Switzedland: Springer.
• Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.
• Quaigrain, K., & Arhin, A. K. (2017). Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation. Cogent Education, 4(1), 1–11.
• Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institute. (Reprinted 1980 by University of Chicago Press).
• Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4(3), 207–230.
• Retnawati, H., Kartowagiran, B., Arlinwibowo, J., & Sulistyaningsih, E. (2017). Why are the mathematics national examination items difficult and what is teachers' strategy to overcome it?. International Journal of Instruction, 10(3), 257-276.
• Roberts, M. R., Alves, C. B., Chu, M. W., Thompson, M., Bahry, L. M., & Gotzmann, A. (2014). Testing expert based versus student based cognitive models for a Grade 3 diagnostic mathematics assessment. Applied Measurement in Education, 27(3), 173–195.
• Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16(1), 1-10.
• Sia, C. J. L., & Lim, C. S. (2018). Cognitive diagnostic assessment: An alternative mode of assessment for learning. In D. R. Thompson, M. Burton, A. Cusi, & D. Wright (Eds.), Classroom assessment in mathematics (pp. 123–137). Cham, Switzedland: Springer
• Stemler, S. E., & Naples, A. (2021). Rasch Measurement v. Item Response Theory: Knowing when to cross the line. Practical Assessment, Research, and Evaluation, 26(1), Article 11. https://doi.org/10.7275/v2gd-4441
• Sullivan, G. M. (2011). A primer on the validity of assessment instruments. Journal of Graduate Medical Education, 3(2), 119-120.
• Syahfitri, J., Firman, H., Redjeki, S., & Srivati, S. (2019). Development and validation of critical thinking disposition test in biology. International Journal of Instruction, 12(4), 381-392
• Tang, W. L., Tsai, J. T., & Huang, C. Y. (2020). Inheritance coding with Gagné-based learning hierarchy approach to developing mathematics skills assessment systems. Applied Sciences, 10(4), 1465–1483.
• Tavakol, M., & Dennick, R. (2013). Psychometric evaluation of a knowledge based examination using Rasch analysis: An illustrative guide: AMEE guide no. 72. Medical teacher, 35(1), 838-848.
• Thompson, N. A. (2010). KR-20. In N. Salkind (Ed.), Encyclopedia of research design (pp. 667–668). Thousand Oaks, CA: Sage.
• Ulusoy, F. (2016). The role of learners’ example spaces in example generation and determination of two parallel and perpendicular line segments. In C. Csíkos, A. Rausch, & J. Szitányi (Eds.), Proceedings of the 40th Conference of the International Group for the Psychology of Mathematics Education (Vol. 4, pp. 299-306). Szeged, Hungary: PME.
• Yao, X., & Manouchehri, A. (2019). Middle school students’ generalizations about properties of geometric transformations in a dynamic geometry environment. The Journal of Mathematical Behavior, 55, Article 100703. https://doi.org/10.1016/j.jmathb.2019.04.002