The Methodological Quality of Experimental STEM Education Articles Published in Scholarly Journals from 2014 to 2020

Year 2022, Volume: 9 Issue: 2, 290 - 318, 26.06.2022

Ramazan Avcu , Seher Avcu

https://doi.org/10.21449/ijate.946743

Cited By: 2

Abstract

Experimental studies have a considerable impact on the educational policies and practices of many countries. In Turkey, policymakers are planning to initiate a STEM education reform in K-12 schools based on experimental studies. However, the methodological flaws in these studies may lead to biased outcomes and may mislead the STEM education community. Despite the importance of methodological quality, to the best of our knowledge, there are no studies that investigate the methodological quality of experimental STEM education articles published in scholarly journals. Therefore, in this study, we conducted a methodological review to examine the methodological quality of experimental STEM education articles published in refereed Turkish journals from 2014 to 2020. During the targeted period, we located 68 articles. We analyzed these articles by developing a coding framework. We found that the selected articles suffer seriously from various methodological flaws. We discuss the findings in light of the literature on methodological quality and suggest ways to improve the rigor of the experimental designs used. Ultimately, we discuss some implications for authors, journals editors, policymakers, and curriculum developers.

Keywords

STEM education, Methodological quality, Experimental research, Scholarly journal articles

References

• Ahern, K.J. (1999). Ten tips for reflexive bracketing. Qualitative Health Research, 9(3), 407–411. https://doi.org/10.1177/104973239900900309
• Akgündüz, D., Aydeniz, M., Çakmakçı, G., Çavaş, B., Çorlu, M.S., Öner, T., & Özdemir, S. (2015). STEM eğitimi Türkiye raporu: Günün modası mı yoksa gereksinim mi? [A report on STEM education in Turkey: A provisional agenda or a necessity?]. Scala Press.
• Aktemur, Ş. (2015). Review of aviation research: A content analysis of articles published in the Collegiate Aviation Review, 2007–2012 [Unpublished master’s thesis]. Florida Institute of Technology.
• Aron, A., Coups, E.J., & Aron, E.N. (2019). Statistics for the behavioral and social sciences: A brief course (6th ed.). Pearson.
• Ary, D., Jacobs, L.C., Sorensen, C.K., & Walker, D. (2014). Introduction to research in education (9th ed.). Wadsworth Cengage Learning.
• Aydın Günbatar, S., & Tabar, V. (2019). Türkiye’de gerçekleştirilen STEM araştırmalarının içerik analizi [Content analysis of Science, Technology, Engineering and Mathematics (STEM) research conducted in Turkey]. Yüzüncü Yıl University Journal of Education, 16(1), 1054–1083. http://dx.doi.org/10.23891/efdyyu.2019.153
• Baydaş, Ö., Küçük, S., Yılmaz, R. M., Aydemir, M., & Göktaş, Y. (2015). Educational technology research trends from 2002 to 2014. Scientometrics, 105, 709–725. https://doi.org/10.1007/s11192-015-1693-4
• Borman, G.D., Slavin, R.E., Cheung, A., Chamberlain, A.M., Madden, N.A., & Chambers, B. (2005). Success for all: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1 22. https://doi.org/10.3102/01623737027001001
• Brown, J. (2012). The current status of STEM education research. Journal of STEM Education: Innovations and Research, 13(5), 7–11.
• Brown, J.R., & Dant, R.P. (2008). On what makes a significant contribution to the retailing literature. Journal of Retailing, 84(2), 131 135. https://doi.org/10.1016/j.jretai.2008.05.002
• Campbell, D.T., & Boruch, R.F. (1975). Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In C. A. Bennett & A. A. Lumsdaine (Eds.), Evaluation and experiment: Some critical issues in assessing social programs (pp. 195–296). Academic Press.
• Çavaş, P., Ayar, A., Bula Turuplu, S., & Gürcan, G. (2020). Türkiye’de STEM eğitimi üzerine yapılan araştırmaların durumu üzerine bir çalışma [A study on the status of STEM education research in Turkey]. Yüzüncü Yıl University Journal of Education, 17(1), 823–854. https://doi.org/10.33711/yyuefd.751853
• Çevik, M. (2018). Impacts of the project based (PBL) science, technology, engineering and mathematics (STEM) education on academic achievement and career interests of vocational high school students. Pegem Journal of Education and Instruction, 8(2), 281–306. http://dx.doi.org/10.14527/pegegog.2018.012
• Cheung, C.K., & Slavin, R.E. (2012). How features of educational technology applications affect student reading outcomes: A meta-analysis. Educational Research Review, 7(3), 198–215. https://doi.org/10.1016/j.edurev.2012.05.002
• Coe, R.J. (2021). Effect size. In J. Arthur, M. Waring, R. Coe, & L. V. Hedges (Eds.), Research methods and methodologies in education (3rd ed., pp. 368–377). Sage.
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
• Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037/0003-066X.49.12.997
• Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge.
• Creswell, J.W. (2015). Educational research: Planning, conducting and evaluating quantitative and qualitative research (5th ed.). Pearson Education.
• Creswell, J.W., & Creswell, J.D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.
• D’Agostino, J. (2005). Measuring learning outcomes: Reliability and validity issues. In G. D. Phye, D. H. Robinson, & J. Levin (Eds.). Empirical methods for evaluating educational interventions (pp. 113–145). Elsevier Academic Press.
• Daşdemir, İ., Cengiz, E., & Aksoy, G. (2018). Türkiye’de FeTeMM (STEM) eğitimi eğilim araştırması [An investigation of research trends in the field of STEM education in Turkey]. Yüzüncü Yıl University Journal of Education, 15(1), 1161–1183. http://dx.doi.org/10.23891/efdyyu.2018.100
• Dochy, F. (2006). A guide for writing scholarly articles or reviews for the Educational Research Review. Retrieved March 15, 2021, from https://www.elsevier.com/__data/promis_misc/edurevReviewPaperWriting.pdf
• Duman, G., Orhon, G., & Gedik, N. (2015). Research trends in mobile assisted language learning from 2000 to 2012. ReCALL, 27(2), 197 216. https://doi.org/10.1017/S0958344014000287
• Elmalı, Ş., & Balkan Kıyıcı, F. (2017). Türkiye’de yayınlanmış FeTeMM eğitimi ile ilgili çalışmaların incelenmesi [Review of STEM studies published in Turkey]. Sakarya University Journal of Education, 7(3), 684–696. https://doi.org/10.19126/suje.322791
• Erdoğan, M., Marcinkowski, T., & Ok, A. (2009). Content analysis of selected features of K–8 environmental education research studies in Turkey, 1997–2007. Environmental Education Research, 15(5), 525–548. https://doi.org/10.1080/13504620903085776
• Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
• Finn, J.D., & Achilles, C.M. (1999). Tennessee’s class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21, 97–109. https://doi.org/10.3102/01623737021002097
• Fraenkel, J.R. (1987). Toward improving research in social studies education. Theory & Research in Social Education, 15(3), 203 222. https://doi.org/10.1080/00933104.1987.10505546
• Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education (8th ed.). McGraw-Hill.
• Frey, B.B. (2018). The SAGE encyclopedia of educational research, measurement, and evaluation. Sage.
• Gall, J.P., Gall, M.D., & Borg, W.R. (2014). Applying educational research: How to read, do, and use research to solve problems in practice (6th ed.). Pearson Education.
• Gall, M., Gall, J., & Borg, R. (2007). Educational research: An introduction (8th ed.). Pearson Education.
• Gravetter, F., Wallnau, L., Forzano, L., & Witnauer, J. (2021). Essentials of statistics for the behavioral sciences (10 ed.). Cengage Learning.
• Hair, J.F., Black, W.C., Babin, B.J., & Anderson, R.E. (2019). Multivariate data analysis (8th ed.). Cengage Learning.
• Hanel, P.H., & Mehler, D.M. (2019). Beyond reporting statistical significance: Identifying informative effect sizes to improve scientific communication. Public Understanding of Science, 28(4), 468–485. https://doi.org/10.1177/0963662519834193
• Hedges, L.V., & Schauer, J. (2018). Randomised trials in education in the USA. Educational Research, 60(3), 265–275. https://doi.org/10.1080/00131881.2018.1493350
• Horton, P.B., McConney, A.A., Woods, A.L., Barry, K., Krout, H.L., & Doyle, B.K. (1993). A content analysis of research published in the Journal of Research in Science Teaching from 1985 through 1989. Journal of Research in Science Teaching, 30(8), 857–869. https://doi.org/10.1002/tea.3660300805
• Horváth, I. (2016). Theory building in experimental design research. In P. Cash, T. Stankovic, & M. Storga (Eds.), Experimental design research: Approaches, perspectives, applications (pp. 209–231). Springer International Publishing.
• Howell, D.C. (2017). Fundamental statistics for the behavioral sciences (9th ed.). Cengage Learning.
• Johnson, R.B., & Christensen, L.B. (2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage.
• Kaya, A., & Ayar, M.C. (2020). Türkiye örnekleminde STEM eğitimi alanında yapılan çalışmaların içerik analizi [Content analysis of STEM education studies in Turkey]. İstanbul Aydın University Journal of Education, 6(2), 275–306.
• Kirk, R.E. (2013). Experimental design: Procedures for the behavioral sciences (4th ed.). Sage Publications.
• LaCosse, J., Ainsworth, S.E., Shepherd, M.A., Ent, M., Klein, K.M., Holland-Carter, L.A., Moss, J.H., Licht, M., & Licht, B. (2017). An active-learning approach to fostering understanding of research methods in large classes. Teaching of Psychology, 44(2), 117–123. https://doi.org/10.1177/0098628317692614
• Li, Y., Wang, K., Xiao, Y., & Froyd, J. E. (2020). Research and trends in STEM education: A systematic review of journal publications. International Journal of STEM Education, 7, 1. https://doi.org/10.1186/s40594-020-00207-6
• Lincoln, Y.S., & Guba, E. (1985). Naturalistic inquiry. Sage.
• Miles, M.B., Huberman, M.A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Sage.
• Mills, G.E., & Gay, L.R. (2016). Educational research: Competencies for analysis and applications (11th ed.). Pearson Education.
• Mills, G.E., & Gay, L.R. (2019). Educational research: Competencies for analysis and applications (12th ed.). Pearson.
• Ministry of National Education. (2016). STEM eğitimi raporu [STEM education report]. Innovation and Educational Technologies General Directorate.
• Ministry of National Education. (2018). Küresel bağlamda STEM yaklaşımları [STEM approaches in a global context]. Innovation and Educational Technologies General Directorate.
• Mizell, S., & Brown, S. (2016). The current status of STEM education research 2013-2015. Journal of STEM Education, 17(4), 52–56.
• Nelson, J.L., & Shaver, J.P. (1985). On research in social education. In W. B. Stanley (Ed.), Review of research in social studies education: 1976-1983 (pp. 401–433). National Council for the Social Studies.
• Orne, M.T. (1981). The why and how of a contribution to the literature: A brief communication. International Journal of Clinical and Experimental Hypnosis, 29(1), 1–4. https://doi.org/10.1080/00207148108409137
• Özcan, H., & Koca, E. (2019). The impact of teaching the subject “pressure” with STEM approach on the academic achievements of the secondary school 7th grade students and their attitudes towards STEM. Education and Science, 44(198), 201–227. http://dx.doi.org/10.15390/EB.2019.7902
• Pagano, R.R. (2013). Understanding statistics in the behavioral sciences (10th ed.). Wadsworth Cengage Learning.
• Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using the SPSS program (6th ed.). Open University Press.
• Plano Clark, V.L., & Creswell, J.W. (2015). Understanding research: A consumer’s guide (2nd ed.). Pearson Education.
• Privitera, G.J. (2019). Essential statistics for the behavioral sciences (2nd ed.). Sage.
• Randolph, J.J., Griffin, A.E., Zeiger, S.R., Falbe, K.N., Freeman, N.A., Taylor, B.E., Westbrook, A.F., Lico, C.C., Cristy, N. S., Sprull, N. M., Holt, C., Smith, K., & McAnespie, H. (2013). A methodological review of the articles published in Georgia Educational Researcher from 2003-2010. Georgia Educational Researcher Online Edition, 10(1), Article 1. https://doi.org/10.20429/ger.2013.100101
• Schulz, K.F., Altman, D.G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Trials, 11(1), Article 32. https://doi.org/10.1136/bmj.c332
• Schweinhart, L.J., Barnes, H.V., & Weikart, D.P. (1993). Significant benefits: The High/Scope Perry Preschool Study through age 27. High/Scope Press.
• Shapiro, G.M. (2008). Sample size. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (Vol. 2, pp. 781–783). Sage.
• Shaver, J.P., & Norton, R.S. (1980). Populations, samples, randomness, and replication in two social studies journals. Theory & Research in Social Education, 8(2), 1–10. https://doi.org/10.1080/00933104.1980.10506078
• Shukla, A. (2017). Literature review: An oblivious yet grounding task of research. Management Insight, 13(1), 7–15. https://doi.org/10.21844/mijia.v13i01.8363
• Slavin, R.E. (2008). Perspectives on evidence-based research in education—What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5–14. https://doi.org/10.3102/0013189X08314117
• Stebbins, L.F. (2006). Student guide to research in the digital age: How to locate and evaluate information sources. Libraries Unlimited.
• Sung, Y.T., Lee, H.Y., Yang, J.M., & Chang, K.E. (2019). The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool. Educational Research Review, 28, 100279. https://doi.org/10.1016/j.edurev.2019.05.001
• Tabachnick, B.G., & Fidell, L.S. (2019). Using multivariate statistics (7th ed.). Pearson.
• United States Department of Education. (2020). ED delivers historic investment in STEM. https://content.govdelivery.com/accounts/USED/bulletins/2ad85c3
• Valentine, J.C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13(2), 130–149. https://doi.org/10.1037/1082-989X.13.2.130
• Wallen, N.E., & Fraenkel, J.R. (1988). An analysis of social studies research over an eight year period. Theory & Research in Social Education, 16(1), 1 22. https://doi.org/10.1080/00933104.1988.10505553
• Warner, R.M. (2013). Applied statistics: From bivariate through multivariate techniques (2nd ed.). Sage.
• What Works Clearinghouse. (2020a). Standards handbook, version 4.1. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf
• What Works Clearinghouse. (2020b). Procedures handbook, version 4.1. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf