The general attitudes towards artificial intelligence (GAAIS): A meta-analytic reliability generalization study

Year 2024, Volume: 11 Issue: 2, 303 - 319, 20.06.2024

Melek Gülşah Şahin , Yıldız Yıldırım

https://doi.org/10.21449/ijate.1369023

Abstract

This study aims to generalize the reliability of the GAAIS, which is known to perform valid and reliable measurements, is frequently used in the literature, aims to measure one of today's popular topics, and is one of the first examples developed in the field. Within the meta-analytic reliability generalization study, moderator analyses were also conducted on some categorical and continuous variables. Cronbach's α values for the overall scale and the positive and negative subscales, and McDonald's ω coefficients for positive and negative subscales were generalized. Google Scholar, WOS, Taylor & Francis, Science Direct, and EBSCO databases were searched to obtain primary studies. As a result of the screening, 132 studies were found, and these studies were reviewed according to the inclusion criteria. Reliability coefficients obtained from 19 studies that met the criteria were included in the meta-analysis. While meta-analytic reliability generalization was performed according to the random effects model, moderator analyses were performed according to the mixed effect model based on both categorical variables and continuous variables. As a result of the research pooled, Cronbach's α was 0.881, 0.828, and 0.863 for total, the negative, and positive subscales respectively. Also, McDonald's ω was 0.873 and 0.923 for negative and positive subscales respectively. It was found that there were no significant differences between the reliability coefficients for all categorical variables. On the other hand, all continuous moderator variables (mean age, standard deviation age, and rate of female) had a significant effect.

Keywords

General attitudes towards artificial intelligence scale, Reliability generalization, Cronbach’s α, Mc Donald’s ω, Moderator analysis

References

• Alcocer‐Bruno, C., Ferrer‐Cascales, R., Rubio‐Aparicio, M., & Ruiz‐Robledillo, N. (2020). The medical outcome study‐HIV health survey: A systematic review and reliability generalization meta‐analysis. Research in Nursing & Health, 43(6), 610-620. https://doi.org/10.1002/nur.22070
• Arslan, K. (2020). Eğitimde yapay zekâ ve uygulamaları [Artificial intelligence and applications in education]. The Western Anatolia Journal of Educational Sciences, 11(1), 71-88. https://dergipark.org.tr/tr/pub/baebd/issue/55426/690058
• Aslan, Ö.S., Gocen, S., & Sen, S. (2022). Reliability generalization meta-analysis of mathematics anxiety scale for primary school students. Journal of Measurement and Evaluation in Education and Psychology, 13(2), 117 133. https://doi.org/10.21031/epod.1119308
• Begg, C.B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088. https://doi.org/10.2307/2533446
• *Bellaiche, L., Shahi, R., Turpin, M.H., Ragnhildstveit, A., Sprockett, S., Barr, N., ... & Seli, P. (2023). Humans versus AI: Whether and why we prefer human-created compared to AI-created artwork. Cognitive Research: Principles and Implications, 8(1), 1-22. https://doi.org/10.1186/s41235-023-00499-6
• Beretvas, S.N., Meyers, J.L., & Leite, W.L. (2002). A reliability generalization study of the Marlowe Crowne Social Desirability Scale. Educational and Psychological Measurement, 62(4), 570-589. https://doi.org/10.1177/0013164402062004003
• Beretvas, S.N., Suizzo, M.A., Durham, J.A., & Yarnell, L.M. (2008). A reliability generalization study of scores on Rotter's and Nowicki-Strickland's locus of control scales. Educational and Psychological Measurement, 68(1), 97 119. https://doi.org/10.1177/0013164407301529
• *Bergdahl, J., Latikka, R., Celuch, M., Savolainen, I., Mantere, E.S., Savela, N., & Oksanen, A. (2023). Self-determination and attitudes toward artificial intelligence: Cross-national and longitudinal perspectives. Telematics and Informatics, 82, 102013. https://doi.org/10.1016/j.tele.2023.102013
• Borenstein, M., Hedges, L.V., Higgins, J.P., & Rothstein, H.R. (2009). Introduction to meta-analysis. John Wiley & Sons.
• Breazeal, C. (2004). Designing sociable robots. MIT.
• Card, N. (2012). Applied meta-analysis for social science research. Guilford.
• *Carolus, A., Koch, M., Straka, S., Latoschik, M.E., & Wienrich, C. (2023). MAILS-Meta AI Literacy Scale: Development and testing of an AI Literacy Questionnaire based on well-founded competency models and psychological change-and meta-competencies. arXiv preprint. https://doi.org/10.48550/arXiv.2302.09319
• Caruso, J.C., & Edwards, S. (2001). Reliability generalization of the Junior Eysenck Personality Questionnaire. Personality and Individual Differences, 31, 173 184. https://doi.org/10.1016/S0191-8869(00)00126-4
• Cochran, W.G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129. https://doi.org/10.2307/3001666
• *Cruz, J.P., Sembekova, A., Omirzakova, D., Bolla, S.R., & Balay-odao, E.M. (2023). General attitudes towards and readiness for medical artificial intelligence among medical and health sciences students in Kazakhstan. https://doi.org/10.2196/preprints.49536.
• *Darda, K., Carre, M., & Cross, E. (2023). Value attributed to text-based archives generated by artificial intelligence. Royal Society Open Science, 10: 220915. https://doi.org/10.1098/rsos.220915
• DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled clinical trials, 7(3), 177 188. https://www.biostat.jhsph.edu/~fdominic/teaching/bio656/references/sdarticle.pdf
• Eser, M.T., & Dogan, N. (2023). Life Satisfaction Scale: A meta-analytic reliability generalization study in Turkey sample. Turkish Psychological Counseling and Guidance Journal, 13(69), 224-239. https://doi.org/10.17066/tpdrd.1223320mn
• *Gabbiadini, A., Dimitri, O., Cristina, B., & Anna, M. (2023). Does ChatGPT pose a threat to human identity. SSRN, 4377900. https://doi.org/10.2139/ssrn.4377900
• *Gozzo, M., Woldendorp, M.K., & De Rooij, A. (2021, December). Creative collaboration with the “brain” of a search engine: Effects on cognitive stimulation and evaluation apprehension. In International Conference on ArtsIT, Interactivity and Game Creation (pp. 209-223). Springer International Publishing.
• Grassini, S. (2023). Development and validation of the AI attitude scale (AIAS-4): A brief measure of general attitude toward artificial intelligence. Frontiers in Psychology, 14: 1191628. https://doi.org/10.3389/fpsyg.2023.1191628
• *Hadlington, L., Binder, J., Gardner, S., Karanika-Murray, M., & Knight, S. (2023). The use of artificial intelligence in a military context: Development of the Attitudes Toward AI in Defense (AAID) Scale. Frontiers in Psychology, 14, 1164810. https://doi.org/ 10.3389/fpsyg.2023.1164810
• Hedges, L.V., & Pigott, T.D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426 445. https://doi.org/10.1037/1082 989x.9.4.426
• *Heim, S., & Chan-Olmsted, S. (2023). Consumer trust in AI–human news collaborative continuum: preferences and influencing factors by news production phases. Journalism and Media, 4(3), 946-965. https://doi.org/10.3390/journalmedia4030061
• Henson, R.K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability generalization” studies. Measurement and Evaluation in Counseling and Development, 35(2), 113-127. https://doi.org/10.1080/07481756.2002.12069054
• Hess, T.J., McNab, A.L., & Basoglu, K.S. (2014). Reliability generalization of perceived ease of use, perceived usefulness, and behavioral intentions. MIS Quarterly, 38, 1-28. https://doi.org/10.25300/MISQ/2014/38.1.01
• Higgins, J.P.T., & Thompson, S.G. (2002), Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558. https://doi.org/10.1002/sim.1186
• Hopcan, S., Turkmen, G., & Polat, E. (2023). Exploring the artificial intelligence anxiety and machine learning attitudes of teacher candidates. Education and Information Technologies, 1-21. https://doi.org/10.1007/s10639-023-12086-9
• Huang, S.P. (2018). Effects of using artificial intelligence teaching system for environmental education on environmental knowledge and attitude. Eurasia Journal of Mathematics, Science and Technology Education, 14(7), 3277 3284. https://doi.org/10.29333/ejmste/91248
• Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016, October). Artificial intelligence and computer science in education: From kindergarten to university. In 2016 Institute of electrical and electronics engineers - Frontiers in education conference (IEEE-FIE) (pp. 1-9). IEEE. https://doi.org/10.1109/FIE.2016.7757570
• Kaplan, A., & Haenlein, M. (2019). Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Business Horizons, 62(1), 15-25. https://doi.org/10.1016/j.bushor.2018.08.004
• *Kaya, F., Aydin, F., Schepman, A., Rodway, P., Yetisensoy, O., & Demir Kaya, M. (2022). The roles of personality traits, AI anxiety, and demographic factors in attitudes toward artificial intelligence. International Journal of Human–Computer Interaction, 1-18. https://doi.org/10.1080/10447318.2022.2151730
• Kieslich, K., Lünich, M., & Marcinkowski, F. (2021). The threats of artificial intelligence scale (TAI) development, measurement and test over three application domains. International Journal of Social Robotics, 13, 1563-1577. https://doi.org/10.1007/s12369-020-00734
• Kristof, W. (1974). Estimation of reliability and true score variance from a split of a test into three arbitrary parts. Psychometrika, 39, 491-499. https://doi.org/10.1007/BF02291670
• *Kwak, Y., Ahn, J.W., & Seo, Y.H. (2022). Influence of AI ethics awareness, attitude, anxiety, and self-efficacy on nursing students’ behavioral intentions. BMC Nursing, 21(1), 1-8. https://doi.org/10.1186/s12912-022-01048-0
• *Kwak, Y., Seo, Y.H., & Ahn, J.W. (2022). Nursing students' intent to use AI-based healthcare technology: Path analysis using the unified theory of acceptance and use of technology. Nurse Education Today, 119: 105541. https://doi.org/10.1016/j.nedt.2022.105541
• McCarthy, J. (2004). What is artificial intelligence?. http://www.formal.stanford.edu/jmc/whatisai/
• *Mohamed, H.A., Awad, S.G., Eldiasty, N.E.M.M, & ELsabahy, H.E. (2023). Effect of the artificial intelligence enhancement program on head nurses' managerial competencies and flourishing at work. Egyptian Journal of Health Care, 14(1), 624 645. https://doi.org/10.21608/EJHC.2023.287188
• Nica, E., Sabie, O.M., Mascu, S., & Luţan, A.G. (2022). Artificial intelligence decision-making in shopping patterns: Consumer values, cognition, and attitudes. Economics, Management and Financial Markets, 17(1), 31 43. https://doi.org/10.22381/emfm17120222.
• *Nguyen, E. (2023). Trust and algorithmic decision making. UC Santa Barbara, 3(2022), 1-15. https://escholarship.org/content/qt5z86t0dx/qt5z86t0dx.pdf
• Novick, M.R., & Lewis, C.L. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13. https://doi.org/10.1007/BF02289400
• Nunnally, J.C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
• Osburn, H.G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5(3), 343–355. https://doi.org/10.1037/1082-989X.5.3.343
• Ozdemir, V., Yildirim, Y., & Tan, S. (2020). A meta-analytic reliability generalization study of the Oxford Happiness Scale in Turkish sample. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 374-404. https://doi.org/10.21031/epod.766266
• Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron I., Hoffmann, T.C., Mulrow, C.D., …, & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372: 71. https://doi.org/10.1136/bmj.n71
• Persson, A., Laaksoharju, M., & Koga, H. (2021). We mostly think alike: Individual differences in attitude towards AI in Sweden and Japan. The Review of Socionetwork Strategies, 15(1), 123-142. https://doi.org/10.1007/s12626-021-00071-y
• Pinto dos Santos, D., Giese, D., Brodehl, S., Chon, S.H., Staab, W., Kleinert, R., ... & Baeßler, B. (2019). Medical students' attitude towards artificial intelligence: A multicentre survey. European radiology, 29, 1640-1646. https://doi.org/10.1007/s00330-018-5601-1
• Polesie, S., Gillstedt, M., Kittler, H., Lallas, A., Tschandl, P., Zalaudek, I., & Paoli, J. (2020). Attitudes towards artificial intelligence within dermatology: An international online survey. British Journal of Dermatology, 183(1), 159 161. https://doi.org/ 10.1111/bjd.18875
• Rothstein, H.R., Sutton, A.J., & Borenstein, M. (2005). Publication bias in meta‐analysis: Prevention, assessment and adjustments. John Wiley & Sons.
• Rosenthal, R. (1979). The ‘‘file drawer problem’’ and tolerance for null results. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638
• *Saddique, F., Usman, M., Nawaz, M., & Mushtaq, N. (2020). Entrepreneurial orientation and human resource management: The mediating role of Artificial Intelligence. Elementary Education Online, 19(4), 4969-4978. https://doi.org/10.17051/ilkonline.2021.05.777
• Sánchez-Meca J, Marín-Martínez F, López-López JA, … & López-Nicolás, P. (2021). Improving the reporting quality of reliability generalization meta-analyses: The REGEMA checklist. Research Synthesis Methods, 12, 516 536. https://doi.org/10.1002/jrsm.1487
• *Schepman, A., & Rodway, P. (2020). Initial validation of the general attitudes towards Artificial Intelligence Scale. Computers in Human Behavior Reports, 1, 100014. https://doi.org/10.1016/j.chbr.2020.100014
• *Schepman, A., & Rodway, P. (2022). The General Attitudes towards Artificial Intelligence Scale (GAAIS): Confirmatory validation and associations with personality, corporate distrust, and general trust. International Journal of Human–Computer Interaction, 39(13), 2724-2741. https://doi.org/10.1080/10447318.2022.2085400
• *Seo, Y.H., & Ahn, J.W. (2022). The validity and reliability of the Korean version of the General Attitudes towards Artificial Intelligence Scale for nursing students. The Journal of Korean Academic Society of Nursing Education, 28(4), 357 367. https://doi.org/10.5977/jkasne.2022.28.4.357
• Sindermann, C., Sha, P., Zhou, M., Wernicke, J., Schmitt, H.S., Li, M., ... & Montag, C. (2021). Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language. KI-Künstliche Intelligenz, 35, 109-118. https://doi.org/10.1007/s13218-020-00689-0
• Thompson, B., & Cook, C. (2002). Stability of the reliability of libqual+™ scores a reliability generalization meta-analysis study. Educational and Psychological Measurement, 62(4), 735-743. https://doi.org/10.1177/0013164402062004013
• Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174 195. https://doi.org/10.1177/00131640021970448
• Turkle, S., Breazeal, C., Dasté, O., & Scassellati, B. (2006). Encounters with kismet and cog: Children respond to relational artifacts. Digital media: Transformations in human communication, 120. http://web.mit.edu/people/sturkle/encounterswithkismet.pdf
• Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58(1), 6–20. https://doi.org/10.1177/0013164498058001002
• Vacha-Haase, T., Kogan, L.R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60(4), 509 522. https://doi.org/10.1177/00131640021970682
• Vassar, M.A. (2008). Note on the score reliability for the Satisfaction with Life Scale: An RG study. Soc Indic Res, 86, 47–57. https://doi.org/10.1007/s11205-007-9113-7
• Waliszewski, K., & Warchlewska, A. (2020). Attitudes towards artificial intelligence in the area of personal financial planning: A case study of selected countries. Entrepreneurship and Sustainability Issues, 8(2), 399-420. https://doi.org/10.9770/jesi.2020.8.2(24)
• Wallace, K.A., & Wheeler, A.J. (2002). Reliability generalization of the life satisfaction index. Educational and Psychological Measurement, 62(4), 674 684. https://doi.org/10.1177/0013164402062004009
• *Wang, H., Sun, Q., Gu, L., Lai, K., & He, L. (2022). Diversity in people's reluctance to use medical artificial intelligence: Identifying subgroups through latent profile analysis. Frontiers in Artificial Intelligence, 5: 1006173. https://doi.org/10.3389/frai.2022.1006173
• Warrens, M.J. (2014). On Cronbach’s alpha as the mean of all possible-split alphas. Advances in Statistics. 742863. https://doi.org/10.1155/2014/742863
• Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60(2), 201-223. https://doi.org/10.1177/00131640021970466
• Youngstrom, E.A., & Green, K.W. (2003). Reliability generalization of self-report of emotions when using the Differential Emotions Scale. Educational and Psychological Measurement, 63(2), 279-295. https://doi.org/10.1177/00131644032532
• Yoruk, S., & Sen, S. (2023). A reliability generalization meta-analysis of the creative achievement questionnaire. Creativity Research Journal, 35(4), 714 729. https://doi.org/10.1080/10400419.2022.2148073
• Yuzbasioglu, E. (2021). Attitudes and perceptions of dental students towards artificial intelligence. Journal of Dental Education, 85(1), 60 68. https://doi.org/10.1002/jdd.12385