The Replication Crisis and Psychotherapy Efficacy Studies

Year 2022, Volume: 42 Issue: 1, 1 - 41, 29.04.2022

Kutlu Kağan Türkarslan

https://doi.org/10.26650/SP2021-959170

Abstract

In the last ten years, the replication crisis, which has manifested itself and shaken the field of psychology profoundly, has led researchers in every subfield of psychology to reconsider research findings and general assumptions. Though late to this trend, clinical psychology researchers have started to participate in these discussions. Psychotherapy efficacy studies, carried out over many years, constitute a major experimental area of clinical psychology. Psychotherapy efficacy studies are conducted to evaluate the effects of psychological treatments on various mental disorders. The findings of the replication crisis indicate that randomized controlled trials, which are frequently used to examine the efficacy of psychotherapies and meta-analyses that help evaluate the results of randomized controlled trials, have several methodological problems and biases. Randomized controlled trials have problems and biases such as low sample size, allegiance bias, selective outcome reporting, strictness of inclusion or exclusion criteria, using wait-list as a control group, problems of randomization and blinding, exclusion of drop-out subjects from the analyses, and ignoring psychotherapists effects. The main methodological issues of meta-analyses are related to publication bias and the inclusion of low-quality studies in meta-analyses. Research over the years has demonstrated that psychotherapies are effective tools for treating various mental disorders. However, there are two critical points to note. First, the studies showing the efficacy of psychotherapies barely meet current scientific methodological standards. Second, the real-life effectiveness of psychotherapies may be lower than those found in clinical trials. This review article aims to increase awareness regarding replication crisis within the field of clinical psychology and strengthen the methodological approach of future studies by addressing the methodological problems of and suggesting solutions for psychotherapy efficacy studies/meta-analyses

Keywords

Psychotherapy, replication crisis, randomized controlled trail, meta-analysis, bias, dodo bird verdict

Tekrarlama Krizi ve Psikoterapi Etkililik Çalışmaları

Abstract

Son 10 yıldır psikoloji alanında kendini gösteren ve alanı derinden sarsan tekrarlama krizi, psikolojinin her alt alanını, araştırma bulgularını ve genel kabullerini yeniden değerlendirmesi için harekete geçirmiştir. Klinik psikoloji alanının geç de olsa tekrarlama krizi tartışmalarına katılmaya başladığı görülmektedir. Uzun yıllardır gerçekleştirilen psikoterapi etkililik (efficacy) çalışmaları, klinik psikoloji içindeki başlıca deneysel alanlardan birini oluşturmaktadır. Psikoterapi etkililik çalışmaları çeşitli psikolojik tedavilerin psikolojik rahatsızlıklar üzerindeki etkilerini değerlendirmek için gerçekleştirilmektedir. Tekrarlama krizi bağlamında ortaya çıkan bulgular, psikoterapilerin etkililiklerini değerlendirmek için sıkça kullanılan seçkisiz kontrollü çalışmaların ve bu çalışmaların sonuçlarının toplu olarak değerlendirilmesine yardımcı olan meta-analizlerin pek çok yöntemsel problem ve yanlılık içerdiğini ortaya çıkarmıştır. Seçkisiz kontrollü çalışmalar düşük örneklem sayısı, bağlılık yanlılığı, seçici sonuç raporlama, örnekleme dahil etme ya da dışarıda bırakma kriterlerinin katılığı, kontrol grubu olarak bekleme listesi kullanımı, seçkisizleştirme ve körleştirme sorunları, çalışmayı bırakan katılımcıların analizlere dahil edilmemesi ve psikoterapist etkilerinin ihmal edilmesi gibi problemler ve yanlılıklar içermektedir. Meta-analizler ile ilişkili başlıca sorunlar ise yayın yanlılığı ve düşük kaliteli çalışmaların meta-analizlere dahil edilmesidir. Tüm bu durumlar psikoterapilerin olumlu etkilerini olduğundan daha yüksek gösterirken, edinilen bulguların sağlıklı bir şekilde değerlendirilmesini engellemektedir. Uzun yıllardır gerçekleştirilen çalışmalar, psikoterapilerin çeşitli psikolojik rahatsızlıkların tedavisinde tercih edilebilecek etkili araçlar olduğunu göstermektedir. Ancak dikkat edilmesi gereken iki önemli husus vardır. Bunlardan ilki psikoterapilerin etkililiğini gösteren çalışmaların yöntemsel anlamda güncel bilimsel standartları yakalamakta zorlanmasıdır. İkinci önemli husus ise psikoterapilerin gerçek etkilerinin çalışmalarda bulunan etkilerden daha az olabileceğidir. Bu derleme makalesinin amacı, psikoterapi etkililik çalışmalarını ve meta-analizleri etkileyen yöntemsel problemlere ve onların çözümlerine değinerek klinik psikoloji alanında bu konuda ortaya çıkan farkındalığın arttırılmasına ve gerçekleştirilecek çalışmaların yöntemsel olarak güçlendirilmesine yardımcı olmaktır.

Keywords

Psikoterapi, tekrarlama krizi, seçkisiz kontrollü çalışma, meta-analiz, yanlılık, dodo kuşu yargısı

References

• Abraham, W. T., ve Russell, D. W. (2008). Statistical power analysis in psychological research. Social and Personality Psychology Compass, 2(1), 283-301. https://doi.org/10.1111/j.1751-9004.2007.00052.x google scholar
• Addis, M. E. ve Krasnow, A. D. (2000). A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. Journal of Consulting and Clinical Psychology, 68(2), 331-339. https://doi.org/10.1037/0022-006X.68.2.331 google scholar
• APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. The American Psychologist, 61(4), 271-285. https://doi.org/10.1037/0003-066X.61.4.271 google scholar
• Arnberg, A. ve Öst, L.-G. (2014). CBT for children with depressive symptoms: A meta-analysis. Cognitive Behaviour Therapy, 43(4), 275-288. https://doi.org/10.1080/16506073.2014.947316 google scholar
• Baardseth, T. P., Goldberg, S. B., Pace, B. T., Wislocki, A. P., Frost, N. D., Siddiqui, J. R., Lindemann, A. M., Kivlighan, D. M. III, Laska, K. M., Del Re, A. C., Minami, T., & Wampold, B. E. (2013). Cognitive-behavioral therapy versus other therapies: Redux. Clinical Psychology Review, 33(3), 395-405. https://doi.org/10.1016Zj.cpr.2013.01.004 google scholar
• Berkeljon, A., & Baldwin, S. A. (2009). An introduction to meta-analysis for psychotherapy outcome research. Psychotherapy Research, 19(4-5), 511-518. https://doi.org/10.1080/10503300802621172 google scholar
• Bradley, H. A., Rucklidge, J. J. ve Mulder, R. T. (2017). A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatrica Scandinavica, 135(1), 65-77. https://doi.org/10.1111/acps.12647 google scholar
• Budd, R. ve Hughes, I. (2009). The Dodo Bird Verdict—controversial, inevitable and important: A commentary on 30 years of meta-analyses. Clinical Psychology & Psychotherapy, 16(6), 510-522. https://doi.org/10.1002/cpp.648 google scholar
• Button, K. S., loannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. ve Munafö, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3475 google scholar
• Carroll, H. A., Toumpakari, Z., Johnson, L. ve Betts, J. A. (2017). The perceived feasibility of methods to reduce publication bias. PloS One, 12(10), e0186472. https://doi.org/10.1371/journal. pone.0186472 google scholar
• Chambless, D. L. ve Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7-18. https://doi.org/10.1037/0022-006X.66.1.7 google scholar
• Christley, R. M. (2010). Power and Error: Increased Risk of False Positive Results in Underpowered Studies. The Open Epidemiology Journal, 3(1), 16-19. http://dx.doi. org/10.2174/1874297101003010016 google scholar
• Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi. org/10.1037//0033-2909.112.1.155 google scholar
• Cooper, A. A. ve Conklin, L. R. (2015). Drop out from individual psychotherapy for major depression: A meta-analysis of randomized clinical trials. Clinical Psychology Review, 40, 57-65. https://doi. org/10.1016/j.cpr.2015.05.001 google scholar
• Coyne, J. C. ve Kok, R. N. (2014). Salvaging psychotherapy research: A manifesto. Journal of Evidence-Based Psychotherapies, 14(2), 105-124. google scholar
• Craigie, M. A. ve Nathan, P. (2009). A nonrandomized effectiveness comparison of broad-spectrum group CBT to individual CBT for depressed outpatients in a community mental health setting. Behavior therapy, 40(3), 302-314. https://doi.org/10.1016/j.beth.2008.08.002 google scholar
• Crameri, A., von Wyl, A., Koemeda, M., Schulthess, P. ve Tschuschke, V. (2015). Sensitivity analysis in multiple imputation in effectiveness studies of psychotherapy. Frontiers in Psychology, 6, 1042. https://doi.org/10.3389/fpsyg.2015.01042 google scholar
• Cristea, I. A., Stefan, S., Karyotaki, E., David, D., Hollon, S. D. ve Cuijpers, P. (2017). The effects of cognitive behavioral therapy are not systematically falling: A revision of Johnsen and Friborg (2015). Psychological Bulletin, 143(3), 326-340. https://doi.org/10.1037/bul0000062 google scholar
• Cuijpers, P. ve Cristea, I. A. (2016). How to prove that your therapy is effective, even when it is not: A guideline. Epidemiology and Psychiatric Sciences, 25(5), 428-435. https://doi.org/10.1017/ S2045796015000864 google scholar
• Cuijpers, P., Cristea, I. A., Karyotaki, E., Reijnders, M. ve Huibers, M. J. H. (2016). How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry, 15(3), 245-258. https://doi.org/10.1002/wps.20346 google scholar
• Cuijpers, P., Karyotaki, E., Reijnders, M. ve Ebert, D. D. (2018). Was Eysenck right after all? A reassessment of the effects of psychotherapy for adult depression. Epidemiology and Psychiatric Sciences, 28(1), 21-30. https://doi.org/10.1017/S2045796018000057 google scholar
• Cuijpers, P., Karyotaki, E., Reijnders, M. ve Ebert, D. D. (2019). Is psychotherapy effective? Pretending everything is fine will not help the field forward. Epidemiology and Psychiatric Sciences, 28(3), 356-357. https://doi.org/10.1017/S204579601800080X google scholar
• Cuijpers, P., Reijnders, M. ve Huibers, M. J. H. (2019). The Role of Common Factors in Psychotherapy Outcomes. Annual Review of Clinical Psychology, 15(1), 207-231. https://doi.org/10.1146/annurev-clinpsy-050718-095424 google scholar
• Cuijpers, P., van Straten, A., Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010). The effects of psychotherapy for adult depression are overestimated: A meta-analysis of study quality and effect size. PsychologicalMedicine, 40(2), 211-223. https://doi.org/10.1017/S0033291709006114 google scholar
• de Felice, G., Giuliani, A., Halfon, S., Andreassi, S., Paoloni, G. ve Orsucci, F. F. (2019). The misleading Dodo Bird verdict. How much of the outcome variance is explained by common and specific factors? New Ideas in Psychology, 54, 50-55. https://doi.org/10.1016/j.newideapsych.2019.01.006 google scholar
• Del Re, A. C., Flückiger, C., Horvath, A. O., Symonds, D. ve Wampold, B. E. (2012). Therapist effects in the therapeutic alliance-outcome relationship: A restricted-maximum likelihood meta-analysis. Clinical Psychology Review, 32(7), 642-649. https://doi.org/10.1016/j.cpr.2012.07.002 google scholar
• Donnon, T. (2012). Experimental or RCT research designs: A crisis of nomenclature in medical education. Canadian Medical Education Journal, 3(2), 82-84. google scholar
• Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P. ve Turner, E. H. (2015). Does Publication Bias Inflate the Apparent Efficacy of Psychological Treatment for Major Depressive Disorder? A Systematic Review and Meta-Analysis of US National Institutes of Health-Funded Trials. PloS One, 10(9), e0137864. https://doi.org/10.1371/journal.pone.0137864 google scholar
• Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F. ve Munafö, M. R. (2017). Low statistical power google scholar
• in biomedical science: A review of three human research domains. Royal Society Open Science, 4(2), 160254. https://doi.org/10.1098/rsos.160254 google scholar
• Easterbrook, P. J., Gopalan, R., Berlin, J. A. ve Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337(8746), 867-872. https://doi.org/10.1016/0140-6736(91)90201-Y google scholar
• Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16(5), 319-324. https://doi.org/10.1037/h0063633 google scholar
• Eysenck, H. J. (1964). The outcome problem in psychotherapy: A reply. Psychotherapy: Theory, Research & Practice, 1(3), 97-100. https://doi.org/10.1037/h0088591 google scholar
• Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891-904. https://doi.org/10.1007/s11192-011-0494-7 google scholar
• Faul, F., Erdfelder, E., Lang, A.-G., ve Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191. https://doi.org/10.3758/BF03193146 google scholar
• Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532-538. https://doi.org/10.1037/a0015808 google scholar
• Fensterheim, H. ve Raw, S. D. (1996). Psychotherapy Research Is Not Psychotherapy Practice. Clinical Psychology: Science and Practice, 3(2), 168-171. https://doi.org/10.1111/j.1468-2850.1996. tb00067.x google scholar
• Fernandez, E., Salem, D., Swift, J. K. ve Ramtahal, N. (2015). Meta-analysis of dropout from cognitive behavioral therapy: Magnitude, timing, and moderators. Journal of Consulting and Clinical Psychology, 83(6), 1108-1122. https://doi.org/10.1037/ccp0000044 google scholar
• Flint, J., Cuijpers, P., Horder, J., Koole, S. L. ve Munafö, M. R. (2015). Is there an excess of significant findings in published studies of psychotherapy for depression? Psychological Medicine, 45(2), 439446. https://doi.org/10.1017/S0033291714001421 google scholar
• Fonagy, P., Luyten, P. ve Bateman, A. (2017). Treating Borderline Personality Disorder With Psychotherapy: Where Do We Go From Here? JAMA Psychiatry, 74(4), 316-317. https://doi. org/10.1001/jamapsychiatry.2016.4302 google scholar
• Freire, E. S. (2006). Randomized Controlled Clinical Trial in Psychotherapy Research: An Epistemological Controversy. Journal of Humanistic Psychology, 46(3), 323-335. https://doi. org/10.1177/0022167806286276 google scholar
• Friborg, O. ve Johnsen, T. J. (2017). The effect of cognitive-behavioral therapy as an antidepressive treatment is falling: Reply to Ljotsson et al. (2017) and Cristea et al. (2017). Psychological Bulletin, 143(3), 341-345. https://doi.org/10.1037/bul0000090 google scholar
• Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., Chen, P., Hunot, V., & Churchill, R. (2014). Waiting list may be a nocebo condition in psychotherapy trials: A contribution from network meta-analysis. Acta Psychiatrica Scandinavica, 130(3), 181192. https://doi.org/10.1111/acps.12275 google scholar
• Gelman, A. ve Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641-651. https://doi. org/10.1177/1745691614551642 google scholar
• Gelo, O. C. G., Lagetto, G., Dinoi, C., Belfiore, E., Lombi, E., Blasi, S., Aria, M., & Ciavolino, E. (2020). Which methodological practice(s) for psychotherapy science? A systematic review and a proposal. Integrative Psychological & Behavioral Science, 54(1), 215-248. https://doi.org/10.1007/ s12124-019-09494-3 google scholar
• Glass, G. V. (1982). Meta-analysis: An Approach to the Synthesis of Research Results. Journal of research in science teaching, 19(2), 93-112. google scholar
• Goldfried, M. R. (2020). The field of psychotherapy: Over 100 years old and still an infant science. Clinical Psychology in Europe, 2(1), e2753. https://doi.org/10.32872/cpe.v2i1.2753 google scholar
• Goldfried, M. R. ve Wolfe, B. E. (1998). Toward a more clinically valid approach to therapy research. Journal of Consulting and Clinical Psychology, 66(1), 143-150. https://doi.org/10.1037/0022-006X.66.1.143 google scholar
• Gonzalez-Blanch, C. ve Carral-Fernandez, L. (2017). Cage up Dodo, please! The tale of all psychotherapies being equally effective. Papeles del Psicologo, 38(2), 94-106. https://doi. org/10.23923/pap.psicol2017.2828 google scholar
• Hansen, W. B. ve Collins, L. M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph, 142, 184-195. google scholar
• Hengartner, M. P. (2018). Raising awareness for the replication crisis in clinical psychology by focusing on inconsistencies in psychotherapy research: How much can we rely on published findings from efficacy trials? Frontiers in Psychology, 9, Article 256. https://doi.org/10.3389/fpsyg.2018.00256 google scholar
• Hoertel, N., de Maricourt, P., Katz, J., Doukhan, R., Lavaud, P., Peyre, H. ve Limosin, F. (2014). Are participants in pharmacological and psychotherapy treatment trials for social anxiety disorder representative of patients in real-life settings? Journal of Clinical Psychopharmacology, 34(6), 697-703. https://doi.org/10.1097/JCP.0000000000000204 google scholar
• Hoertel, N., Lopez, S., Wang, S., Gonzalez-Pinto, A., Limosin, F. ve Blanco, C. (2015). Generalizability of pharmacological and psychotherapy clinical trial results for borderline personality disorder to community samples. Personality Disorders: Theory, Research, and Treatment, 6(1), 81-87. https:// doi.org/10.1037/per0000091 google scholar
• Hopewell, S., Clarke, M. ve Mallett, S. (2005). Grey Literature and Systematic Reviews. H. R. Rothstein & A. J. Sutton (Ed.), Publication Bias in Meta-Analysis içinde (s. 49-72). John Wiley & Sons, Ltd. https://doi.org/10.1002/0470870168.ch4 google scholar
• Hotopf, M. (2002). The pragmatic randomised controlled trial. Advances in Psychiatric Treatment, 8(5), 326-333. doi:10.1192/apt.8.5.326 google scholar
• Hsu, L. M. (1992). Random sampling, randomization, and equivalence of contrasted groups in psychotherapy outcome research. A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research içinde (s. 91-105). American Psychological Association. https://doi.org/10.1037/10109-018 google scholar
• Johns, R. G., Barkham, M., Kellett, S. ve Saxon, D. (2019). A systematic review of therapist effects: A critical narrative update and refinement to Baldwin and Imel’s (2013) review. Clinical Psychology Review, 67, 78-93. https://doi.org/10.1016/j.cpr.2018.08.004 google scholar
• Johnsen, T. J. ve Friborg, O. (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: A meta-analysis. Psychological Bulletin, 141(4), 747-768. https://doi. org/10.1037/bul0000015 google scholar
• Jones, J. L. ve Mehr, S. L. (2007). Foundations and assumptions of the scientist-practitioner model. American Behavioral Scientist, 50(6), 766-771. https://doi.org/10.1177/0002764206296454 google scholar
• Kelley, K. ve Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137-152. https://doi. org/10.1037/a0028086 google scholar
• Kendall, J. (2003). Designing a research project: Randomised controlled trials and their principles. Emergency Medicine Journal: EMJ, 20(2), 164-168. https://dx.doi.org/10.1136%2Femj.20.2.164 google scholar
• Kiai, A. (2019). To protect credibility in science, banish “publish or perish”. Nature Human Behaviour, 3(10), 1017-1018. https://doi.org/10.1038/s41562-019-0741-0 google scholar
• Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R. ve Williamson, P. R. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, c365. https://doi.org/10.1136/bmj.c365 google scholar
• Lambert, M. J. (2013). The Efficacy and Effectiveness of Psychotherapy. M. J. Lambert (Ed.), Bergin and Garfields Handbook of Psychotherapy and Behavior Change (6. Bs) içinde (s. 169-218). John Wiley & Sons, Ltd. google scholar
• Landman, J. T. ve Dawes, R. M. (1982). Psychotherapy outcome: Smith and Glass’ conclusions stand up under scrutiny. American Psychologist, 37(5), 504-516. https://doi.org/10.1037/0003-066X.37.5.504 google scholar
• Laska, K. M., Gurman, A. S. ve Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467-481. https://doi. org/10.1037/a0034332 google scholar
• Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S. ve Steinert, C. (2017). Biases in research: Risk factors for non-replicability in psychotherapy and pharmacotherapy research. Psychological Medicine, 47(6), 1000-1011. https:// doi.org/10.1017/S003329171600324X google scholar
• Leichsenring, F., Abbass, A., Hilsenroth, M. J., Luyten, P., Munder, T., Rabung, S. ve Steinert, C. (2018). “Gold standards,” plurality and monocultures: The need for diversity in psychotherapy. Frontiers in Psychiatry, 9, 159. https://doi.org/10.3389/fpsyt.2018.00159 google scholar
• Leichsenring, F. ve Rabung, S. (2011). Long-term psychodynamic psychotherapy in complex mental disorders: Update of a meta-analysis. The British Journal of Psychiatry, 199(1), 15-22. https://doi. org/10.1192/bjp.bp.110.082776 google scholar
• Leykin, Y. ve DeRubeis, R. J. (2009). Allegiance in Psychotherapy Outcome Research: Separating Association From Bias. Clinical Psychology: Science and Practice, 16(1), 54-65. https://doi. org/10.1111/j.1468-2850.2009.01143.x google scholar
• Lilienfeld, S. O. (2014). The Dodo Bird verdict: Status in 2014. The Behavior Therapist, 37(4), 91-95. google scholar
• Linardon, J., Kothe, E. J. ve Fuller-Tyszkiewicz, M. (2019). Efficacy of psychotherapy for bulimia nervosa and binge-eating disorder on self-esteem improvement: Meta-analysis. European Eating Disorders Review, 27(2), 109-123. https://doi.org/10.1002/erv.2662 google scholar
• Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., Halperin, G., Bishop, M., Berman, J. S., & Schweizer, E. (1999). The researcher’s own therapy allegiances: A “wild card” in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6(1), 95-106. https://doi.Org/10.1093/clipsy.6.1.95 google scholar
• Luborsky, L., Rosenthal, R., Diguer, L., Andrusyna, T. P., Berman, J. S., Levitt, J. T., Seligman, D. A., & Krause, E. D. (2002). The dodo bird verdict is alive and well--mostly. Clinical Psychology: Science and Practice, 9(1), 2-12. https://doi.org/10.1093/clipsy.9.1.2 google scholar
• Luborsky, L., Singer, B. ve Luborsky, L. (1975). Comparative studies of psychotherapies: Is it true that “everyone has won and all must have prizes”? Archives of General Psychiatry, 32(8), 995-1008. https://doi.org/10.1001/archpsyc.1975.01760260059004 google scholar
• Marcus, D. K., O’Connell, D., Norris, A. L. ve Sawaqdeh, A. (2014). Is the Dodo bird endangered in the 21st century? A meta-analysis of treatment comparison studies. Clinical Psychology Review, 34(7), 519-530. https://doi.org/10.1016/j.cpr.2014.08.001 google scholar
• Maxwell, S. E., Lau, M. Y. ve Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487-498. https://doi. org/10.1037/a0039400 google scholar
• Miguel, C., Karyotaki, E., Cuijpers, P. ve Cristea, I. A. (2021). Selective outcome reporting and the effectiveness of psychotherapies for depression. World Psychiatry, 20(3), 444-445. https://dx.doi. org/10.1002%2Fwps.20900 google scholar
• Miller, L. E., ve Stewart, M. E. (2011). The blind leading the blind: Use and misuse of blinding in randomized controlled trials. Contemporary Clinical Trials, 32(2), 240-243. https://doi. org/10.1016/j.cct.2010.11.004 google scholar
• Morrison, K. H., Bradley, R. ve Westen, D. (2003). The external validity of controlled clinical trials of psychotherapy for depression and anxiety: A naturalistic study. Psychology and Psychotherapy: Theory, Research and Practice, 76(2), 109-132. https://doi.org/10.1348/147608303765951168 google scholar
• Mulder, R., Singh, A. B., Hamilton, A., Das, P., Outhred, T., Morris, G., Bassett, D., Baune, B. T., Berk, M., Boyce, P., Lyndon, B., Parker, G., & Malhi, G. S. (2018). The limitations of using randomised controlled trials as a basis for developing treatment guidelines. Evidence-based mental health, 21 (1), 4-6. https://doi.org/10.1136/eb-2017-102701 google scholar
• Munder, T., Brütsch, O., Leonhart, R., Gerger, H. ve Barth, J. (2013). Researcher allegiance in psychotherapy outcome research: An overview of reviews. Clinical Psychology Review, 33(4), 501-511. https://doi.org/10.1016/j.cpr.2013.02.002 google scholar Munder, T., Flückiger, C., Leichsenring, F., Abbass, A. A., Hilsenroth, M. J., Luyten, P., Rabung, S., Steinert, C., & Wampold, B. E. (2019). Is psychotherapy effective? A re-analysis of treatments for depression. Epidemiology and psychiatric sciences, 28(3), 268-274. https://doi.org/10.1017/ S2045796018000355 google scholar
• Münch, R. (2014). Academic capitalism: Universities in the global struggle for excellence. Routledge. google scholar
• Nahum, D., Alfonso, C. A. ve Sönmez, E. (2019). Common Factors in Psychotherapy. A. Javed ve K. N. Fountoulakis (Ed.), Advances in Psychiatry içinde (s. 471-481). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-70554-5_29 google scholar
• Niemeyer, H., Musch, J. ve Pietrowsky, R. (2012). Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for schizophrenia. Schizophrenia Research, 138(2), 103-112. https://doi.org/10.1016/j.schres.2012.03.023 google scholar
• Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., Ishiyama, J., ... Yarkoni, T. (2015). SCIENTIFIC STANDARDS. Promoting an open research culture. Science (New York, N.Y.), 348(6242), 1422-1425. https://doi.org/10.1126/science.aab2374 google scholar
• Open Science Collaboration. (2015). PSYCHOLOGY. Estimating the reproducibility of psychological science. Science, 349, Article aac4716. https://doi.org/10.1126/science.aac4716 google scholar
• O’rourke, K. (2007). An historical perspective on meta-analysis: Dealing quantitatively with varying study results. Journal of the Royal Society of Medicine, 100(12), 579-582. https://doi. org/10.1177/0141076807100012020 google scholar
• Owen, J., Drinane, J. M., Idigo, K. C. ve Valentine, J. C. (2015). Psychotherapist effects in meta-analyses: How accurate are treatment effects? Psychotherapy, 52(3), 321-328. https://doi. org/10.1037/pst0000014 google scholar
• Peng, C.-Y. J. ve Chen, L.-T. (2014). Beyond Cohen’s d: Alternative Effect Size Measures for Between-Subject Designs. The Journal of Experimental Education, 82(1), 22-50. https://doi.org/10.1080/00 220973.2012.745471 google scholar
• Persons, J. B. (1991). Psychotherapy outcome studies do not accurately represent current models of psychotherapy: A proposed remedy. American Psychologist, 46(2), 99-106. https://doi.org/10.1037/0003-066X.46.2.99 google scholar
• Persons, J. B., Roberts, N. A., Zalecki, C. A. ve Brechwald, W. A. (2006). Naturalistic outcome of case formulation-driven cognitive-behavior therapy for anxious depressed outpatients. Behaviour Research and Therapy, 44(7), 1041-1051. https://doi.org/10.1016/j.brat.2005.08.005 google scholar
• Persons, J. B. ve Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists? Journal of Consulting and Clinical Psychology, 66(1), 126-135. https://doi. org/10.1037/0022-006X.66.1.126 google scholar
• Philips, B., & Falkenström, F. (2021). What Research Evidence Is Valid for Psychotherapy Research?. Frontiers in psychiatry, 11, 625380. https://doi.org/10.3389/fpsyt.2020.625380 google scholar
• Rifkin, A. (2007). Randomized controlled trials and psychotherapy research [Editorial]. The American Journal of Psychiatry, 164(1), 7-8. https://doi.Org/10.1176/appi.ajp.164.1.7 google scholar
• Rosenzweig, S. (1954). A transvaluation of psychotherapy: A reply to Hans Eysenck. The Journal of Abnormal and Social Psychology, 49(2), 298-304. https://doi.org/10.1037/h0061172 google scholar
• Rossi, J. S. (2013). Statistical power analysis. J. A. Schinka, W. F. Velicer ve I. B. Weiner (Ed.), Handbook ofpsychology: Research methods in psychology içinde (s. 71-108). John Wiley & Sons, Inc.. google scholar
• Rothstein, H. R. ve Hopewell, S. (2009). Grey literature. H. Cooper, L. V. Hedges ve J. C. Valentine (Ed.), The handbook of research synthesis and meta-analysis içinde (s. 103-125). Russell Sage Foundation. google scholar
• Rubio-Aparicio, M., Marin-Martinez, F., Sanchez-Meca, J. ve L6pez-L6pez, J. A. (2018). A methodological review of meta-analyses of the effectiveness of clinical psychology treatments. Behavior Research Methods, 50(5), 2057-2073. https://doi.org/10.3758/s13428-017-0973-8 google scholar
• Sakaluk, J. K., Williams, A. J., Kilshaw, R. E. ve Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500-509. https://doi.org/10.1037/abn0000421 google scholar
• Schafer, T. ve Schwarz, M. A. (2019). The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.00813 google scholar
• Shapiro, D. A. ve Shapiro, D. (1982). Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychological Bulletin, 92(3), 581-604. https://doi.org/10.1037/0033-2909.92.3.581 google scholar
• Shean, G. (2012). Some limitations on the external validity of psychotherapy efficacy studies and suggestions for future research. American Journal of Psychotherapy, 66(3), 227-242. https://doi. org/10.1176/appi.psychotherapy.2012.66.3.227 google scholar
• Shean, G. (2016). Psychotherapy Outcome Research: Issues and Questions. Psychodynamic Psychiatry, 44(1), 1-24. https://doi.org/10.1521/pdps.2016.44.1.1 google scholar
• Shedler, J. (2018). Where Is the Evidence for “Evidence-Based” Therapy? Psychiatric Clinics ofNorth America, 41(2), 319-329. https://doi.org/10.1016/j.psc.2018.02.001 google scholar
• Shelby, L. B. ve Vaske, J. J. (2008). Understanding meta-analysis: A review of the methodological literature. Leisure Sciences, 30(2), 96-110. https://doi.org/10.1080/01490400701881366 google scholar
• Shinohara, K., Tajika, A., Imai, H., Takeshima, N., Hayasaka, Y. ve Furukawa, T. A. (2015). Protocol registration and selective outcome reporting in recent psychiatry trials: New antidepressants and cognitive behavioural therapies. Acta Psychiatrica Scandinavica, 132(6), 489-498. https://doi. org/10.1111/acps.12502 google scholar
• Smit, Y., Huibers, M. J. H., Ioannidis, J. P. A., van Dyck, R., van Tilburg, W. ve Arntz, A. (2012). google scholar
• The effectiveness of long-term psychoanalytic psychotherapy—A meta-analysis of randomized controlled trials. Clinical Psychology Review, 32(2), 81-92. https://doi.org/10.1016/j. cpr.2011.11.003 google scholar
• Smith, M. L. ve Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752-760. https://doi.org/10.1037/0003-066X.32.9.752 google scholar
• Sohn, D. (1996). Publication bias and the evaluation of psychotherapy efficacy in reviews of the research literature. Clinical Psychology Review, 16(2), 147-156. https://doi.org/10.1016/0272-7358(96)00005-0 google scholar
• Remission in CBT for adult anxiety disorders: A meta-analysis. Clinical Psychology Review, 61, 1-8. https://doi.org/10.1016/j.cpr.2018.03.002 google scholar
• Stricker, J. ve Günther, A. (2019). Scientific misconduct in psychology: A systematic review of prevalence estimates and new empirical data. Zeitschrift für Psychologie, 227(1), 53-63. http:// dx.doi.org/10.1027/2151-2604/a000356 google scholar
• Strupp, H. H. (1963). The outcome problem in psychotherapy revisited. Psychotherapy: Theory, Research & Practice, 1(1), 1-13. https://doi.org/10.1037/h0088565 google scholar
• Tackett, J. L., Brandes, C. M., King, K. M. ve Markon, K. E. (2019). Psychology’s replication crisis and clinical psychological science. Annual Review of Clinical Psychology, 15, 579-604. https://doi. org/10.1146/annurev-clinpsy-050718-095710 google scholar
• Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s time to broaden the replicability conversation: Thoughts for and from clinical psychological science. Perspectives on PsychologicalScience, 12(5), 742-756. https://doi. org/10.1177/1745691617690042 google scholar
• Task Force on Promotion and Dissemination of Psychological Procedures Division of Clinical Psychology. (1995). Training in and dissemination of empirically-validated psychological treatments: Report and recommendations. The Clinical Psychologist, 48(1), 3-23. google scholar
• Thase, M. E. (1999). What is the investigator allegiance effect and what should we do about it? Clinical Psychology: Science and Practice, 6(1), 113-115. https://doi.org/10.1093/clipsy.6.1.113 google scholar
• Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W. ve Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69(3), 218-229. https://doi.org/10.1037/ a0035099 google scholar
• Tolin, D. F. (2010). Is cognitive-behavioral therapy more effective than other therapies?: A meta-analytic review. Clinical Psychology Review, 30(6), 710-720. https://doi.org/10.1016/j.cpr.2010.05.003 google scholar
• Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D. ve Thombs, B. D. (2015). Empirically supported treatment: Recommendations for a new model. Clinical Psychology: Science and Practice, 22(4), 317-338. https://doi.org/10.1111/cpsp.12122 google scholar
• van der Zweerde, T., Bisdounis, L., Kyle, S. D., Lancee, J. ve van Straten, A. (2019). Cognitive behavioral therapy for insomnia: A meta-analysis of long-term effects in controlled studies. Sleep Medicine Reviews, 48, 101208. https://doi.org/10.1016/j.smrv.2019.08.002 google scholar
• Vevea, J. L. ve Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10(4), 428-443. google scholar
• von Wolff, A., Jansen, M., Hölzel, L. P., Westphal, A., Harter, M. ve Kriston, L. (2014). Generalizability of findings from efficacy trials for chronic depression: An analysis of eligibility criteria. Psychiatric Services, 65(7), 897-904. https://doi.org/10.1176/appi.ps.201300309 google scholar
• Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Routledge. google scholar
• Wampold, B. E. (2015). How important are the common factors in psychotherapy? An update. World Psychiatry, 14(3), 270-277. https://doi.org/10.1002/wps.20238 google scholar
• Wampold, B. E. (2019). The basics of psychotherapy: An introduction to theory and practice (2. Bs.). American Psychological Association. https://doi.org/10.1037/0000117-000 google scholar
• Wampold, B. E., Hollon, S. D. ve Hill, C. E. (2011). Unresolved questions and future directions in psychotherapy research. History of psychotherapy: Continuity and change (2. Bs) içinde (s. 333356). American Psychological Association. https://doi.org/10.1037/12353-011 google scholar
• Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K. ve Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empiricially, “all must have prizes.” Psychological Bulletin, 122(3), 203-215. https://doi.org/10.1037/0033-2909.122.3.203 google scholar
• Weinberger, J. (1995). Common factors aren’t so common: The common factors dilemma. Clinical Psychology: Science and Practice, 2(1), 45-69. https://doi.Org/10.1111/j.1468-2850.1995. tb00024.x google scholar
• Westbrook, D. ve Kirk, J. (2005). The clinical effectiveness of cognitive behaviour therapy: Outcome for a large sample of adults treated in routine practice. Behaviour Research and Therapy, 43(10), 1243-1261. https://doi.org/10.1016/j.brat.2004.09.006 google scholar
• Westen, D. ve Bradley, R. (2005). Empirically supported complexity: Rethinking evidence-based practice in psychotherapy. Current Directions in Psychological Science, 14(5), 266-271. https:// doi.org/10.1111/j.0963-7214.2005.00378.x google scholar
• Westen, D. ve Morrison, K. (2001). A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: An empirical examination of the status of empirically supported therapies. Journal of Consulting and Clinical Psychology, 69(6), 875-899. https://doi. org/10.1037/0022-006X.69.6.875 google scholar
• Westen, D., Novotny, C. M. ve Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130(4), 631-663. https://doi.org/10.1037/0033-2909.130.4.631 google scholar
• Wierzbicki, M. ve Pekarik, G. (1993). A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice, 24(2), 190-195. https://doi.org/10.1037/0735-7028.24.2.190 google scholar
• Wojnarowski, C., Firth, N., Finegan, M. ve Delgadillo, J. (2019). Predictors of depression relapse and recurrence after cognitive behavioural therapy: A systematic review and meta-analysis. Behavioural and Cognitive Psychotherapy, 47(5), 514-529. https://doi.org/10.1017/S1352465819000080 google scholar