Bilgi Erişim Değerlendirmeleri ve Optimizasyonları İçin Modüler Bir Verimlilik Belirleme Formülü

Year 2023, Volume: 7 Issue: 1, 209 - 228, 02.01.2024

Veli Özcan Budak

https://doi.org/10.26650/acin.1198925

Abstract

Verimlilik kavramı, Bilgi Erişim (BE) literatüründe temel olarak kullanıcılardan ziyade sistemlerin verimliliği ile ilişkilendirilmiştir. Kullanılabilirlik literatüründe ise bu kavram, bir kullanıcının bir görevi ne kadar sürede tamamladığına karşılık gelen kullanıcı tabanlı bir bakış açısıyla tanımlanır. Yine de, ortak amaç her iki literatür için de zamanı verimli kullanmaktır. Bu çalışma, BE literatüründeki etkinlik kavramını, kullanılabilirlik literatüründeki kullanıcı tabanlı etkinlik penceresinden incelemektedir. Bu çalışmada, kullanılabilirlik perspektifinden BE sistem değerlendirmelerine ve optimizasyonlarına odaklanarak farklı verimlilik göstergeleri oluşturmak için modüler bir verimlilik belirleme formülü (MEDEF) önerilmiştir. MEDEF, BE çalışmalarında hâlihazırda kullanılan etkililik metriklerine ve verimlilik göstergelerine dayalı bir verimlilik göstergesi üreticisi şeklinde düşünülebilir. Bu çalışma kapsamında, sekiz MEDEF tabanlı verimlilik göstergesi oluşturulmuş ve hâlihazırda BE çalışmalarında kullanılan birkaç temel verimlilik göstergesiyle karşılaştırılmıştır. Çalışmanın ilk amacı, MEDEF temelli göstergelerin ne kadar tutarlı olduğunu ve bu göstergelerin mevcut temel göstergelere göre daha başarılı/güvenilir olup olmadığını ortaya koymak iken, ikincisi, kullanılabilirlik açısından BE sistemlerinin değerlendirmelerinde verimlilik göstergelerinin kullanımına bir örnek oluşturmaktır. Bir aylık etkileşimli kullanıcı davranışlarından elde edilen genel bulgular, MEDEF tabanlı göstergelerin temel göstergelerden daha iyi performans gösterdiğini ve temel göstergelerdeki yansımaları daha da güçlendirdiğini göstermiştir. MEDEF‘in potansiyeline ilişkin çeşitli kullanım senaryoları da çalışma kapsamında paylaşılmakta ve tartışılmaktadır.

Keywords

İnsan-Bilgisayar Etkileşimi, Verimlilik Göstergesi, Oturum Terk Etme, Etkileşimli Bilgi Erişim

A Modular Efficiency Determination Formula for Information Retrieval Evaluations and Optimizations

Abstract

The notion of efficiency has typically been associated with the efficiency of systems rather than users in Information Retrieval (IR) literature. In the usability literature, on the other hand, this notion is defined from a user-based perspective, corresponding to how long a user accomplishes a task. Despite this, the common aim for both has to do with the efficient use of time. This study examines the efficiency notion in the IR literature from a user-based efficiency window in the usability literature. In the present study, a modular efficiency determination formula (MEDEF) to create different efficiency indicators by focusing on IR system evaluations and optimizations from the usability perspective is proposed. The MEDEF can be thought of as an efficiency indicator creator based on both effectiveness metrics and efficiency indicators already used in IR studies. In the scope of this study, eight MEDEF-based efficiency indicators were created and compared to several baseline efficiency indicators already used in IR studies. While the study’s first aim is to reveal how consistent the MEDEF-based indicators are and whether these indicators are more successful/reliable than the baselines, the second is to set an example of the usage of efficiency indicators in evaluations of IR systems from a usability perspective. General findings from interactive user behaviour for one month show that the MEDEF-based indicators outperform the baseline indicators and further strengthen the reflections in the baseline indicators. Several usage scenarios regarding the potential of the MEDEF are also shared and discussed in the scope of the study.

Keywords

Human-Computer Interaction, Efficiency Indicator, Session Abandonment, Interactive Information Retrieval

References

• Agichtein, E., Brill, E., & Dumais, S. (2006). Improving web search ranking by incorporating user behavior information. Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2006, 19-26. New York, New York, USA: ACM Press. https://doi.org/10.1145/1148170.1148177 google scholar
• Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006). Learning user interaction models for predicting web search result preferences. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’06, 3. New York, New York, USA: ACM Press. https://doi.org/10.1145/1148170.1148175 google scholar
• Alhabashneh, O., Iqbal, R., Doctor, F., & James, A. (2017). Fuzzy rule based profiling approach for enterprise information seeking and retrieval. Information Sciences, 394-395, 18-37. https://doi.org/10.1016/J.INS.2016.12.040 google scholar
• Arguello, J. (2014). Predicting search task difficulty. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 8416 LNCS (pp. 88-99). https://doi.org/10.1007/978-3-319-06028-6_8 google scholar
• Arkhipova, O., Grauer, L., Kuralenok, I., & Serdyukov, P. (2015). Search Engine Evaluation based on Search Engine Switching Prediction. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 723-726. New York, NY, USA: ACM. https://doi.org/10.1145/2766462.2767786 google scholar
• Aula, A., Khan, R. M., & Guan, Z. (2010). How does search behavior change as search becomes more difficult? Proceedings of the 28th International Conference on Human Factors in Computing Systems - CHI ’10, 35. New York, New York, USA: ACM Press. https://doi.org/10.1145/1753326.1753333 google scholar
• Balakrishnan, V., & Zhang, X. (2014). Implicit user behaviours to improve post-retrieval document relevancy. Computers in Human Behavior, 33, 104-112. https://doi.org/10.1016/J.CHB.2014.01.001 google scholar
• Beierle, F., Aizawa, A., Collins, A., & Beel, J. (2020). Choice overload and recommendation effectiveness in related-article recommendations. International Journal on Digital Libraries, 21(3), 231-246. https://doi.org/10.1007/s00799-019-00270-7 google scholar
• Belkin, N. J., Kelly, D. F., Kim, G., Kim, J. Y., Lee, H., Muresan, G., Tang, M. C., Yuan, X., Cool, C. (2003). Query Length in Interactive Information Retrieval. SIGIR Forum (ACM Special Interest Group on Information Retrieval), (SPEC. ISS.), 205-212. New York, New York, USA: ACM Press. https://doi.org/10.1145/860472.860474 google scholar
• Borisov, A., Markov, I., de Rijke, M., & Serdyukov, P. (2016). A Context-aware Time Model for Web Search. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 205-214. New York, NY, USA: ACM. https://doi.org/10.1145/2911451.2911504 google scholar
• Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval - SIGIR ’04, 25. New York, New York, USA: ACM Press. https://doi.org/10.1145/1008992.1009000 google scholar
• Büttcher, S., Clarke, C. L. A., & Cormack, G. V. (2010). Information Retrieval: Implementing and Evaluating Search Engines. The MIT Press. google scholar
• Cen, R., Liu, Y., Zhang, M., Ru, L., & Ma, S. (2009). Study on the click context of web search users for reliability analysis. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 5839 LNCS (pp. 397-408). https://doi.org/10.1007/978-3-642-04769-5_35 google scholar
• Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009). Expected reciprocal rank for graded relevance. Proceeding of the 18th ACM Conference on Information and Knowledge Management - CIKM ’09, 621. New York, New York, USA: ACM Press. https://doi.org/10.1145/1645953.1646033 google scholar
• Claypool, M., Brown, D., Le, P., & Waseda, M. (2001). Inferring user interest. IEEE Internet Computing, 5(6), 32-39. https://doi.org/10.1109/4236.968829 google scholar
• Colace, F., De Santo, M., Greco, L., & Napoletano, P. (2015). Improving relevance feedback-based query expansion by the use of a weighted word pairs approach. Journal of the Association for Information Science and Technology, 66(11), 2223-2234. https://doi.org/10.1002/asi.23331 google scholar
• Croft, B., Metzler, D., & Strohman, T. (2009). Search Engines: Information Retrieval in Practice (1st ed.). Boston: Pearson. google scholar
• Diriye, A., White, R., Buscher, G., & Dumais, S. (2012). Leaving so soon?: understanding and predicting web search abandonment rationales. Proceedings of the 21st ACM International Conference on Information and Knowledge Management - CIKM ’12, 1025. New York, New York, USA: ACM Press. https://doi.org/10.1145/2396761.2398399 google scholar
• Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E., and Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825-2830. Retrieved from http://scikit-learn.sourceforge.net. google scholar
• Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve Web search. ACM Transactions on Information Systems, 23(2), 147-168. https://doi.org/10.1145/1059981.1059982 google scholar
• Frokjrer, E., Hertzum, M., & Hornbrek, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’00, 345-352. New York, New York, USA: ACM Press. https://doi. org/10.1145/332040.332455 google scholar
• Hassan, A. (2012). A semi-supervised approach to modeling web search satisfaction. Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’12, 275. New York, New York, USA: ACM Press. https://doi.org/10.1145/2348283.2348323 google scholar
• Hassan, A., Song, Y., & He, L. (2011). A task level metric for measuring web search satisfaction and its application on improving relevance estimation. Proceedings of the 20th ACM International Conference on Information and Knowledge Management - CIKM ’11, 125. New York, New York, USA: ACM Press. https://doi.org/10.1145/2063576.2063599 google scholar
• Hassan, A., White, R. W., Dumais, S. T., & Wang, Y.-M. (2014). Struggling or exploring? Disambiguating Long Search Sessions. Proceedings of the 7th ACM International Conference on Web Search and Data Mining, 53-62. New York, NY, USA: ACM. https://doi.org/10.1145/2556195.2556221 google scholar
• Hornbsk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102. https://doi.org/10.1016/J.IJHCS.2005.06.002 google scholar
• Jansen, B. J., & Spink, A. (2005). An analysis of Web searching by European AlltheWeb.com users. Information Processing & Management, 41(2),361-381. https://doi.org/10.1016/S0306-4573(03)00067-0 google scholar
• Jansen, B. J., Spink, A., Bateman, J., & Saracevic, T. (1998). Real life information retrieval: a study of user queries on the Web. ACM SIGIR Forum, 32(1), 5-17. https://doi.org/10.1145/281250.281253 google scholar
• Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the web. Information Processing & Management, 36(2), 207-227. https://doi.org/10.1016/S0306-4573(99)00056-4 google scholar
• Jarvelin, K., & Kekalainen, J. (2000). IR evaluation methods for retrieving highly relevant documents. SIGIR Forum (ACM Special Interest Group on Information Retrieval), 41-48. New York, New York, USA: ACM Press. https://doi.org/10.1145/3130348.3130374 google scholar
• Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422-446. https://doi.org/10.1145/582415.582418 google scholar
• Jiang, D., Leung, K. W. T., Yang, L., & Ng, W. (2015). Query suggestion with diversification and personalization. Knowledge-Based Systems, 89, 553-568. https://doi.org/10.1016/J.KNOSYS.2015.09.003 google scholar
• Joachims, T. (2002). Optimizing search engines using clickthrough data. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 133-142. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/775047.775067 google scholar
• Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. SIGIR 2005 -Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 154-161. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1076034.1076063 google scholar
• Jung, S., Herlocker, J. L., & Webster, J. (2007). Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3), 791-807. https://doi.org/10.1016/J.IPM.2006.07.021 google scholar
• Kelly, D., & Fu, X. (2007). Eliciting better information need descriptions from users of information search systems. Information Processing & Management,43(1), 30-46. https://doi.org/10.1016/J.IPM.2006.03.006 google scholar
• Kim, Y., Hassan, A., White, R. W., & Zitouni, I. (2014). Modeling dwell time to predict click-level satisfaction. Proceedings ofthe 7th ACM International Conference on Web Search and Data Mining, 193-202. New York, NY, USA: ACM. https://doi.org/10.1145/2556195.2556220 google scholar
• Kuhar, M., & Mercun, T. (2022). Exploring user experience in digital libraries through questionnaire and eye-tracking data. Library & Information Science Research, 44(3), 101175. https://doi.org/10.1016/J.LISR.2022.101175 google scholar
• Lee, C.-J., Teevan, J., & de la Chica, S. (2014). Characterizing multi-click search behavior and the risks and opportunities of changing results during use. Proceedings ofthe 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 515-524. New York, NY, USA: ACM. https://doi.org/10.1145/2600428.2609588 google scholar
• Liu, Y., Chen, Y., Tang, J., Sun, J., Zhang, M., Ma, S., & Zhu, X. (2015). Different Users, Different Opinions. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 493-502. New York, NY, USA: ACM. https://doi.org/10.1145/2766462.2767721 google scholar
• Liu, C., Gwizdka, J., & Liu, J. (2010). Helping identify when users find useful documents: examination of query reformulation intervals. Proceeding of the Third Symposium on Information Interaction in Context - IIiX ’10, 215. New York, New York, USA: ACM Press. https://doi.org/10.1145/1840784.1840816 google scholar
• Liu, C., Liu, J., & Belkin, N. J. (2014). Predicting Search Task Difficulty at Different Search Stages. Proceedings of the 23rd ACM International Conference on Conference on Information andKnowledgeManagement, 569-578. New York, NY, USA: ACM. https://doi.org/10.1145/2661829.2661939 google scholar
• Makkar, A., & Kumar, N. (2018). User behavior analysis-based smart energy management for webpage ranking: Learning automata-based solution. google scholar
• Sustainable Computing: Informatics and Systems, 20, 174-191. https://doi.org/10.1016/J.SUSCOM.2018.02.003 google scholar
• Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge University Press. google scholar
• Moffat, A., & Zobel, J. (2008). Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1), 1-27. https://doi.org/10.1145/1416950.1416952 google scholar
• Nasir, J. A., Varlamis, I., & Ishfaq, S. (2019). A knowledge-based semantic framework for query expansion. Information Processing & Management, 56(5), 1605-1617. https://doi.org/10.1016/j.ipm.2019.04.007 google scholar
• Nie, L., Jiang, H., Ren, Z., Sun, Z., & Li, X. (2016). Query Expansion Based on Crowd Knowledge for Code Search. IEEE Transactions on Services Computing, 9(5), 771-783. https://doi.org/10.1109/TSC.2016.2560165 google scholar
• Nielsen, J. (1993). Usability Engineering (1st ed.). Morgan Kaufmann. google scholar
• Radlinski, F., Kurup, M., & Joachims, T. (2008). How does clickthrough data reflect retrieval quality? Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM ’08, 43. New York, New York, USA: ACM Press. https://doi.org/10.1145/1458082.1458092 google scholar
• Rosenzweig, E. (2015). Successful User Experience: Strategies and Roadmaps (1st ed.). Morgan Kaufmann. google scholar
• Rubin, J., & Chisnell, D. (2008). Handbook Of Usability Testing How To Plan Design And Conduct Effective Tests. Wiley Pub. google scholar
• Sarkar, S., Mitsui, M., Liu, J., & Shah, C. (2020). Implicit information need as explicit problems, help, and behavioral signals. Information Processing & Management, 57(2), 102069. https://doi.org/10.1016/J.IPM.2019.102069 google scholar
• Singh, J., & Sharan, A. (2017). A new fuzzy logic-based query expansion model for efficient information retrieval using relevance feedback approach. Neural Computing and Applications, 28(9), 2557-2580. https://doi.org/10.1007/s00521-016-2207-x google scholar
• Singla, A., & White, R. W. (2010). Sampling high-quality clicks from noisy click data. Proceedings of the 19th International Conference on World Wide Web - WWW ’10, 1187. New York, New York, USA: ACM Press. https://doi.org/10.1145/1772690.1772867 google scholar
• Song, Y., Shi, X., White, R., & Awadallah, A. H. (2014). Context-aware web search abandonment prediction. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 93-102. New York, NY, USA: ACM. https://doi.org/10.1145/2600428.2609604 google scholar
• Stamou, S., & Efthimiadis, E. N. (2010). Interpreting user inactivity on search results. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Vol. 5993 LNCS (pp. 100-113). https://doi. org/10.1007/978-3-642-12275-0_12 google scholar
• Vidinli, I. B., & Ozcan, R. (2016). New query suggestion framework and algorithms: A case study for an educational search engine. Information Processing and Management, 52(5), 733-752. https://doi.org/10.1016/j.ipm.2016.02.001 google scholar
• Voorhees, E. M. (1999). The TREC-8 Question Answering Track Report. Text REtrieval Conference, 77-82. google scholar
• Zhai, C., & Massung, S. (2016). Text Data Management and Analysis: A Practical Introduction to Information Retrieval and Text Mining. ACM. https:// doi.org/10.1145/2915031 google scholar