A Comprehensive Study on Question Answering Systems: Datasets, Methods and Open Research Areas

Year 2021, Volume: 14 Issue: 3, 239 - 254, 31.07.2021

Gülsüm Yiğit , Fatih Amasyalı

https://doi.org/10.17671/gazibtd.810362

Cited By: 2

Abstract

Question Answering (QA) systems allow users to get direct answers to questions they ask in natural language instead of listing documents or links. In this study, current QA datasets are introduced and compared according to various properties. Unlike other studies in QA, this study focuses on the methods used in current QA systems. These methods are discussed in four different categories and include recent studies and technologies. The models are compared with various factors such as techniques used, external knowledge, or language model. In general, attention mechanisms, language models, graph neural networks, external knowledge, collective learning, and deep learning architectures positively affect the success of QA systems. In addition, current open research areas of QA systems and possible solutions are determined, and suggestions for future QA systems are given. Systems on languages that do not have enough data, systems that can work on more than one language, systems that require the use of many information sources, and speech-based systems stand out as future research areas.

Keywords

question answering, deep learning, knowledge bases, memory networks, seq2seq

Soru Cevaplama Sistemleri Üzerine Detaylı Bir Çalışma: Veri Kümeleri, Yöntemler ve Açık Araştırma Alanları

Abstract

Soru Cevaplama (QA) sistemleri, kullanıcıların doğal dilde sordukları sorulara belge veya bağlantıları listelemek yerine doğrudan cevap almalarını sağlayan sistemlerdir. Bu çalışmada, QA sistemlerinde yaygın kullanılan veri kümeleri tanıtılmış ve çeşitli özelliklere göre karşılaştırılmıştır. Ayrıca, QA alanındaki diğer çalışmalardan farklı olarak bu çalışmada son yıllarda literatürde yer alan QA sistemlerinin arkasında kullanılan yöntemlere odaklanılmıştır. Bu yöntemler dört farklı grupta ele alınmış olup literatürdeki güncel çalışmaları ve teknolojileri içermektedir. Bu modeller kullanılan teknikler, harici bilgi kaynaklarının veya dil modelinin kullanılıp kullanılmadığı gibi faktörlere göre karşılaştırılmıştır. Dikkat mekanizmasının, dil modellerinin, çizge işleyen ağların, harici bilgi kaynaklarının, kolektif öğrenmenin ve derin öğrenme mimarilerinin QA sistemlerinin başarısı üzerinde genel olarak olumlu etkisi olduğu görülmüştür. Ayrıca, bu çalışmada QA sistemlerinin günümüzdeki açık araştırma alanları ve olası çözüm yolları belirlenerek gelecekteki QA sistemleri için önerilerde bulunulmuştur. Gelecekteki araştırma alanları olarak yeterli veriye sahip olmayan diller üzerindeki sistemler, birden fazla dil üzerinde çalışabilen sistemler, çok sayıda bilgi kaynağının kullanılmasının gerekli olduğu sistemler ve karşılıklı konuşmaya dayalı sistemler öne çıkmaktadır.

Keywords

soru cevaplama, derin öğrenme, bilgi tabanları, bellek ağları, seq2seq

References

• D. Kapashi, P. Shah, “Answering Reading Comprehension Using Memory Networks”, Stanford Deep Learn. NLP Course, 2015.
• J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation”, Conference on Empirical Methods in Natural Language Processing, Katar, 1532-1543, 2014.
• P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, “Enriching Word Vectors with Subword Information”, Trans. Assoc. Comput. Linguist., 5, 135-146, 2017.
• A. Kumar et al., “Ask me anything: Dynamic memory networks for natural language processing”, 33rd International Conference on Machine Learning, NewYork, A.B.D., 1378-1387, 2016.
• C. Xiong, V. Zhong, R. Socher, “Dynamic coattention networks for question answering”, 5th International Conference on Learning Representations, Toulon, Fransa, Nisan 2017.
• D. Golub and X. He, “Character-level question answering with attention”, EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Texas, A.B.D. 2016.
• K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor, “Freebase: A collaboratively created graph database for structuring human knowledge”, Proceedings of the ACM SIGMOD International Conference on Management of Data, A.B.D, 1247-1250, 2008.
• F. M. Suchanek, G. Kasneci, G. Weikum, “Yago: A core of semantic knowledge”, 16th International World Wide Web Conference, Alberta, Kanada, 697-706, 2007.
• Y. Chen, L. Wu, M. J. Zaki, “Bidirectional attentive memory networks for question answering over knowledge bases”, 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, A.B.D, 2019.
• D. Sorokin and I. Gurevych, “Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering”, arXiv preprint arXiv:1808.04126, 2018.
• W. T. Yih, M. W. Chang, X. He, J. Gao, “Semantic parsing via staged query graph generation: Question answering with knowledge base”, 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, 1, 1321-1331, 2015.
• L. Su, T. He, Z. Fan, Y. Zhang, M. Guizani, “Answer Acquisition for Knowledge Base Question Answering Systems Based on Dynamic Memory Network”, IEEE Access, 7, 161329-161339, 2019.
• M. Wasim, D. Waqar, D. Usman, “A Survey of Datasets for Biomedical Question Answering Systems”, Int. J. Adv. Comput. Sci. Appl., 8(7), 484-488, 2017.
• O. Kolomiyets and M. F. Moens, “A survey on question answering technology from an information retrieval perspective”, Inf. Sci. (Ny)., 181(24), 5412-5434, 2011.
• D. Diefenbach, V. Lopez, K. Singh, P. Maret, “Core techniques of question answering systems over knowledge bases: a survey”, Knowl. Inf. Syst., 55(3), 529-569, 2018.
• J. Weston et al., “Towards AI-complete question answering: A set of prerequisite toy tasks”, 4th International Conference on Learning Representations, ICLR, San Juan, Puerto Rico, 2016.
• G. Yiğit, M. F. Amasyalı, “Ask me: A Question Answering System via Dynamic Memory Networks,” 2019 Innovations in Intelligent Systems and Applications Conference, ASYU, İzmir, Türkiye, 1-5, 2019.
• Z. Yang et al., “Hotpotqa: A dataset for diverse, explainable multi-hop question answering”, 2018 Conference on Empirical Methods in Natural Language Processing, Brüksel, Belçika, 2369-2380, 2018.
• P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, “SQuad: 100,000+ questions for machine comprehension of text”, EMNLP Conference on Empirical Methods in Natural Language Processing, Texas, A.B.D, 2383-2392, 2016.
• P. Rajpurkar, R. Jia, P. Liang, “Know What You Don’t Know: Unanswerable Questions for SQuAD”, arXiv preprint arXiv:1806.03822, 2018.
• S. Reddy, D. Chen, C. D. Manning, “CoQA: A Conversational Question Answering Challenge”, Trans. Assoc. Comput. Linguist, 249-266, 2019.
• M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, S. Fidler, “MovieQA: Understanding stories in movies through question-answering”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, A.B.D, 4631-4640, 2016.
• A. Trischler et al., “NewsQA: A Machine Comprehension Dataset”, ACL, 191-200, 2017.
• J. Welbl, P. Stenetorp, S. Riedel, “Constructing Datasets for Multi-hop Reading Comprehension Across Documents”, Trans. Assoc. Comput. Linguist., 6, 287-302, 2018.
• M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension”, ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Vancouver, Canada, 1, 1601-1611,2017.
• J. Berant, A. Chou, R. Frostig, P. Liang, “Semantic parsing on freebase from question-answer pairs”, EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, Washington, A.B.D, 1533-1544, 2013.
• T. Nguyen et al., “MS MARCO: A human generated MAchine reading COmprehension dataset”, CEUR Workshop Proceedings, 2640-2660, 2016.
• A. Bordes, N. Usunier, S. Chopra, J. Weston, “Large-scale Simple Question Answering with Memory Networks”, arXiv:1506.02075, 2015.
• K. Papineni, S. Roukos, T. Ward, W. Zhu, “BLEU : a Method for Automatic Evaluation of Machine Translation”, Comput. Linguist., 311-318, 2002.
• R. D. Banker and S. M. Datar, “Sensitivity, Precision, and Linear Aggregation of Signals for Performance Evaluation”, J. Account. Res., 27(1), 21-39,1989.
• C. Y. Lin, “Rouge: A package for automatic evaluation of summaries”, Proc. Work. text Summ. branches out (WAS 2004), 74-81, 2004.
• Q. Xiao, X. Chang, X. Zhang, X. Liu, “Multi-Information Spatial-Temporal LSTM Fusion Continuous Sign Language Neural Machine Translation”, IEEE Access, 8, 216718-28,2020.
• J. Cheng, F. Zhang, X. Guo, “A Syntax-Augmented and Headline-Aware Neural Text Summarization Method”, IEEE Access, 2020.
• K. Palasundram, N. Mohd Sharef, K. A. Kasmiran, A. Azman, “Enhancements to the Sequence-to-Sequence-Based Natural Answer Generation Models”, IEEE Access, 8:218360-71, 2020.
• S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory”, Neural Comput., 9(8), 1735-80, 1997.
• J. Chung, C. Gulcehre, K. Cho, Y. Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling,”, arXiv:1412.3555, 2014.
• C. Tan, F. Wei, N. Yang, B. Du, W. Lv, M. Zhou, “S-Net: From answer extraction to answer synthesis for machine reading comprehension”, 32nd AAAI Conference on Artificial Intelligence, AAAI, Louisiana, A.B.D, 32(1), Şubat, 2018.
• T. Rocktäschel, E. Grefenstette, K. M. Hermann, T. Kočiský, P. Blunsom, “Reasoning about entailment with neural attention”, 4th International Conference on Learning Representations, ICLR, San Juan, Puerto Rico, 2016.
• Y. Hao et al., “An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge”, ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Vancouver, Kanada, 1, 221-231, 2017.
• M. Seo, A. Kembhavi, A. Farhadi, H. Hajishirzi, “Bidirectional Attention Flow for Machine Comprehension”, arXiv:1611.01603, 2016.
• W. Wang, N. Yang, F. Wei, B. Chang, M. Zhou, “Gated self-matching networks for reading comprehension and question answering”, ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Vancouver, Kanada, 1, 189-198, 2017.
• J. Weston, S. Chopra, A. Bordes, “Memory networks”, 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, San Diego, A.B.D, 2015.
• C. Xiong, S. Merity, R. Socher, “Dynamic memory networks for visual and textual question answering”, 33rd International Conference on Machine Learning, ICML, NY, A.B.D., 2397-2406, 2016.
• T. Yu, J. Yu, Z. Yu, Q. Huang, Q. Tian, “Long-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks”, IEEE Trans. Circuits Syst. Video Technol., 2021.
• F. Ma et al., “Long-term memory networks for question answering”, in CEUR Workshop Proceedings, 1986, 7-14, 2017.
• Y. Shen, P. Sen Huang, J. Gao, W. Chen, “ReasoNet: Learning to stop reading in machine comprehension”, in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, A.B.D, 1047-1055, 2017.
• L. Dong, F. Wei, M. Zhou, K. Xu, “Question answering over freebase with multi-column convolutional neural networks”, ACL-IJCNLP 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, 1, 260-269, 2015.
• K. Xu, S. Reddy, Y. Feng, S. Huang, D. Zhao, “Question answering on freebase via relation extraction and textual evidence,” 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Almanya, 1, 2326-2336, 2016.
• F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, “The graph neural network model”, IEEE Trans. Neural Networks, 20(1), 61-80,2009.
• Y. Xiao, G. Zhou, “Syntactic edge-enhanced graph convolutional networks for aspect-level sentiment classification with interactive attention”, IEEE Access, 157068-157080, 2020.
• N. de Cao, W. Aziz, I. Titov, “Question answering by reasoning across documents with graph convolutional networks”, in NAACL HLT Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, Minneapolis, 1, 2306-2317, 2019.
• Y. Cao, M. Fang, D. Tao, “BAG: Bi-directional attention entity graph convolutional network for multi-hop reasoning question answering”, in NAACL HLT Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, 1, 357-362, 2019.
• E. KARTAL et al., “Bir Öğrenciyi Üstün Zekâlı ve Yetenekli Olarak Aday Göstermek İçin Doğru Soruları Sormak: Bir Makine Öğrenmesi Yaklaşımı”, Bilişim Teknol. Derg., 13(4), 2020.
• A. Özgür and H. Erdem, “Saldırı Tespit Sistemlerinde Kullanılan Kolay Erişilen Makine Öğrenme Algoritmalarının Karşılaştırılması”, Bilişim Teknol. Derg., 5(2), 41-48, 2012.
• A. Ittycheriah, M. Franz, W.-J. Zhu, A. Ratnaparkhi, R. J. Mammone, “IBM’s Statistical Question Answering System,” in Proceedings of TREC-9 Conference, 2000.
• B. F. Green, A. K. Wolf, C. Chomsky, K. Laughery, “Baseball: An automatic question-answerer”, in Proceedings of the Western Joint Computer Conference: Extending Man’s Intellect, IRE-AIEE-ACM 1961, 219-224, 1961.
• M. Banko, E. Brill, S. Dumais, J. Lin, M. Way, “AskMSR: Question answering using the worldwide Web”, Proc. AAAI Spring Symp. Min. Answers, 7-9, 2002.
• Z. Zheng, “AnswerBus question answering system”, InHuman Language Technology Conference (HLT), 27, 2002.
• S. Stoyanchev, Y. Song, W. Lahti, “Exact phrases in information retrieval for question answering”, 9–16, 2008.
• K. Zhang and J. Zhao, “A Chinese question-answering system with question classification and answer clustering”, 7th International Conference on Fuzzy Systems and Knowledge Discovery, A.B.D., 6, 2692-2696, 2010.
• M. Al-Shenak, K. M. O. Nahar, K. M. H. Halawani, “Aqas: Arabic question answering system based on svm, svd, and lsi”, J. Theor. Appl. Inf. Technol., 97(2), 681-91, 2019.
• S. Ilhan, N. Duru, Ş. Karagöz, M. Sağır, “Metin Madenciliği ile Soru Cevaplama Sistemi”, Elektron. ve Bİlgisayar Mühendisliği Sempozyumu, Bursa, 26-30, 2008.
• M. F. Amasyalı and B. Di̇ri̇, “Bir Soru Cevaplama Sistemi: BayBilmiş”, Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 1(8), 2016.
• L. Yang and L. Song, “Contextual Aware Joint Probability Model Towards Question Answering System”, arXiv:1904.08109, 2019.
• Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, Q. V. Le, “XLNet: Generalized autoregressive pretraining for language understanding”, Advances in Neural Information Processing Systems, 2019.
• W. Wang, M. Yan, C. Wu, “Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering”, 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Melbourne, Avustralya, 1, 2018.
• W. Cui, Y. Xiao, H. Wang, Y. Song, S. W. Hwang, W. Wang, “KBQA: Learning question answering over QA corpora and knowledge bases”, VLDB Endowment, 10(5), 565-576, 2016.
• K. Xu, S. Zhang, Y. Feng, D. Zhao, “Answering Natural Language Questions via Phrasal Semantic Parsing”, in Natural Language Processing and Chinese Computing, Berlin, Heidelberg, 333–344, 2014.
• X. Liu, Y. Shen, K. Duh, J. Gao, “Stochastic answer networks for machine reading comprehension”, 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), 1, 1694-1704, 2018.
• D. Weissenborn, G. Wiese, L. Seiffe, “FastQA: A Simple and Efficient Neural Architecture for Question Answering”, arXiv:1703.04816, 2017.
• C. Tan, F. Wei, N. Yang, B. Du, W. Lv, M. Zhou, “S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension”, 32nd AAAI Conf. Artif. Intell. AAAI, Louisiana, A.B.D, Şubat 2017.
• L. Song, Z. Wang, M. Yu, Y. Zhang, R. Florian, D. Gildea, “Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks”, arXiv:1809.02040, 2018.
• J. Devlin, M. W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, NAACL HLT Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, 2019.
• D. Chen, A. Fisch, J. Weston, A. Bordes, “Reading Wikipedia to answer open-domain questions”, 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Vancouver, Kanada, 1, 1870-1879, 2017.
• Z. Wang, H. Mi, W. Hamza, R. Florian, “Multi-Perspective Context Matching for Machine Comprehension”, arXiv: 1612.04211, 2016.
• K. Xu, Y. Lai, Y. Feng, Z. Wang, “Enhancing key-value memory neural networks for knowledge based question answering”, NAACL Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, 1, 2937-2947, 2019.
• A. Abujabal, M. Riedewald, M. Yahya, G. Weikum, “Automated template generation for question answering over knowledge graphs”, 26th International World Wide Web Conference, İsviçre, 1191-1200, 2017.
• S. Hakimov, C. Unger, S. Walter, P. Cimiano, “Applying semantic parsing to question answering over linked data: Addressing the lexical gap”, International Conference on Applications of Natural Language to Information Systems, 103-109, 2015.