The Effect of Asynchronous Message Queues on the Communication of IoT Devices

Year 2021, Volume: 4 Issue: 3, 44 - 53, 30.12.2021

Ahmet Toprak , Abdül Halim Zaim , Feyzanur Sağlam Toprak

Abstract

Nowadays, IoT (Internet of Things) devices have reached quite high numbers. This situation brought with it the presence of high density, unstructured data. The difficulties in obtaining, processing, storing and visualizing this data made it necessary to use the components of the big data system. High density data should be taken from IoT devices instantly, meaningful data should be obtained from these unstructured data, the meaningful data should be stored and presented at the request of the user when needed. In this article, a model is designed to process the data obtained from IoT devices and transmit them instantly to the end user. In the study, unstructured data collected primarily from IoT devices were subjected to data pre-processing steps. Significant words were determined from the data obtained after the data pre-processing steps. For this purpose, the Helmholtz Principle has been applied. After meaningful word detection, it is directed to both Rabbit MQ messaging queue and IBM MQ message queue separately to instantly process data on the subject of each meaningful word. Apache Storm topology was used to instantly receive and process the messages transmitted to the queues. According to the results obtained, messages sent to the IBM MQ message queue are consumed 30% faster than the Rabbit MQ message queue.

Keywords

Apache Storm, Big Data, Elasticsearch, Hadoop, Helmholtz Principle, Ibm Mq, Internet of Things, Rabbit Mq

References

• Lausch, Angela & Schmidt, Andreas & Tischendorf, Lutz. (2015). Data mining and linked open data – New perspectives for data analysis in environmental research. Ecological Modelling. 295. 5-17. 10.1016/j.ecolmodel.2014.09.018.
• Shaqiri, Bledi. (2017). Exploring Techniques of Improving Security and Privacy in Big Data. 10.13140/RG.2.2.23201.10089.
• S. G. Manikandan and S. Ravi, "Big Data Analysis Using Apache Hadoop," 2014 International Conference on IT Convergence and Security (ICITCS), Beijing, 2014, pp. 1-4, doi: 10.1109/ICITCS.2014.7021746.
• M. Sogodekar, S. Pandey, I. Tupkari and A. Manekar, "Big data analytics: hadoop and tools," 2016 IEEE Bombay Section Symposium (IBSS), Baramati, 2016, pp. 1-6, doi: 10.1109/IBSS.2016.7940204.
• K. Singh and R. Kaur, "Hadoop: Addressing challenges of Big Data," 2014 IEEE International Advance Computing Conference (IACC), Gurgaon, 2014, pp. 686-689, doi: 10.1109/IAdCC.2014.6779407.
• A. Verma, A. H. Mansuri and N. Jain, "Big data management processing with Hadoop MapReduce and spark technology: A comparison," 2016 Symposium on Colossal Data Analysis and Networking (CDAN), Indore, 2016, pp. 1-4, doi: 10.1109/CDAN.2016.7570891.
• S. Sagiroglu and D. Sinanc, "Big data: A review," 2013 International Conference on Collaboration Technologies and Systems (CTS), San Diego, CA, 2013, pp. 42-47, doi: 10.1109/CTS.2013.6567202.
• G. Harerimana, B. Jang, J. W. Kim and H. K. Park, "Health Big Data Analytics: A Technology Survey," in IEEE Access, vol. 6, pp. 65661-65678, 2018, doi: 10.1109/ACCESS.2018.2878254.
• A. R. Reddy and P. S. Kumar, "Predictive Big Data Analytics in Healthcare," 2016 Second International Conference on Computational Intelligence & Communication Technology (CICT), Ghaziabad, 2016, pp. 623-626, doi: 10.1109/CICT.2016.129.
• J. Liao, X. Zhuang, R. Fan and X. Peng, "Toward a General Distributed Messaging Framework for Online Transaction Processing Applications," in IEEE Access, vol. 5, pp. 18166-18178, 2017, doi: 10.1109/ACCESS.2017.2717930.
• F. Famili, W. Shen, R. Weber and E. Simoudis, “Data preprocessing and intelligent data analysis,” Intell. Data Anal, vol. 1(4), pp. 3-23, 1997. https://doi.org/10.1016/S1088-467X (98)00007-9
• V. Agarwal, “Research on data preprocessing and categorization technique for smartphone review analysis,” International Journal of Computer Applications, vol. 131(4), pp. 30-36, 2015. https://www.doi.org/10.5120/ijca2015907309
• B. Dadachev, A. Balinsky, H. Balinsky and S. Simske, “On the Helmholtz Principle for data mining,” Third International Conference on Emerging Security Technologies, pp. 99-102, 2012. https://www.doi.org/10.1109/EST.2012.11
• S. Jabri, A. Dahbi, T. Gadi and A. Bassir, “Ranking of text documents using TF-IDF weighting and association rules mining,” 2018 4th International Conference on Optimization and Applications (ICOA), pp. 1-6, 2018. https://www.doi.org/10.1109/ICOA.2018.837057
• A.G. Jivani, “A comparative study of stemming algorithms,” Int. J. Comp. Tech. Appl, vol. 2(6), pp. 1930-1938, 2011.
• Caner Tosun. “RabbitMQ Nedir? Windows Üzerinde Kurulumu”. http://www.canertosuner.com/post/rabbitmq-nedir-windows-uzerinde-kurulumu (18.06.2020).
• Oguzhan İnan. “Hadoop Ekosistemi ve Kullanılan Araçlar”. https://oguzhaninan.gitlab.io/Hadoop-Ekosistemi-ve-Kullanilan-Araclar/#what-is-storm (18.06.2020).
• Womaneng. “Kafka-Flink-Storm-Platformları”. https://womaneng.com/kafka-flink-storm-platformlari/ (18.06.2020).
• Büyükveri. “Büyük Veri Ekosistemi”. http://www.buyukveri.co/buyuk-veri-ekosistemi/ (18.06.2020).
• Devnot. “Bir Bakışta ElasticSearch”. http://devnot.com/2017/bir-bakista-elasticsearch/ (18.06.2020).
• Elastic. “Elasticsearch and Kibana Deployments on Azure”. https://www.elastic.co/blog/elasticsearch-and-kibana-deployments-on-azure
• Burcu Altınok. “REST API Nedir?” https://burcualtinok.com.tr/blog/rest-api-nedir/ (18.06.2020).
• Devnot. “REST Mimarisi ve RESTful Servisler”. http://devnot.com/2016/rest-mimarisi-ve-restful-servisler/(18.06.2020).
• Deniz İrgin. “REST ve RESTful Web Servis Kavramı”. https://denizirgin.com/rest-ve-restful-web-servis-kavram%C4%B1-30bc4400b9e0/(18.06.2020).