A Mobile Robot Application for Constructing Semantic and Metric Maps of Search and Rescue Arenas with Point-Based Deep Learning

Year 2021, , 11 - 22, 21.06.2021

Muhammed Kocaoğlu , Yunus Emre Işıkdemir , Muhammed Ali Uzun , Kaya Turgut , Muhammed Oguz Tas , Burak Kaleci

https://doi.org/10.5281/zenodo.4589489

Abstract

This study aims to create semantic and metric maps of a post-disaster indoor environment similar to standard the National Institute of Standards and Technology (NIST) search and rescue test arenas that first-responders can easily read. We prefer to use point cloud data acquired with an RGB-D camera since it does not be affected by post-disaster environments’ dusty and dull nature. Besides, each point cloud data is processed separately so that the semantic and metric maps grow incrementally. The Dynamic Graph Convolutional Neural Network (DGCNN) is used to classify points as sematic categories such as walls, terrain, and inclined and straight ramps. RTAB-Map and the semantic map are utilized to generate the octree-based 3D metric map. The experiments are conducted in a simulated environment modelled with Gazebo similar to NIST test arenas to show the effectiveness of the proposed method.

Keywords

Search and Rescue, Mobile Robot, 3D Semantic Map, 3D Metric Map, Point Cloud, Point-Based Deep Learning

Arama Kurtarma Alanlarında Metrik ve Anlamsal Harita Üretmek için Nokta Tabanlı Derin Öğrenme ile Bir Gezgin Robot Uygulaması

Abstract

Zehirli madde yayılımı, sel, yangın ve deprem gibi afetlerden sonra bina içi ortamlarda arama ve kurtarma görevleri için robotların kullanılmasına yönelik çalışmalar son yıllarda hız kazanmıştır. Bu çalışmanın ana motivasyonu, ilk yardım ekiplerinin kolaylıkla kullanabileceği afet sonrası bina içi ortamın metrik ve anlamsal haritalarını oluşturmaktır. Bu çalışmada, afet ortamında karşılaşılabilecek toz, duman ve yetersiz ışıklandırma gibi faktörlerden etkilenmeyen ve nesnelerin geometrik yapısını yüksek doğrulukta temsil edebilen nokta bulutu verilerinin kullanılmasına karar verilmiştir. Her bir adımda alınan nokta bulutu ayrı ayrı işlenerek önerilen yöntemin hesaplama karmaşıklığının düşürülmesi amaçlanmıştır. Anlamsal haritanın üretilmesi aşamasında geçmiş çalışmalardan farklı olarak nokta tabanlı derin öğrenme mimarisi DGCNN kullanılmıştır. Böylece nokta bulutunda yer alan her noktanın anlamsal sınıfı (duvar, zemin, eğimli ve düz rampa) belirlenmiştir. 3B metrik haritanın oluşturulması için RTAB-Map ve anlamsal harita birlikte kullanılarak 8-li ağaç yapısında bir gösterim elde edilmiştir. Bu haritada önceki çalışmalardan farklı olarak sadece dolu ve boş vokseller değil, aynı zamanda duvar, zemin ve rampa sınıflarına ait olan vokseller de yer almaktadır. Önerilen yöntemin test edilmesi için Gazebo benzetim ortamında NIST ortamlarına benzer bir test alanı modellenmiş ve bir Pionner 3-AT gezgin robot teleoperasyon yöntemi ile gezdirilmiştir. Test sonuçları önerilen yöntemin başarılı bir şekilde anlamsal ve metrik harita üretebildiğini göstermiştir.

Keywords

Arama ve Kurtarma, Gezgin Robot, 3B Anlamsal Harita, 3B Metrik Harita, Nokta Bulutu, Nokta Tabanlı Derin Öğrenme

References

• H. Kitano and S. Tadokoro, “RoboCup Rescue: A Grand Challenge for Multiagent and Intelligent Systems,” AI Magazine, vol. 22, no. 1, pp. 39-52, 2001.
• A. Jacoff, E. Messina, B. A. Weiss, S. Tadokoro, and Y. Nakagawa, "Test arenas and performance metrics for urban search and rescue robots," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 2003.
• H. L. Akin, N. Ito, A. Jacoff, A. Kleiner, J. Pellenz, and A. Visser, “Robocup rescue robot and simulation leagues”. AI magazine, vol. 34, no. 1, pp. 78-86, 2013.
• S. Balakirsky, S. Carpin, A. Kleiner, M. Lewis, A. Visser, J. Wang, and V. A. Ziparo, “Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from robocup rescue”, Journal of Field Robotics, vol. 24, no. 11, pp. 943-967, 2007.
• S. Balakirsky, S. Carpin, and A. Visser, “Evaluating the robocup 2009 virtual robot rescue competition.”, In Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems, pp. 109-114, 2009.
• F. Amigoni, A. Visser, and M. Tsushima, “RoboCup 2012 Rescue Simulation League Winners.”, In Robocup 2012: Robot Soccer World Cup XVI,Lecture Notes in Artificial Intelligence, Berlin, Springer, 2013.
• R. Sheh, S. Schwertfeger, A. Visser, “16 Years of RoboCup Rescue,” Künstliche Intelligenz, vol. 30, no.3, pp. 267-277, 2016.
• ROS, Available: https://www.ros.org/, Accessed Time: 15.02.2021
• GazeboSim, Available: http://gazebosim.org/, Accessed Time: 15.02.2021
• K. Ito, K. Yokochi, and Y. Tanaka, “Chukyo Rescue A Team Description Paper for Robocup 2017 Virtual Robot League.”, 2017.
• G. Grisetti, C. Stachniss, and W. Burgard, “Improved techniques for grid mapping with rao-blackwellized particle filters.”, IEEE transactions on Robotics, vol. 23 no. 1, pp. 34-46, 2007.
• S. Kohlbrecher, O. Von Stryk, J. Meyer, and U. Klingauf, "A flexible and scalable SLAM system with full 3D motion estimation.", IEEE international symposium on safety, security, and rescue robotics, pp.155-160, 2011.
• S. Yavuz, M. F. Amasyalı, E. Uslu, F. Cakmak, N. Altuntas, S. Marangoz, M. B. Dilaver, A. E. Kırlı, “YILDIZ Team Description Paper for Rescue Simulation League Virtual Robots Competition 2017”, 2017.
• A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, "OctoMap: An efficient probabilistic 3D mapping framework based on octrees.", Autonomous robots, vol. 34 no. 3, pp. 189-206, 2013.
• S. M. Ahmadi, M. H. G. Nejad, and A. Torabian, “SOS RS Team Description Paper RoboCup 2018 Rescue Virtual Robot League”, 2018.
• M. Montemerlo, S. Thrun, D. Koller and B.Wegbreit, “FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem”, Association for the Advancement of Artificial Intelligence (AAAI) Conference, pp. 593-598, 2002.
• J. Ziegler, J. Gleichhauf and C. Pfitzner, “RoboCup Rescue 2018 Team Description Paper AutonOHM”, 2018.
• S. May, P. Koch, R. Koch, C. Merkl, C. Pfitzner, and A. Nuechter, “A Generalized 2D and 3D Multi-Sensor Data Integration Approach based on Signed Distance Functions for Multi-Modal Robotic Mapping.”, In Vision, Modeling, and Visualization (VMV), pp. 95-102, 2014.
• J. H. Kim, X. Lin, N. Kanyok, A. Shaker, P. Poudel, I. Cardenas, N. Karina, C. Cabrera, H. Jeong, and G. P. Sharma, "RoboCup Rescue 2019 TDP Virtual Robot Simulation." 2019.
• M. Labbé and F. Michaud, “Memory management for real-time appearance-based loop closure detection.”, IEEE/RSJ international conference on intelligent robots and systems, pp. 1271-1276, 2011.
• rtabmap_ros, Available: http://wiki.ros.org/rtabmap_ros. Accessed Time: 15.02.2021.
• T. Rabbani, F. A. Van Den Heuvel, G. Vosselman, "Segmentation of point clouds using smoothness constraints," ISPRS commission V symposium: image engineering and vision metrology. International Society for Photogrammetry and Remote Sensing (ISPRS), Dresden, Germany, 2006.
• M. A. Fischler, R. C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, 1981.
• W. Deng, K. Huang, X. Chen, Z. Zhou, C. Shi, R. Guo, and H. Zhang,)., “Semantic RGB-D SLAM for Rescue Robot Navigation.”, IEEE Access, vol. 8, pp.221320-221329, 2020.
• K. Turgut and B. Kaleci, "Comparison of Deep Learning Techniques for Semantic Classification of Ramps in Search and Rescue Arenas," 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), Istanbul, Turkey, pp. 1-6, 2020.
• C. R. Qi, H. Su, M. Kaichun, L. J. Guibas, ‘‘PointNet: Deep learning on point sets for 3D classification and segmentation,’’ in Proc. IEEE Conf. Comput. Vision. Pattern Recognit. (CVPR), Honolulu, HI, USA, 2017.
• C. R. Qi, L. Yi, H. Su, L. J. Guibas, ‘‘PointNet++: Deep hierarchical feature learning on point sets in a metric space,’’ Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 2017.
• Y. Li, R. Bu, M. Sun, W. Wu, X. Di, B. Chen, “PointCNN: Convolution on x-transformed points,” Conference on Neural Information Processing Systems (NIPS), Montreal, Canada, 2018.
• Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” ACM Transactions on Graphics, vol. 38, no. 5, Article 146, 12 pages, 2019.
• Point Cloud Library Plane Segmentation Code. Available: https://pointclouds.org/documentation/tutorials/random_sample_consensus.html, Access date: 15.02.2021.
• S. Thrun, “Learning Occupancy Grid Maps with Forward Sensor Models”. Autonomous Robots, vol. 15, pp. 111–127, 2003.
• A. Vo, L. Truong-Hong, D. F. Laefer, and M. Bertolotto, “Octree-based region growing for point cloud segmentation”, ISPRS Journal of Photogrammetry and Remote Sensing, vol. 104, pp. 88-100, 2015.
• Hector_nist_arenas_gazebo, Available: http://wiki.ros.org/hector_nist_arenas_gazebo, Accessed Time: 15.02.2021
• teleop_twist_keyboard, Available: http://wiki.ros.org/teleop_twist_keyboard, Accessed Time: 15.02.2021
• Tensorflow, Available: https://www.tensorflow.org/, Accessed Time: 15.02.2021
• pybind11, Available: https://github.com/pybind/pybind11, Accessed Time: 15.02.2021
• octree_viewer, Available: https://github.com/PointCloudLibrary/pcl/blob/master/tools/octree_viewer.cpp, Accessed Time: 15.02.2021
• R. B. Rusu, S. Cousins, “3D is here: Point Cloud Library (PCL),” IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 2011.