A novel hyperparameter tuning method for enhanced intrusion detection in network security

Year 2025, Volume: 9 Issue: 3, 519 - 534

Vahid Sinap

https://doi.org/10.31127/tuje.1624366

Abstract

Intrusion Detection Systems (IDS) are essential for ensuring the security of enterprise networks and cloud-based systems, as they defend against sophisticated and evolving cyberattacks. Machine learning (ML) techniques have emerged as effective tools to enhance IDS performance, addressing the limitations of traditional methods. This study proposes a novel hyperparameter tuning method for ML-based IDS, leveraging the NSL-KDD dataset with extensive feature selection and preprocessing to address data imbalance and redundancy. The method, integrating adaptive refinement with stochastic perturbation, optimizes classifiers such as Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB), achieving both higher detection accuracy (99.90% with RF) and improved computational efficiency. This approach excels due to its dynamic adjustment of parameter ranges and controlled randomness, converging faster than traditional Grid Search and Random Search by reducing iterations by up to 87.5%. The experimental results demonstrate that tree-based models, particularly RF, outperform others due to their ability to model complex, non-linear patterns, enhanced by the proposed tuning method. Measured in terms of convergence speed, CPU time, and memory usage, this method proves suitable for deployment in real-time, resource-constrained environments, offering a scalable and efficient solution for network security.

Keywords

Real-Time Attack Identification, Intrusion Detection Systems, Cybersecurity, Network Security, Adaptive Hyperparameter Optimization

References

• Rudner, M. (2013). Cyber-threats to critical national infrastructure: An intelligence challenge. International Journal of Intelligence and CounterIntelligence, 26(3), 453-481.
• Patel, A., Taghavi, M., Bakhtiyari, K., & Júnior, J. C. (2013). An intrusion detection and prevention system in cloud computing: A systematic review. Journal of Network and Computer Applications, 36(1), 25-41.
• Basil, N., Ahammad, S. H., & Elsayed, E. E. (2024). Enhancing wireless subscriber performance through AODV routing protocol in simulated mobile Ad-hoc networks. Engineering Applications, 3(1), 16-26.
• Kothamali, P. R., & Banik, S. (2022). Limitations of signature-based threat detection. Revista de Inteligencia Artificial en Medicina, 13(1), 381-391.
• Olateju, O. O., Okon, S. U., Igwenagu, U. T. I., Salami, A. A., Oladoyinbo, T. O., & Olaniyi, O. O. (2024). Combating the challenges of false positives in AI-driven anomaly detection systems and enhancing data security in the cloud. Asian Journal of Research in Computer Science, 17(6), 264-292.
• Zuech, R., Khoshgoftaar, T. M., & Wald, R. (2015). Intrusion detection and big heterogeneous data: A survey. Journal of Big Data, 2, 1-41.
• Ferdous, J., Islam, R., Mahboubi, A., & Islam, M. Z. (2023). A review of state-of-the-art malware attack trends and defense mechanisms. IEEE Access, 11, 121118–121141.
• Anwar, S., Mohamad Zain, J., Zolkipli, M. F., Inayat, Z., Khan, S., Anthony, B., & Chang, V. (2017). From intrusion detection to an intrusion response system: Fundamentals, requirements, and future directions. Algorithms, 10(2), 39.
• Raza, S., Wallgren, L., & Voigt, T. (2013). SVELTE: Real-time intrusion detection in the Internet of Things. Ad Hoc Networks, 11(8), 2661-2674.
• Ali, S., Rehman, S. U., Imran, A., Adeem, G., Iqbal, Z., & Kim, K. I. (2022). Comparative evaluation of AI-based techniques for zero-day attacks detection. Electronics, 11(23), 3934.
• Shyalika, C., Wickramarachchi, R., & Sheth, A. P. (2024). A comprehensive survey on rare event prediction. ACM Computing Surveys, 57(3), 1-39.
• Nayak, A. K., Reimers, A., Feamster, N., & Clark, R. (2009, August). Resonance: Dynamic access control for enterprise networks. In Proceedings of the 1st ACM workshop on Research on Enterprise Networking (pp. 11-18). ACM.
• Zheng, Y., Li, Z., Xu, X., & Zhao, Q. (2022). Dynamic defenses in cyber security: Techniques, methods and challenges. Digital Communications and Networks, 8(4), 422-435.
• İncekara, Ç. Ö. (2023). Industrial internet of things (IIoT) in energy sector. Advanced Engineering Science, 3, 21-30.
• Mema, B., Basholli, F., & Hyka, D. (2024). Learning transformation and virtual interaction through ChatGPT in Albanian higher education. Advanced Engineering Science, 4, 130-140.
• Yang, L., & Shami, A. (2020). On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415, 295-316.
• Keskin, S., & Sevli, O. (2024). Machine learning based classification for spam detection. Sakarya University Journal of Science, 28(2), 270-282.
• Hao, P., Liu, H., Feng, S., Wang, G., Zhang, R., & Wang, B. (2023). A high-dimensional optimization method combining projection correlation-based Kriging and multimodal parallel computing. Structural and Multidisciplinary Optimization, 66(1), 18.
• Vibhute, A. D., Patil, C. H., Mane, A. V., & Kale, K. V. (2024). Towards detection of network anomalies using machine learning algorithms on the NSL-KDD benchmark datasets. Procedia Computer Science, 233, 960-969.
• Barach, J. (2024, December). Enhancing intrusion detection with CNN attention using NSL-KDD dataset. In 2024 Artificial Intelligence for Business (AIxB) (pp. 15-20). IEEE.
• Abdullah, H. S. A. (2024). A comparison of several intrusion detection methods using the NSL-KDD dataset. Wasit Journal of Computer and Mathematics Science, 3(2), 32-41.
• Zakariah, M., AlQahtani, S. A., Alawwad, A. M., & Alotaibi, A. A. (2023). Intrusion detection system with customized machine learning techniques for NSL-KDD dataset. Computers, Materials & Continua, 77(3), 4025-4054.
• Türk, F. (2023). Analysis of intrusion detection systems in UNSW-NB15 and NSL-KDD datasets with machine learning algorithms. Bitlis Eren University Journal of Science, 12(2), 465-477.
• Rastogi, S., Shrotriya, A., Singh, M. K., & Potukuchi, R. V. (2022). An analysis of intrusion detection classification using supervised machine learning algorithms on NSL-KDD dataset. Journal of Computing Research and Innovation, 7(1), 124-137.
• Ravipati, R. D., & Abualkibash, M. (2019). Intrusion detection system classification using different machine learning algorithms on KDD-99 and NSL-KDD datasets: A review paper. International Journal of Computer Science & Information Technology (IJCSIT), 11(3), 65-80.
• Shrivas, A. K., & Dewangan, A. K. (2014). An ensemble model for classification of attacks with feature selection based on KDD99 and NSL-KDD data set. International Journal of computer applications, 99(15), 8-13.
• Tavallaee, M., Bagheri, E., Lu, W., & Ghorbani, A. A. (2009). A detailed analysis of the KDD CUP 99 data set. 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), 1–6.
• Mao, Y., Li, Y., Teng, F., Sabonchi, A. K., Azarafza, M., & Zhang, M. (2024). Utilizing hybrid machine learning and soft computing techniques for landslide susceptibility mapping in a Drainage Basin. Water, 16(3), 380.
• Zela, K., & Saliaj, L. (2023). Forecasting through neural networks: Bitcoin price prediction. Engineering Applications, 2(3), 218-224.
• James, G., Witten, D., Hastie, T., Tibshirani, R., & Taylor, J. (2023). Tree-based methods. In an Introduction to Statistical Learning: With Applications in Python (pp. 331–366). Springer International Publishing.
• Liu, W., Fan, H., & Xia, M. (2022). Credit scoring based on tree-enhanced gradient boosting decision trees. Expert Systems with Applications, 189, 116034.
• Tangirala, S. (2020). Evaluating the impact of GINI index and information gain on classification using decision tree classifier algorithm. International Journal of Advanced Computer Science and Applications, 11(2), 612-619.
• Karabatak, M. (2015). A new classifier for breast cancer detection based on Naïve Bayesian. Measurement, 72, 32-36.
• Budholiya, K., Shrivastava, S. K., & Sharma, V. (2022). An optimized XGBoost based diagnostic system for effective prediction of heart disease. Journal of King Saud University-Computer and Information Sciences, 34(7), 4514-4523.
• Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W. (2023). Cross validation for model selection: a review with examples from ecology. Ecological Monographs, 93(1), e1557.
• Mema, B., & Basholli, F. (2023). Internet of Things in the development of future businesses in Albania. Advanced Engineering Science, 3, 196-205
• Lazrek, G., Chetioui, K., Balboul, Y., & Mazer, S. (2024). An RFE/Ridge-ml/dl based anomaly intrusion detection approach for securing IoMT system. Results in Engineering, 23, 102659.
• Sinap, V. (2024). Comparative analysis of machine learning techniques for credit card fraud detection: Dealing with imbalanced datasets. Turkish Journal of Engineering, 8(2), 196-208.
• Polater, S. N., & Sevli, O. (2024). Deep learning based classification for alzheimer's disease detection using MRI images. Turkish Journal of Engineering, 8(4), 729-740.
• Sıngh, S., Kumar, K., & Kumar, B. (2024). Analysis of feature extraction techniques for sentiment analysis of tweets. Turkish Journal of Engineering, 8(4), 741-753.
• Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(2012), 281–305.
• Vo, Q., Ea, P., Salem, O., & Mehaoua, A. (2024, October). Detecting network anomalies in NetFlow traffic with machine learning algorithms. In 2024 IEEE 49th Conference on Local Computer Networks (LCN) (pp. 1-8). IEEE.
• Al-Tarawneh, M. A., Al-irr, O., Al-Maaitah, K. S., Kanj, H., & Aly, W. H. F. (2024). Enhancing fake news detection with word embedding: A machine learning and deep learning approach. Computers, 13(9), 239.
• Corona, I., Giacinto, G., & Roli, F. (2013). Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues. Information Sciences, 239, 201-225.
• Bolívar, A., García, V., Alejo, R., Florencia-Juárez, R., & Sánchez, J. S. (2024). Data-centric solutions for addressing big data veracity with class imbalance, high dimensionality, and class overlapping. Applied Sciences, 14(13), 5845.
• Khraisat, A., & Alazab, A. (2021). A critical review of intrusion detection systems in the internet of things: Techniques, deployment strategy, validation strategy, attacks, public datasets and challenges. Cybersecurity, 4, 1-27.
• Basholli, F., Mema, B., & Basholli, A. (2024). Training of information technology personnel through simulations for protection against cyber attacks. Engineering Applications, 3(1), 45-58.
• Mohammed, Y. R., Basil, N., Bayat, O., & Hamid, A. (2020). A new novel optimization techniques implemented on the AVR control system using MATLAB-SIMULINK. International Journal of Advanced Science and Technology, 29(5), 4515-4521.
• Marhoon, H. M., Ibrahim, A. R., & Basil, N. (2021). Enhancement of electro hydraulic position servo control system utilising ant lion optimiser. International Journal of Nonlinear Analysis and Applications, 12(2), 2453-2461.
• Mohamadwasel, N. B., & Kurnaz, S. (2021). Implementation of the parallel robot using FOPID with fuzzy type-2 in use social spider optimization algorithm. Applied Nanoscience, 13, 1389–1399.
• Basil, N., Marhoon, H. M., & Mohammed, A. F. (2024). Evaluation of a 3-DOF helicopter dynamic control model using FOPID controller-based three optimization algorithms. International Journal of Information Technology, 1-10.
• Basil, N., Sabbar, B. M., Marhoon, H. M., Mohammed, A. F., & Ma'arif, A. (2024). Systematic review of unmanned aerial vehicles control: Challenges, solutions, and meta-heuristic optimization. International Journal of Robotics & Control Systems, 4(4), 1794-1818.