AI-Driven Security Approaches for Detecting Botnet Attacks in IoT Environments

Abstract

The Internet of Things (IoT) continues to expand rapidly, but this growth also exposes networks to sophisticated botnet attacks such as Mirai and Gafgyt that compromise large numbers of devices. Intrusion Detection Systems (IDS) are critical for securing IoT environments. Yet, traditional models often fail to address the dual challenges of extreme class imbalance and diverse attack subtypes in real-world traffic. This study analyzes a large-scale IoT IDS dataset containing over 7 million network flow samples, where normal traffic constitutes only 8.54% of records, and attack traffic includes UDP, TCP, SYN, SCAN, and ACK floods. The dataset provides 23 engineered temporal and relational features, including entropy, mutual information, host-host statistics, jitter, and host-port behavior that capture subtle patterns of malicious activity. This research proposes a novel hybrid ensemble model that integrates cost-sensitive LightGBM, XGBoost, deep neural networks, and Isolation Forest anomaly detection, combining supervised learning with unsupervised detection of zero-day threats. Experimental evaluation demonstrates that the proposed model significantly outperforms baseline approaches such as logistic regression and random forests, achieving superior Precision-Recall AUC, F1-score, and Matthews Correlation Coefficient. These results highlight the potential of advanced AI techniques to enhance resilience against evolving botnet threats in IoT environments.