A Hybrid Deep Learning Framework for Malware Detection Using Metaheuristic Feature Selection and Explainable AI: A Comprehensive Literature Review

Abstract

Malware, including ransomware, trojans, rootkits, spyware, and advanced persistent threats (APTs), poses a growing challenge to modern computing systems. Traditional detection methods, such as signature- and heuristic-based approaches, struggle to detect polymorphic, metamorphic, and zero-day malware. Deep learning has emerged as a powerful solution due to its ability to automatically learn hierarchical features. However, key challenges remain: lack of model transparency, high-dimensional feature redundancy from multimodal analysis, and poor cross-dataset generalization. This paper presents a systematic literature review of state-of-the-art malware detection techniques published between 2020 and 2025, covering static, dynamic, visualization-based, and deep learning approaches (e.g., CNN, LSTM, BiLSTM, and hybrid models), along with metaheuristic feature selection and Explainable AI (XAI) methods such as SHAP, Grad-CAM, and LIME. Analysis of 35 studies identifies critical gaps, particularly the absence of integrated metaheuristic optimization and XAI-driven hybrid frameworks, motivating future research directions.