Slope instability in open-cast coal mines remains a critical challenge affecting operational safety, economic efficiency, and environmental integrity. This study develops an enhanced Geo-Spatial Artificial Intelligence (GeoAI) framework integrating Deep Neural Networks (DNN) and CatBoost models to predict slope failure risks with higher precision. A comprehensive geospatial dataset comprising 300 samples from Indian mining regions was analyzed using parameters such as slope angle, curvature, rainfall intensity, flow accumulation, aspect, and distance to drainage. The DNN achieved 95.2% accuracy, 0.84 F1-score, and 0.96 ROC-AUC, performing competitively with the CatBoost model (96.0% accuracy, 0.83 F1-score, 0.97 ROC-AUC). To enhance real-time decision-making, a Landslide Risk Index (LRI) ranging from 0–100 was introduced, translating model probabilities into continuous risk levels categorized as low, moderate, high, and critical. This probabilistic framework enables dynamic early warning and risk prioritization in active mining zones. Feature analysis identified slope angle, rainfall intensity, and flow accumulation as the most influential factors. The study demonstrates that DNN-based GeoAI systems offer robust predictive capabilities, improved interpretability, and scalability over traditional ensemble models, contributing to safer and more intelligent slope monitoring and early warning mechanisms in geotechnical applications.