An Unsupervised Machine Learning Approach for Dynamic Anomaly Detection and Risk Defense in Cloud Servers
Keywords:
Cloud Security, Intrusion Detection, Unsupervised Machine Learning, Hybrid Learning Models, Cloud FirewallsAbstract
This paper presents a hybrid unsupervised machine learning model for real-time anomaly detection and dynamic risk defence in the cloud environment. Traditional security mechanisms such as Intrusion Detection Systems (IDS) and fixed firewall rules are often not sufficient to deal with the emerging threats in cloud computing, especially zero-day exploits and polymorphic malware that evade signature-based detection. The proposed system combines Isolation Forest (IF), Local Outlier Factor (LOF) and Density Based Spatial Clustering of Applications with Noise (DBSCAN) to identify both point and cluster anomalies from unlabelled cloud traffic. Integration with AWS Web Application Firewall (WAF) allows it to update its rules automatically and independently mitigate the threats detected. The model was trained and validated on 2.3 million AWS EC2 traffic records, in addition to the CICIDS2017 dataset which was split into a 70-30 training-validation split. The computation environment consisted of AWS EC2 t2.xlarge (4vCPUs, 16GB RAM) instances of Python 3.8, scikit-learn 0.24.2, MongoDB 4.4, and TensorFlow 2.6. Experiments showed a detection accuracy of 92 per cent with a false positive rate of four per cent. The comparative analysis demonstrated better adaptability and less manual intervention in comparison to traditional IDS (Snort, Suricata: 88 -90% accuracy, 7-9% false positives) and standalone ML models (IF: 87-90% accuracy, LOF: 86 -90% accuracy, DBSCAN: 84 -90% accuracy). The system was able to detect and block port scanning, DDoS, brute-force and data exfiltration patterns in real time with latency of less than 50ms. The reduction of false-positive by 43-56% led to 150-200 alerts per day being reduced in enterprise settings. The hybrid unsupervised model makes the cloud more resilient with adaptive defence without the need for labelled data. Removing manual firewall updates will save 15-20 hours per week for security teams. Future directions are encrypted traffic analysis based on metadata-based behavioural profiling, largescale distributed data processing (10M+ requests/minute), and multi-cloud integration between AWS, Azure, and GCP.
References
N. W. C. Lasantha, R. Abeysekara, and M. Maduranga, “A Novel Framework for Real-Time IP Reputation Validation Using Artificial Intelligence,” Int. J. Wirel. Microw. Technol., vol. 14, no. 2, pp. 1–16, 2024, doi: 10.5815/ijwmt.2024.02.01.
N. W. C. Lasantha, R. Abeysekara, and M. W. P. Maduranga, “Defending Cloud Web Applications using Machine Learning-Driven Triple Validation of IP Reputation by Integrating Security Operation Center,” Int. J. Comput. Netw. Inf. Secur., vol. 24, no. 1, pp. 1–15, 2024.
C. Griffy-Brown, D. Lazarikos, and M. Chun, “Agile Business Growth and Cyber Risk,” in Proc. 2018 IEEE Technol. Eng. Manag. Conf., San Jose, CA, USA, Jun. 2018, pp. 1–6, doi: 10.1109/TEMSCON.2018.8488397.
I. Kunz, A. Schneider, and C. Banse, “A Continuous Risk Assessment Methodology for Cloud Infrastructures,” in Proc. 2022 22nd IEEE Int. Symp. Clust. Cloud Internet Comput., Heidelberg, Germany, Nov. 2022, pp. 1042–1051, doi: 10.48550/arXiv.2206.07323.
S. Farahmandian and D. Hoang, “Policy-based Interaction Model for Detection and Prediction of Cloud Security Breaches,” J. Technol. Dev. Econ., vol. 9, pp. 92–116, 2021, doi: 10.18080/JTDE.V9N2.364.
Y. Gao, Y. Liu, Y. Jin, J. Chen, and H. Wu, “A Novel Semi-Supervised Learning Approach for Network Intrusion Detection on Cloud-Based Robotic System,” IEEE Access, vol. 6, pp. 50927–50938, 2018, doi: 10.1109/ACCESS.2018.2868171.
I. Lafram, N. Berbiche, and J. El Alami, “Artificial Neural Networks Optimized with Unsupervised Clustering for IDS Classification,” in Proc. 2019 1st Int. Conf. Smart Syst. Data Sci., Rabat, Morocco, Oct. 2019, pp. 1–7, doi: 10.1109/ICSSD47982.2019.9002827.
H. Choi, M. Kim, G. Lee, and W. Kim, “Unsupervised learning approach for network intrusion detection system using autoencoders,” J. Supercomput., vol. 76, no. 4, pp. 1–25, Apr. 2020, doi: 10.1007/s11227-019-02805-w.
G. K. Bada, W. K. Nabare, and D. Quansah, “Comparative Analysis of the Performance of Network Intrusion Detection Systems: Snort, Suricata and Bro Intrusion Detection Systems in Perspective,” Int. J. Comput. Appl., vol. 177, no. 3, pp. 1–6, 2020, doi: 10.5120/ijca2020920513.
A. Sahu, Z. Mao, K. Davis, and A. Goulart, “Data Processing and Model Selection for Machine Learningbased Network Intrusion Detection,” in Proc. 2020 IEEE Int. Work. Tech. Comm. Commun. Qual. Reliab., Arlington, VA, USA, May 2020, pp. 1–6, doi: 10.1109/CQR47547.2020.9101394.
Z. Chiba, N. Abghour, K. Moussaid, A. El Omri, and M. Rida, “Newest collaborative and hybrid network intrusion detection framework based on suricata and isolation forest algorithm,” in Proc. 4th Int. Conf. Smart City Appl., Casablanca, Morocco, Oct. 2019, pp. 1–6, doi: 10.1145/3368756.3369061.
A. Gupta and L. Sharma, “Performance Evaluation of Snort and Suricata Intrusion Detection Systems on Ubuntu Server,” in Emerging Technologies in Data Mining and Information Security, Singapore: Springer, 2019, pp. 811–821, doi: 10.1007/978-3-030-29407-6 58.
M. Hasin, B. Madoˇs, J. Palˇsa, and A. Janitor, “Analysis of Network Traffic in CLOUD Environment,” in Proc. 2020 18th Int. Conf. Emerg. eLearning Technol. Appl., Stary Smokovec, Slovakia, Nov. 2020, pp. 130–135, doi: 10.1109/ICETA51985.2020.9379153.
IB. Youssef, M. Nada, and B. Regragui, “Behavioural analysis approach for IDS based on attack pattern and risk assessment in cloud computing,” Int. J. Inf. Comput. Secur., vol. 11, no. 4, pp. 315–331, 2019, doi: 10.1504/IJICS.2019.10013935.
H. Iqbal, A. Singh, and M. Shahzad, “Characterizing the Availability and Latency in AWS Network From the Perspective of Tenants,” IEEE/ACM Trans. Netw., vol. 30, no. 4, pp. 1554–1568, Aug. 2022, doi: 10.1109/tnet.2022.3148701.
S. Garg, K. Kaur, N. Kumar, S. Batra, and M. Obaidat, “HyClass: Hybrid Classification Model for Anomaly Detection in Cloud Environment,” in Proc. 2018 IEEE Int. Conf. Commun., Kansas City, MO, USA, May 2018, pp. 1–7, doi: 10.1109/ICC.2018.8422481.
K. Ghanshala, P. Mishra, R. Joshi, and S. Sharma, “BNID: A Behavior-based Network Intrusion Detection at Network-Layer in Cloud Environment,” in Proc. 2018 First Int. Conf. Secur. Cyber Comput. Commun., Jalandhar, India, Dec. 2018, pp. 100–105, doi: 10.1109/ICSCCC.2018.8703265.
A. Sirisha, K. Chaitanya, K. V. S. S. R. Krishna, and S. Kanumalli, “Intrusion Detection Models Using Supervised and Unsupervised Algorithms – A Comparative Estimation,” Int. J. Saf. Secur. Eng., vol. 11, no. 1, pp. 61–68, 2021, doi: 10.18280/IJSSE.110106.
T. Ahmad, M. Anwar, and M. Haque, “Machine Learning Techniques for Intrusion Detection,” in Intelligent Systems and Applications in Multi-Agent Systems, IGI Global, 2020, pp. 47–65, doi: 10.4018/978-1-7998-2242-4.ch003.
F. G. Portela, F. Almenares Mendoza, and L. C. Benavides, “Evaluation of the performance of supervised and unsupervised Machine learning techniques for intrusion detection,” in Proc. 2019 IEEE Int. Conf. Appl. Sci. Adv. Technol., Quezon City, Philippines, Nov. 2019, pp. 1–8, doi: 10.1109/iCASAT48251.2019.9069538.
M. Leon, T. Markovic, and S. Punnekkat, “Comparative Evaluation of Machine Learning Algorithms for Network Intrusion Detection and Attack Classification,” in Proc. 2022 Int. Jt. Conf. Neural Networks, Padua, Italy, Jul. 2022, pp. 1–8, doi: 10.1109/IJCNN55064.2022.9892293.
A. Howe and M. Papa, “Feature Engineering in Machine Learning-Based Intrusion Detection Systems for OT Networks,” in Proc. 2023 IEEE Int. Conf. Smart Comput., Nashville, TN, USA, Jun. 2023, pp. 361–366, doi: 10.1109/SMARTCOMP58114.2023.00086.
A. S. Guptha, H. Murali, and S. T, “A Comparative Analysis of Security Services in Major Cloud Service Providers,” in Proc. 2021 5th Int. Conf. Intell. Comput. Control Syst., Madurai, India, May 2021, pp. 129–136, doi: 10.1109/ICICCS51141.2021.9432189.
N. J. Mitchell and K. Zunnurhain, “Google cloud platform security,” in Proc. 4th ACM/IEEE Symp. Edge Comput., Arlington, VA, USA, Nov. 2019, pp. 514–515, doi: 10.1145/3318216.3363371.
H. Gu et al., “DIAVA: A Traffic-Based Framework for Detection of SQL Injection Attacks and Vulnerability Analysis of Leaked Data,” IEEE Trans. Reliab., vol. 69, no. 1, pp. 188–202, Mar. 2020, doi: 10.1109/TR.2019.2925415.
G. H. S. Carvalho, I. Woungang, and A. Anpalagan, “Cloud Firewall Under Bursty and Correlated Data Traffic: A Theoretical Analysis,” IEEE Trans. Cloud Comput., vol. 10, no. 3, pp. 1620–1633, Jul.–Sep. 2022, doi: 10.1109/TCC.2020.3000674.
B. Ramamurthy, “Securing Business IT on the Cloud,” in Cloud Technology: Concepts, Methodologies, Tools, and Applications, IGI Global, 2014, pp. 2022–2032, doi: 10.4018/978-1-4666-5788-5.CH006.
D. Appelt, C. Nguyen, A. Panichella, and L. Briand, “A Machine-Learning-Driven Evolutionary Approach for Testing Web Application Firewalls,” IEEE Trans. Reliab., vol. 67, no. 3, pp. 733–757, Sep. 2018, doi: 10.1109/TR.2018.2805763.
G. Tiwari and R. Jain, “Detecting and Classifying Incoming Traffic in a Secure Cloud Computing Environment Using Machine Learning and Deep Learning System,” in Proc. 2022 IEEE 7th Int. Conf. Smart Cloud, Newark, NJ, USA, Oct. 2022, pp. 16–21, doi: 10.1109/smartcloud55982.2022.00010.
M. Hossen, T. Ahmad, and M. A. R. Putra, “Traffic Classification with Machine Learning for Enhancing Cloud Security,” in Proc. 2023 Intell. Methods, Syst. Appl., Giza, Egypt, Jul. 2023, pp. 86–91, doi: 10.1109/IMSA58542.2023.10217598.
S. Sharma, “Advancements in Machine Learning for Intrusion Detection in Cloud Environments,” Int. J. Sci. Res. Eng. Manag., vol. 7, no. 4, pp. 1–8, 2023, doi: 10.55041/ijsrem24430.
O. Olasehinde, O. V Johnson, and O. Olayemi, “Evaluation Of Selected Meta Learning Algorithms for The Prediction Improvement Of Network Intrusion Detection System,” in Proc. 2020 Int. Conf. Math. Comput. Eng. Comput. Sci., Ayobo, Nigeria, Mar. 2020, pp. 1–7, doi: 10.1109/ICMCECS47690.2020.240893.
S. Rastegari, P. Hingston, and C. Lam, “Evolving statistical rulesets for network intrusion detection,” Appl. Soft Comput., vol. 33, pp. 348–359, Aug. 2015, doi: 10.1016/j.asoc.2015.04.041.
E. Anthi, L. Williams, A. Javed, and P. Burnap, “Hardening machine learning denial of service (DoS) defences against adversarial attacks in IoT smart home networks,” Comput. Secur., vol. 108, p. 102352, Sep. 2021, doi: 10.1016/J.COSE.2021.102352.
M. J. Rani and D. Singh, “Machine Learning Algorithm for Intrusion Detection: Performance Evaluation and Comparative Analysis,” in Proc. 2023 7th Int. Conf. I-SMAC (IoT Soc. Mobile, Anal. Cloud), Kirtipur, Nepal, Oct. 2023, pp. 779–784, doi: 10.1109/I-SMAC58438.2023.10290491.
H. H. Yi and Z. M. Aye, “Machine Learning Based DoS Traffic Analysis on the Testbed Environment,” in Proc. 2023 IEEE Conf. Comput. Appl., Yangon, Myanmar, Feb. 2023, pp. 429–434, doi: 10.1109/ICCA51723.2023.10181878.
U. M. Thanthrige, J. Samarabandu, and X. Wang, “Machine learning techniques for intrusion detection on public dataset,” in Proc. 2016 IEEE Can. Conf. Electr. Comput. Eng., Vancouver, BC, Canada, May 2016, pp. 1–4, doi: 10.1109/CCECE.2016.7726677.
R. A. Elsayed, R. A. Hamada, M. Hammoudeh, M. Abdalla, and S. Elsaid, “A Hierarchical Deep Learning-Based Intrusion Detection Architecture for Clustered Internet of Things,” J. Sens. Actuator Networks, vol. 12, no. 1, p. 3, Jan. 2023, doi: 10.3390/jsan12010003.
A. H. Azizan et al., “A Machine Learning Approach for Improving the Performance of Network Intrusion Detection Systems,” Ann. Emerg. Technol. Comput., vol. 5, no. 5, pp. 201–212, 2021, doi: 10.33166/AETIC.2021.05.025.
N. Keegan, S.-Y. Ji, A. Chaudhary, C. Concolato, B. Yu, and D. Jeong, “A survey of cloud-based network intrusion detection analysis,” Human-centric Comput. Inf. Sci., vol. 6, no. 19, pp. 1–16, Dec. 2016, doi: 10.1186/s13673-016-0076-z.
M. Kuwano, M. Okuma, S. Okada, and T. Mitsunaga, “ATT&CK Behavior Forecasting based on Collaborative Filtering and Graph Databases,” in Proc. 2022 IEEE Int. Conf. Comput., Cebu City, Philippines, Nov. 2022, pp. 191–197, doi: 10.1109/ICOCO56118.2022.10032036.
D. Hermawan, N. G. Novianto, and D. Octavianto, “Development of Open Source-based Threat Hunting Platform,” in Proc. 2021 2nd Int. Conf. Artif. Intell. Data Sci., Ipoh, Malaysia, Sep. 2021, pp. 1–6, doi: 10.1109/AiDAS53897.2021.9574308.
K. Zhang, X. Kang, and S. Li, “Isolation Forest for Anomaly Detection in Hyperspectral Images,” in Proc. IGARSS 2019 – 2019 IEEE Int. Geosci. Remote Sens. Symp., Yokohama, Japan, Jul. 2019, pp. 437–440, doi: 10.1109/IGARSS.2019.8899812.
S. Hariri, M. Kind, and R. Brunner, “Extended Isolation Forest,” IEEE Trans. Knowl. Data Eng., vol. 33, no. 4, pp. 1479–1489, Apr. 2021, doi: 10.1109/TKDE.2019.2947676.
Z. Cheng, C. Zou, and J. Dong, “Outlier detection using isolation forest and local outlier factor,” in Proc. Conf. Res. Adapt. Converg. Syst., Chongqing, China, Sep. 2019, pp. 161–168, doi: 10.1145/3338840.3355641
H. Wang, B. Zhou, J. Zhang, and R. Cheng, “A Novel Density Peaks Clustering Algorithm Based on Local Reachability Density,” Int. J. Comput. Intell. Syst., vol. 13, no. 1, pp. 690–697, Jun. 2020, doi: 10.2991/ijcis.d.200603.001.
S. Xu, H. Liu, L. Duan, and W. Wu, “An Improved LOF Outlier Detection Algorithm,” in Proc. 2021 IEEE Int. Conf. Artif. Intell. Comput. Appl., Kunming, China, Jun. 2021, pp. 113–117, doi: 10.1109/ICAICA52286.2021.9498181.
W. Lai, M. Zhou, F. Hu, K. Bian, and Q. Song, “A New DBSCAN Parameters Determination Method Based on Improved MVO,” IEEE Access, vol. 7, pp. 104085–104095, Jul. 2019, doi: 10.1109/ACCESS.2019.2931334.
H. Zhang, Y. Zhang, P. Lu, and C. Wang, “Research on network intrusion detection based on SMOTEENN and improved CatBoost algorithm,” Proc. SPIE, vol. 12800, pp. 128001U-1–128001U-6, Nov. 2023, doi: 10.1117/12.3003918.
Z. Lyu and Z. Pan, “HAD-IDC: A Hybrid Framework for Data Anomaly Detection based on Isolation, Density, and Clustering,” in Proc. 2022 2nd Int. Conf. Intell. Technol., Hubli, India, Aug. 2022, pp. 1–6, doi: 10.1109/CONIT55038.2022.9848201.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 NW Chanaka Lasantha, MWP Maduranga, Ruvan Abesekara, Sabyasachi Bhattacharyya
This work is licensed under a Creative Commons Attribution 4.0 International License.
