Microsoft Malware Detection

Project Overview

This big data classification project focuses on predicting whether a computer node is at risk of malware infection. By auditing OS versions, active setups, hardware specs, and security indicators, this model provides warning metrics for proactive patch deployments.

Key Features & Objectives

• High-Volume Telemetry Processing: Processes millions of machine state logs with high feature dimensionalities.
• Malware Propensity Auditing: Forecasts binary risk percentages for client machines.
• Class Distribution Optimization: Adjusts model weight balances to resolve target dataset skews.

Technical Stack & Technologies

• Language: Python
• Classifiers: LightGBM, XGBoost, Random Forests
• Data Scaling: Scikit-learn, Pandas, NumPy
• Big Data Optimization: Parquet, memory-saving category datatypes

Core Techniques & Feature Tuning

• Category Dimensionality Reduction: Configured feature maps and frequency-encoding schemas to manage high-cardinality category parameters (e.g. system configurations, update build numbers).
• Gradient-Boosted Decision Trees (GBDT): Utilized LightGBM’s histogram-based split nodes to accelerate training on dense telemetry data.
• Feature Importance Auditing: Audited SHAP values and GINI feature importances to rank factors (such as Windows Defender active states or system build revisions) driving malware vulnerability risks.