Kaggle grandmasters have a saying: "Data beats algorithms." The features you feed your model matter far more than which algorithm you choose. A well-engineered feature can boost a logistic regression past an untuned neural network. Here are the techniques that actually move the needle.
Why Feature Engineering Still Matters in the Deep Learning Era
Deep learning models can learn features automatically from raw data — but only when you have enormous datasets and compute. For most real-world tabular problems (fraud detection, churn prediction, demand forecasting), you have thousands or hundreds of thousands of rows, not billions. In this regime, manual feature engineering is still one of the highest-ROI activities you can do.
Even in NLP and computer vision, preprocessing choices — tokenisation strategy, image normalisation, data augmentation — are a form of feature engineering that consistently impacts final performance.
1. Handle Missing Values Strategically
Never drop rows with missing values as a default. Missing data itself is often a signal. A missing income field on a loan application may indicate the applicant chose not to disclose — which is predictive.
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer
df = pd.read_csv('data.csv')
# Strategy 1: Add a "was missing" indicator BEFORE imputing
df['income_was_missing'] = df['income'].isna().astype(int)
# Strategy 2: Median imputation (robust to outliers)
imputer = SimpleImputer(strategy='median')
df['income'] = imputer.fit_transform(df[['income']])
# Strategy 3: KNN imputation (uses similar rows to fill)
knn_imputer = KNNImputer(n_neighbors=5)
df_imputed = pd.DataFrame(
knn_imputer.fit_transform(df),
columns=df.columns
)Pro tip: The "was_missing" binary indicator is often one of the most predictive features in datasets with informative missingness. Always add it before imputing.
2. Extract Rich Features from Datetime Columns
A raw timestamp is nearly useless to a model. But broken into components, it becomes a rich source of signal — especially for behaviour prediction and demand forecasting.
df['timestamp'] = pd.to_datetime(df['timestamp']) # Decompose into components df['hour'] = df['timestamp'].dt.hour df['day_of_week'] = df['timestamp'].dt.dayofweek # 0=Mon, 6=Sun df['month'] = df['timestamp'].dt.month df['is_weekend'] = (df['day_of_week'] >= 5).astype(int) df['is_holiday'] = df['timestamp'].isin(holiday_list).astype(int) # Cyclical encoding (so hour 23 is "close" to hour 0) df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24) df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24) df['dow_sin'] = np.sin(2 * np.pi * df['day_of_week'] / 7) df['dow_cos'] = np.cos(2 * np.pi * df['day_of_week'] / 7) # Time since a reference event df['days_since_signup'] = (df['timestamp'] - df['signup_date']).dt.days
Cyclical encoding is important: if you give a model raw hour values, it will think hour 23 (11 PM) is far from hour 0 (midnight), when they are actually adjacent. Sine/cosine encoding places them correctly in continuous space.
3. Encode Categorical Variables Correctly
Choosing the wrong encoding strategy is one of the most common mistakes. Here is a practical guide to when to use each method:
| Encoding | Use when | Watch out for |
|---|---|---|
| One-Hot Encoding | Low cardinality (<15 categories) | Curse of dimensionality with many categories |
| Label Encoding | Ordinal categories (Low/Med/High) | Tree models only — implies order to linear models |
| Target Encoding | High cardinality (city, user_id) | Data leakage if not done within CV folds |
| Frequency Encoding | High cardinality, no target info | Loses ordering of rare vs common |
| Binary Encoding | Medium-high cardinality | Less interpretable than OHE |
import category_encoders as ce from sklearn.model_selection import KFold # Target encoding safely (within CV folds to prevent leakage) encoder = ce.TargetEncoder(cols=['city', 'product_category']) encoder.fit(X_train, y_train) X_train_enc = encoder.transform(X_train) X_test_enc = encoder.transform(X_test) # uses training stats only
4. Create Interaction and Ratio Features
Linear models and even tree models sometimes miss relationships between features. Explicitly creating interaction terms and ratios can surface these patterns.
# Ratio features
df['income_per_dependent'] = df['annual_income'] / (df['num_dependents'] + 1)
df['debt_to_income'] = df['total_debt'] / (df['annual_income'] + 1)
df['revenue_per_employee'] = df['revenue'] / (df['employees'] + 1)
# Interaction terms (multiply two features)
df['age_x_experience'] = df['age'] * df['years_experience']
# Polynomial features (for linear models)
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False, interaction_only=False)
X_poly = poly.fit_transform(X[['age', 'income', 'credit_score']])
# Aggregate features (group-level statistics)
df['mean_spend_by_city'] = df.groupby('city')['spend'].transform('mean')
df['count_orders_by_user'] = df.groupby('user_id')['order_id'].transform('count')5. Scale Features — But Only When It Matters
Not every model needs feature scaling, but getting this wrong can severely degrade certain algorithms.
Needs scaling
- • Linear/Logistic Regression
- • SVM (especially RBF kernel)
- • KNN and K-Means
- • Neural Networks
- • PCA and other decompositions
Does NOT need scaling
- • Decision Trees
- • Random Forest
- • XGBoost / LightGBM
- • Naive Bayes
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler # StandardScaler: zero mean, unit variance — use for most cases scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # NEVER fit on test set! # RobustScaler: uses median & IQR — better when you have outliers from sklearn.preprocessing import RobustScaler scaler = RobustScaler() # MinMaxScaler: scales to [0,1] — use for neural nets, image pixels scaler = MinMaxScaler()
Critical rule: Always fit your scaler on the training set only, then apply the same transformation to the test set. Fitting on the full dataset causes data leakage and inflated evaluation scores.
6. Remove Useless Features
More features is not always better. Irrelevant or redundant features add noise, slow training, and can hurt generalisation. Feature selection is the other side of feature engineering.
from sklearn.feature_selection import SelectKBest, f_classif, RFE from sklearn.ensemble import RandomForestClassifier import pandas as pd # Method 1: Correlation filter (remove highly correlated features) corr_matrix = pd.DataFrame(X_train).corr().abs() upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)) to_drop = [col for col in upper.columns if any(upper[col] > 0.95)] X_train.drop(columns=to_drop, inplace=True) # Method 2: Feature importance from Random Forest rf = RandomForestClassifier(n_estimators=100, random_state=42) rf.fit(X_train, y_train) importances = pd.Series(rf.feature_importances_, index=feature_names) top_features = importances.nlargest(20).index.tolist() # Method 3: Recursive Feature Elimination rfe = RFE(estimator=rf, n_features_to_select=15) rfe.fit(X_train, y_train) selected = X_train.columns[rfe.support_].tolist()
The Feature Engineering Mindset
The best feature engineers are domain experts first and ML practitioners second. Ask yourself: "What would a human expert look at to make this decision?" Then encode that reasoning as a feature. The most powerful features are almost always derived from domain insight, not automated search.