Data Preprocessing in Python: Missing Values, Outliers, Scaling and Encoding

By 0xmote · Jun 30, 2026 · 20 min read · Updated Sep 27, 2026 Intermediate Tutorial

Key takeaways

Python data preprocessing: handle missing data, IQR/Z-score outliers, Min-Max and StandardScaler, categorical encoding, and a full sklearn-style pipeline with Pandas.

Introduction

Data preprocessing accounts for a large share of real-world machine learning work. This isn’t a platitude — a model trained on data with unhandled nulls, unscaled features, or leaky outliers will often score well in a notebook and then fail quietly in production, because the training/serving skew shows up as a subtle accuracy drop rather than a crash. Most of the decisions below aren’t mechanical; they’re judgment calls that depend on why the data looks the way it does, and getting that reasoning wrong is a more common source of bad models than picking the wrong algorithm.

Missing values

Detecting nulls

Before deciding how to handle a null, it matters why it’s null. A salary field with 2% random gaps behaves very differently from an age field where every null belongs to users who skipped an optional survey question — the first is closer to random noise, the second is informative (its absence correlates with behavior), and blindly imputing over it can throw away a real signal. df.isnull().sum() only tells you the null count per column; it’s worth also checking whether nulls cluster in specific rows or correlate with another column before choosing a strategy.

import pandas as pd
import numpy as np
df = pd.DataFrame({
    'name': ['Alice', 'Beth', 'Carl', None],
    'age': [25, None, 28, 30],
    'salary': [3000, 4000, None, 5000]
})
print(df.isnull())
print(df.isnull().sum())  # nulls per column

Handling missing values

# Option 1: drop
df_dropped = df.dropna()           # rows with any null
df_dropped = df.dropna(axis=1)     # columns with any null
# Option 2: fill
df_filled = df.fillna(0)
df_filled = df.fillna(df.mean(numeric_only=True))
df_filled = df.ffill()  # forward fill
# Option 3: interpolate
df['age'] = df['age'].interpolate()

Dropping rows (dropna()) is the safest option when nulls are rare and random, but it silently shrinks your dataset and can bias it if the missingness isn’t random — dropping every row where salary is null will systematically remove, say, contractors who were never assigned a salary field, skewing whatever the model later learns about that group. Filling with the column mean is a common default for numeric features because it doesn’t change the mean of the column, but it does artificially shrink the variance (every filled value collapses onto a single point), which understates uncertainty for models sensitive to variance, like linear regression’s confidence intervals. interpolate() is a reasonable choice specifically for values with a natural order — like a time series or, more weakly, age — because it estimates a null from its neighbors rather than from the whole column’s distribution; using it on a truly unordered categorical column would produce meaningless results.

Outliers

IQR rule

IQR (interquartile range) is the more robust of the two outlier rules shown here because it’s based on quantiles, not the mean/standard deviation — a handful of extreme values can’t drag the IQR bounds around the way they drag a mean. The 1.5 × multiplier is a long-standing convention (from Tukey’s box plot), not a law: for a feature you know is naturally skewed (like income or file size, which tend to have a long right tail), 1.5 will flag legitimate high values as outliers, and a looser multiplier like 3 is often more appropriate. The core pitfall with any outlier removal is applying it before you know whether the “outliers” are data errors or the actual, interesting cases (like fraud detection, where the outliers are the entire point) — always visualize before deleting rows based on a rule.

Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df_clean = df[
    (df['salary'] >= lower_bound) &
    (df['salary'] <= upper_bound)
]

One side effect of this filter is easy to miss: comparisons with NaN are always False, so every row whose salary is missing is dropped along with the outliers. If missing values should survive, keep them explicitly with | df['salary'].isna(), or handle missing values first as in section 1.

Z-score rule

from scipy import stats
# Rows with non-null salary: flag |z| < 3 as inliers
sub = df.dropna(subset=['salary'])
z = np.abs(stats.zscore(sub['salary']))
df_clean = sub[z < 3]

Z-score assumes the underlying distribution is roughly normal — the |z| < 3 threshold means “more than 3 standard deviations from the mean,” which is a meaningful cutoff only if standard deviation is a meaningful measure of spread for that column. On a skewed distribution (again, income-like data), the mean and standard deviation themselves get pulled by the same extreme values you’re trying to flag, which can make the Z-score rule both under- and over-aggressive depending on which tail you’re looking at. When you’re unsure which distribution shape you’re dealing with, IQR is the safer default; reach for Z-score specifically when you have reason to believe the feature is normally distributed.

Normalization and standardization

Min–Max scaling (0–1)

Scaling only matters for models that are sensitive to the magnitude of feature values — distance-based algorithms (KNN, k-means, SVM) and gradient-descent-trained models (linear/logistic regression, neural networks) will implicitly let a salary column (values in the thousands) dominate an age column (values under 100) unless both are brought onto comparable scales. Tree-based models (random forests, gradient boosting) split on thresholds per feature independently, so they’re invariant to monotonic rescaling and usually don’t need this step at all — scaling tree-model inputs is harmless but wasted effort.

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df['salary_normalized'] = scaler.fit_transform(df[['salary']])
print(df[['salary', 'salary_normalized']])

Standardization (mean 0, std 1)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df['salary_standardized'] = scaler.fit_transform(df[['salary']])
print(df[['salary', 'salary_standardized']])

Min-Max scaling and standardization solve the same problem differently, and the choice isn’t arbitrary. Min-Max is bounded to a fixed [0, 1] range, which neural networks with sigmoid/tanh activations often expect, but it’s sensitive to outliers — a single extreme value stretches the range so that most of your “normal” data gets squeezed into a tiny sliver near 0. Standardization (z-score scaling) has no fixed bounds, so a single extreme value cannot squeeze the rest of the data into one corner of a fixed interval, and it’s the conventional choice for linear models, PCA, and anything assuming roughly-normal input. It is not outlier-proof, though: the mean and standard deviation it uses are themselves pulled by extreme values. When a feature has heavy outliers you intend to keep, RobustScaler (which centers on the median and scales by the IQR) is the scaler designed for that case. A subtle but important pitfall with both: always call .fit() on the training set only, then .transform() (not .fit_transform()) on the test/validation set — fitting the scaler on the full dataset lets statistics from the test set leak into training, silently inflating your reported accuracy.

Categorical encoding

Label encoding

Label encoding assigns each category an arbitrary integer (Seoul → 0, Busan → 1, …), which is a trap for any model that treats numbers as ordered — a linear model will interpret 2 > 1 > 0 as a real ordering between cities that doesn’t exist, learning nonsense relationships. Integer encoding is the right idea for genuinely ordinal categories (low/medium/high, star ratings), where the order is real and worth preserving as a single numeric column instead of exploding into multiple binary columns, but LabelEncoder is the wrong tool for it. It assigns codes in sorted order, so high, low, medium become 0, 1, 2, an order that has nothing to do with the meaning; and scikit-learn documents it as an encoder for target labels (y), not for input features. For features, use OrdinalEncoder(categories=[['low', 'medium', 'high']]), which takes the order you specify, or a plain mapping with df['level'].map({'low': 0, 'medium': 1, 'high': 2}).

from sklearn.preprocessing import LabelEncoder
df = pd.DataFrame({
    'city': ['Seoul', 'Busan', 'Seoul', 'Daegu', 'Busan']
})
encoder = LabelEncoder()
df['city_encoded'] = encoder.fit_transform(df['city'])
print(df)

One-hot encoding

# Pandas
df_encoded = pd.get_dummies(df, columns=['city'])
print(df_encoded)
# scikit-learn
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(sparse_output=False)
encoded = encoder.fit_transform(df[['city']])

One-hot encoding is the safer default for nominal (unordered) categories because it avoids implying a false ordering, but it has its own cost: a column with hundreds of distinct categories (like a user_id or product_sku) blows up into hundreds of sparse binary columns, which can hurt both memory and model training time — this is the “curse of dimensionality” showing up concretely. For high-cardinality categoricals, target encoding or hashing tricks are usually a better fit than either of the two approaches shown here.

pd.get_dummies and OneHotEncoder also behave differently on new data, which matters as soon as a model is deployed. get_dummies creates columns from whatever categories appear in the DataFrame it is given, so a test set without Daegu produces one column fewer than the training set, and a new city in production produces a column the model has never seen; the model then fails with a feature-count or feature-name mismatch. A fitted OneHotEncoder(handle_unknown='ignore') remembers the training categories, always produces the same columns, and encodes an unseen category as all zeros. That is the reason to prefer the scikit-learn encoder in anything beyond a one-off analysis.

Feature engineering

Deriving features

This is where preprocessing shifts from “cleaning” to actively creating predictive signal, and it’s often the highest-leverage step in the whole pipeline. sales_ma7 (a 7-day moving average) smooths out day-to-day noise so a model can see the underlying trend rather than reacting to single-day spikes; sales_diff captures the rate of change rather than the absolute level, which matters for any target where “is this trending up or down” is more informative than “what’s the raw value right now.” The rolling and diff operations both introduce NaN values at the start of the series (a 7-day rolling average has no value for the first 6 rows) — a preprocessing pipeline that adds these features has to re-run the missing-value step afterward, which is a common ordering bug: engineering features and then forgetting they’ve reintroduced nulls downstream.

df = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=100),
    'sales': np.random.randint(100, 500, 100)
})
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.dayofweek
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['sales_ma7'] = df['sales'].rolling(window=7).mean()
df['sales_diff'] = df['sales'].diff()

End-to-end example

Full preprocessing pipeline

This function is written for readability, but it hides a real bug that’s worth calling out explicitly: it computes the IQR bounds and the scaler’s mean/std from the same DataFrame it’s about to filter and transform. In a real training pipeline, you’d fit the outlier bounds and the StandardScaler on the training split only, then apply those already-fitted bounds/scaler to the validation and test splits — otherwise information about your test set (its mean, its distribution) leaks into preprocessing decisions made “before” you’ve supposedly seen it, and your evaluation metrics become optimistic. sklearn.pipeline.Pipeline exists largely to enforce this fit/transform separation automatically instead of relying on discipline.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, LabelEncoder
def preprocess_data(df):
    """Example preprocessing pipeline."""
    # 1. Missing values
    numeric_cols = df.select_dtypes(include=[np.number]).columns
    df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].mean())
    categorical_cols = df.select_dtypes(include=['object']).columns
    df[categorical_cols] = df[categorical_cols].fillna('Unknown')
    # 2. Outliers (IQR per numeric column)
    for col in numeric_cols:
        Q1 = df[col].quantile(0.25)
        Q3 = df[col].quantile(0.75)
        IQR = Q3 - Q1
        lower = Q1 - 1.5 * IQR
        upper = Q3 + 1.5 * IQR
        df = df[(df[col] >= lower) & (df[col] <= upper)]
    # 3. Label encode categoricals
    for col in categorical_cols:
        le = LabelEncoder()
        df[f'{col}_encoded'] = le.fit_transform(df[col])
    # 4. Standardize numeric columns
    scaler = StandardScaler()
    df[numeric_cols] = scaler.fit_transform(df[numeric_cols])
    return df
raw_data = pd.read_csv('raw_data.csv')
clean_data = preprocess_data(raw_data)
clean_data.to_csv('clean_data.csv', index=False)

Two more pandas-level issues are hiding in this function. The first step assigns into the DataFrame that was passed in, so raw_data itself is modified by the call; starting with df = df.copy() keeps the caller’s data intact. And after df = df[...] in the outlier loop, df may be treated by pandas as a slice of the previous frame, so the later df[f'{col}_encoded'] = ... can raise SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. The .copy() after filtering (or pandas’ copy-on-write mode, the default behaviour planned for pandas 3.0) makes the intent explicit. The label-encoding step has the ordering problem described in section 4; in a real pipeline, ColumnTransformer with OneHotEncoder or OrdinalEncoder per column replaces it.

Preprocessing order of operations

Preprocessing checklist

# ✅ 1. Inspect
df.info()
df.describe()
df.isnull().sum()
# ✅ 2. Missing values
# Decide drop vs impute; use domain knowledge
# ✅ 3. Outliers
# Visualize; apply IQR or Z-score with care
# ✅ 4. Encoding
# Ordinal → OrdinalEncoder with explicit category order
# Nominal → one-hot
# ✅ 5. Scaling
# Distance-based models: usually required
# Tree models: often optional

Next in the series

Hands-on data analysis takes a cleaned dataset like the one produced here and works through exploration, grouping and time series.