Hands-on Data Analysis with Python | Pandas Workflows

By 0xmote · Jun 30, 2026 · 12 min read · Updated Sep 27, 2026 Intermediate Tutorial

Key takeaways

Practical Python data analysis: load data, EDA with plots, groupby and pivot tables, time series with resample and rolling, correlation heatmaps, and insight checklists.

Introduction

Hands-on data analysis is the process of understanding data and extracting meaning. With pandas, the code for most steps is short; the difficulty is that almost every call makes a silent decision for you. mean() skips missing values, groupby drops rows whose key is NaN, rolling(7) counts rows rather than days, and corr() measures only straight-line relationships. This post walks through a typical load → explore → group → time series → report flow and points out, at each step, which of those defaults can change the answer.

Load and explore

First pass

import pandas as pd
import numpy as np
df = pd.read_csv('sales_data.csv')
print(f"Shape: {df.shape}")
print(f"\nColumn info:")
df.info()   # prints directly and returns None, so don't wrap it in print()
print(f"\nSummary statistics:")
print(df.describe())
print(f"\nFirst 5 rows:")
print(df.head())
print(f"\nMissing values per column:")
print(df.isnull().sum())

Running these five calls, in this order, before writing a single line of actual analysis is the habit worth building: df.info() surfaces dtype surprises (a numeric column silently read as object because one row somewhere has a stray non-numeric value), and df.isnull().sum() tells you up front which columns need a decision — drop, impute, or leave as-is — before that decision gets made implicitly and invisibly by whatever the first groupby or .mean() call happens to do with missing data.

When info() shows a numeric-looking column as object, pd.to_numeric(df['price'], errors='coerce') converts it and turns the bad values into NaN; comparing isnull().sum() before and after tells you how many rows were affected, and df.loc[converted.isna() & df['price'].notna(), 'price'] shows what those values actually were (often "1,200", "N/A" or a currency symbol). Dates deserve the same treatment at load time: pd.read_csv('sales_data.csv', parse_dates=['date']) avoids carrying date strings through the whole analysis. The describe() output is also worth reading line by line rather than skimming: a min of -1 in an age column or a max that is 100 times the 75th percentile is usually a data-entry error, not an insight.

Exploratory data analysis (EDA)

Distributions

import matplotlib.pyplot as plt
df['age'].hist(bins=20)
plt.title('Age Distribution')
plt.xlabel('Age')
plt.ylabel('Frequency')
plt.show()
df.boxplot(column='salary', by='department')
plt.title('Salary Distribution by Department')
plt.show()

A histogram and a boxplot answer different questions about the same column, which is why EDA reaches for both rather than picking one. The histogram shows the overall shape of the distribution — is it roughly normal, skewed, bimodal — which a boxplot’s five-number summary (min, quartiles, max, outliers) can hide entirely; two very differently-shaped distributions can produce nearly identical boxplots if their quartiles happen to line up. The boxplot earns its place for the by='department' comparison, though, since laying five or six overlapping histograms side by side is far harder to read at a glance than five or six boxplots next to each other.

Correlation

import seaborn as sns
corr_matrix = df[['age', 'salary', 'experience']].corr()
print(corr_matrix)
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation heatmap')
plt.show()

.corr() computes Pearson correlation by default, which specifically measures linear relationships — two columns with a strong curved (quadratic, exponential) relationship can show a correlation near zero here despite being tightly related, simply because Pearson’s coefficient has no way to represent a non-linear pattern. Always pair a correlation heatmap like this with an actual scatter plot of any pair you’re about to draw conclusions from; the heatmap tells you where to look, not what’s actually there. Pearson is also sensitive to outliers: a handful of extreme salaries can produce a correlation of 0.6 in data that is otherwise flat. df.corr(method='spearman') uses ranks instead of values, which makes it robust to outliers and able to detect any monotonic relationship, curved or not; a large gap between the Pearson and Spearman matrices is itself a hint that outliers or non-linearity are involved. In the example age and experience will almost certainly be strongly correlated with each other, which is the other reason to be careful: when both correlate with salary, the heatmap cannot tell you which one actually matters.

Group analysis

Aggregations

dept_avg = df.groupby('department')['salary'].mean()
print(dept_avg)
result = df.groupby('department').agg({
    'salary': ['mean', 'min', 'max'],
    'age': 'mean',
    'name': 'count'
})
print(result)
pivot = df.pivot_table(
    values='salary',
    index='department',
    columns='gender',
    aggfunc='mean'
)
print(pivot)

pivot_table and the two-step groupby().agg() above it are doing conceptually the same split-apply-combine work, but pivot_table reshapes the combine step differently — instead of one row per group, it spreads a categorical column (gender here) out across the column axis, which is exactly the shape a spreadsheet-style cross-tabulation needs. Reach for groupby when you want one summary row per group; reach for pivot_table the moment you need to compare two categorical dimensions against each other at a glance, since that’s a shape groupby alone can’t produce without an extra .unstack() call.

Two details in this block are easy to misread. 'name': 'count' counts non-null names, not rows, so a group with missing names looks smaller than it is; .size() counts rows regardless of nulls. And groupby drops rows whose key is NaN by default, so employees with no department simply vanish from the result without any warning. Pass dropna=False to keep them as their own group, which is usually what you want while exploring. With a MultiIndex result like result above, columns become tuples such as ('salary', 'mean'); named aggregation, df.groupby('department').agg(avg_salary=('salary', 'mean'), headcount=('name', 'size')), produces flat, readable column names instead.

Time series

Resampling and rolling windows

df['date'] = pd.to_datetime(df['date'])
df = df.set_index('date')
monthly = df.resample('ME').sum(numeric_only=True)   # 'ME' = month end ('M' is deprecated since pandas 2.2)
df['ma7'] = df['sales'].rolling(window=7).mean()
df['ma30'] = df['sales'].rolling(window=30).mean()
plt.figure(figsize=(12, 6))
plt.plot(df.index, df['sales'], label='Daily sales', alpha=0.5)
plt.plot(df.index, df['ma7'], label='7-day moving average')
plt.plot(df.index, df['ma30'], label='30-day moving average')
plt.legend()
plt.title('Sales Trend')
plt.show()

resample and rolling both aggregate over time, but for different purposes, and mixing them up produces confusing results. resample('M') changes the frequency of the data — it collapses daily rows into one row per calendar month, permanently reducing the number of rows. rolling(window=7) keeps the original row count exactly the same, computing a sliding average that includes the 7 most recent rows at every single existing timestamp — which is why ma7 and ma30 can be plotted directly alongside the original daily sales series on the same x-axis, while monthly (a resample result) would need its own separate plot with far fewer points.

rolling(window=7) means “the last 7 rows”, which equals “the last 7 days” only when there is exactly one row per day. Sales data usually has gaps (no row on days without sales) and sometimes duplicates (several transactions per day), and then the 7-row average spans an unpredictable number of days. With a DatetimeIndex, rolling('7D') uses a time-based window instead and handles gaps correctly; alternatively, resample to daily first with df['sales'].resample('D').sum() so missing days become explicit zeros. Also note that rolling(7) produces NaN for the first six rows (min_periods defaults to the window size), and that the index must be sorted: rolling on an unsorted time index gives meaningless results, and a time-based window raises ValueError: index must be monotonic.

Practical example

Customer analytics

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
customers = pd.read_csv('customers.csv')
print("=== Basic Stats ===")
print(f"Total customers: {len(customers)}")
print(f"Average age: {customers['age'].mean():.1f}")
print(f"Average purchase amount: ${customers['purchase_amount'].mean():,.0f}")
customers['age_group'] = pd.cut(
    customers['age'],
    bins=[0, 20, 30, 40, 50, 100],
    labels=['Teens', '20s', '30s', '40s', '50+']
)
age_analysis = customers.groupby('age_group', observed=True).agg({
    'purchase_amount': ['mean', 'sum', 'count']
})
print("\n=== Analysis by Age Group ===")
print(age_analysis)
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
axes[0, 0].hist(customers['age'], bins=20, edgecolor='black')
axes[0, 0].set_title('Age Distribution')
axes[0, 1].hist(customers['purchase_amount'], bins=30, edgecolor='black')
axes[0, 1].set_title('Purchase Amount Distribution')
age_group_avg = customers.groupby('age_group', observed=True)['purchase_amount'].mean()
axes[1, 0].bar(age_group_avg.index.astype(str), age_group_avg.values)
axes[1, 0].set_title('Average Purchase Amount by Age Group')
axes[1, 0].tick_params(axis='x', rotation=45)
numeric_cols = customers[['age', 'purchase_amount', 'visit_count']]
sns.heatmap(numeric_cols.corr(), annot=True, ax=axes[1, 1])
axes[1, 1].set_title('Correlation')
plt.tight_layout()
plt.savefig('customer_analysis.png', dpi=300)
plt.show()
print("\n=== Insights ===")
print(f"Age group with the most customers: {customers['age_group'].value_counts().idxmax()}")
print(f"Age group with the highest average purchase amount: {age_group_avg.idxmax()}")

pd.cut has two defaults that change results. Intervals are closed on the right, so the bins are (0, 20], (20, 30] and so on: a 20-year-old lands in “Teens”, a 30-year-old in ”20s”, and an age of exactly 0 falls outside every bin and becomes NaN. Pass right=False for [20, 30)-style bins, which is what labels like ”20s” usually mean, and include_lowest=True if the first edge should be included. The result is a categorical column, and groupby on a categorical includes every category by default, even empty ones; pandas 2.1+ warns that this default will change, which is why the code passes observed=True explicitly.

The two “insight” lines at the end are also where analyses most often go wrong. “The most customers” and “the highest average purchase” are different questions with different answers (a small group of high spenders versus a large group of typical ones), and an average over a group with 12 customers is much noisier than one over 1,200. Printing the count next to every mean, as age_analysis does, is the minimum before presenting a group comparison to anyone.

This walkthrough is a template worth internalizing rather than a one-off example: basic stats first (sanity-check the numbers before trusting them), then a derived grouping column (age_group, via pd.cut binning a continuous variable into categories), then aggregation by that group, then a 2x2 grid of complementary visualizations (a distribution, a comparison, and a correlation, all in one figure via plt.subplots), and finally a plain-English insights summary. The 2, 2 subplot grid pattern here — one figure, four related views — is a genuinely efficient way to present a full mini-analysis in a single image rather than four separate plots someone has to mentally stitch together.

A repeatable analysis workflow

Analysis workflow

# 1. Understand the data
# - Column meanings
# - dtypes
# - Missing and extreme values
# 2. Hypotheses
# - "Do older customers spend more?"
# - "Are weekends higher revenue?"
# 3. Validate
# - Plots
# - Stats
# - Correlations (with care)
# 4. Insights
# - Business meaning
# - Action items

Going deeper

Outlier flags and group summaries (Pandas)

Mark IQR-based outliers on numeric columns, then summarize by group.

import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
df = pd.DataFrame({
    "region": rng.choice(["A", "B", "C"], size=400),
    "revenue": rng.normal(100, 25, size=400),
})
def iqr_bounds(s: pd.Series, k: float = 1.5):
    q1, q3 = s.quantile([0.25, 0.75])
    iqr = q3 - q1
    lo, hi = q1 - k * iqr, q3 + k * iqr
    return lo, hi
lo, hi = iqr_bounds(df['revenue'])
df['outlier'] = (df['revenue'] < lo) | (df['revenue'] > hi)
summary = df.groupby("region")['revenue'].agg(["count", "mean", "std", "median"])
print(summary)
print("Outlier rate:", df['outlier'].mean())

This IQR method flags a value as an outlier if it falls more than k times the interquartile range beyond the first or third quartile — the classic Tukey’s-fences rule, and the same threshold a standard boxplot uses to decide which points to draw as individual dots beyond the whiskers. On this synthetic normal data, only a small fraction of points (well under 1% for a true normal distribution) should be flagged. On real revenue data, which is usually right-skewed, the same rule flags many legitimate large customers as “outliers” and almost never flags anything on the low side. Computing the bounds per group (df.groupby('region')['revenue'].transform(...)) or on log(revenue) is often more meaningful than one global threshold. And flag, don’t delete: an outlier is a question to investigate, sometimes a data error and sometimes the most important row in the table.

Common mistakes

Computing means and correlations before dropping or imputing missing values.
Using rolling / diff on unsorted time indexes.
Claiming causality from plots without statistical tests.

Caveats

IQR is a weak distributional rule—combine with domain thresholds.
Multiple comparisons may require p-value adjustments.

In production

Note random seeds and data snapshot versions in notebooks.
Document metric definitions (numerator/denominator) with results.

Alternatives

Tool	Role
pandas	Table transforms and aggregates
SQL	Push down large aggregations
Spark	Distributed batch jobs