Optimizing a Slow Python CSV Job Step by Step, with Measured Timings

By 0xmote · Jun 29, 2026 · 12 min read · Updated Sep 24, 2026 Intermediate

Key takeaways

A representative slow CSV job, optimized step by step with timings measured on one machine. The biggest win came from removing work, not from NumPy, Cython, or more cores.

About this walkthrough

“Python is too slow” usually means “this particular Python code does far more work than it needs to”. This article follows a representative batch job — a CSV of values where each row goes through an expensive calculation — and optimizes it step by step. It is a teaching reconstruction of a very common shape of problem, not a report from a specific production system.

Every timing in this article was measured by me on a single Windows machine (CPython 3.11, NumPy 1.26, 16 logical cores) with time.perf_counter. Absolute numbers will differ on your hardware; the ratios and, more importantly, the reasons behind them are the point. Where I did not measure something (Cython, Numba), I say so and give no number.

The starting point

import csv

def process_file(filename):
    with open(filename, newline="") as f:
        return [calculate(row) for row in csv.DictReader(f)]

def calculate(row):
    total = 0
    for i in range(1000):
        for j in range(100):
            total += float(row["value"]) * i * j
    return total

Each row triggers 100,000 inner iterations, and each iteration parses the same string with float() again. On my machine, 200 rows took 2.61 s, about 13 ms per row. That sounds tolerable until you multiply: at that rate a million rows is roughly 3.6 hours of single-core time.

Extrapolating from a small sample like this is a reasonable first estimate, as long as you remember it assumes the per-row cost is constant. If the calculation depends on data size (a lookup in a growing list, for example), the real run can be much worse than the extrapolation.

Profile before changing anything

python -m cProfile -s tottime process_data.py

For a 50-row run, the relevant part of the output was:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       50    0.966    0.019    0.966    0.019 prof.py:2(calculate)
       50    0.000    0.000    0.001    0.000 csv.py:107(__next__)

Two things to read from this. First, practically all time is inside calculate; CSV parsing is noise, so rewriting the file reading would have been wasted effort. Second, notice what is missing: the 5 million float() calls do not appear at all. cProfile records function calls, and calls to built-in types like float are not recorded as separate entries, so their cost is folded into calculate’s tottime. cProfile tells you which function is hot, not which line. For line-level detail, line_profiler (@profile plus kernprof -l -v) or a sampling profiler like py-spy are the right tools.

To save stats for later inspection:

python -m cProfile -o profile.stats process_data.py

import pstats
pstats.Stats("profile.stats").sort_stats("cumulative").print_stats(10)

Remove repeated work: hoist the invariant

row["value"] does not change inside the loops, so parse it once:

def calculate_hoisted(row):
    v = float(row["value"])
    total = 0.0
    for i in range(1000):
        for j in range(100):
            total += v * i * j
    return total

Measured: 4.7 ms per row, down from 13 ms — about 2.8x from moving one line. Nothing clever happened; the interpreter simply stopped doing 99,999 redundant dictionary lookups and string-to-float conversions per row. CPython does not do this kind of loop-invariant code motion for you, because in general it cannot prove that row["value"] or float will not change between iterations.

Remove the loop entirely: look at the math

Once the parsing is out, the loop is visibly just:

sum over i, j of v * i * j  =  v * (0 + 1 + ... + 999) * (0 + 1 + ... + 99)
                            =  v * 499500 * 4950

The constant does not depend on the row at all:

K = sum(range(1000)) * sum(range(100))

def calculate_closed(row):
    return float(row["value"]) * K

Now all 2,000 rows took 0.35 ms in total, in plain Python with no dependencies. Against the original 13 ms per row, that is a speedup of several tens of thousands — not because Python got faster, but because the program now does 2,000 multiplications instead of 200 million.

This is the step I think gets skipped most often. When I have seen optimization efforts go sideways, it has usually been in this order: someone sees a slow nested loop, reaches for multiprocessing or Cython, gets a respectable 4x–20x, and ships it — while the loop was recomputing something constant or doing an O(n²) search that a dict or a sort would have made O(n log n). The compiled version then has to be maintained forever, and the real fix was a few lines of arithmetic. Before any tool, I now ask: what is this loop computing, and does it need to be recomputed per iteration?

Your real calculation will rarely collapse into a closed form this neatly, but the questions generalize: Is anything inside the loop invariant? Is the same value computed for many rows (cache it)? Is there a search that a set or dict would make constant-time?

Vectorize with NumPy

With the per-row work reduced to one multiplication, the remaining cost is Python overhead per row. NumPy removes it by loading the column into one typed array and doing the multiplication in a compiled loop:

import numpy as np

def process_file_numpy(filename):
    values = np.loadtxt(filename, delimiter=",", skiprows=1, usecols=1)
    return values * K

Measured: 5.9 ms for 2,000 rows including loading the file. That is slower than the pure-Python closed form above (0.35 ms), and the comparison is not fair to either side: the NumPy number includes parsing the CSV, the Python one did not. The honest conclusion is that for 2,000 rows, file I/O dominates and vectorization barely matters. For a million already-loaded values, values * K took about 1.2 ms on my machine, which is where NumPy earns its keep. For heavier CSVs, pandas.read_csv (C parser) is usually faster than np.loadtxt, and pyarrow-based readers faster still.

Two pitfalls when moving to NumPy:

Results change in the last digits. The closed form and the original loop agreed only to a relative difference of about 4.5e-15 — the loop accumulates rounding error over 100,000 additions, the closed form does not. == comparisons fail; np.allclose(original, optimized, rtol=1e-12) passed. Decide the tolerance before optimizing so you are not tempted to loosen it until tests pass.
Accidental Python loops over arrays. Iterating an ndarray element by element in Python is slower than iterating a list, because each element is boxed into a NumPy scalar. See NumPy Basics for the details, including dtype overflow when your data is integer.

Output building: a CPython detail that can flip

The original pipeline also wrote results with repeated string concatenation:

output = ""
for r in results:
    output += f"{r}\n"

versus

output = "\n".join(str(r) for r in results) + "\n"

I measured 100,000 lines:

Variant	Time
`+=` inside a function	about 27 ms
`"\n".join(...)`	about 1.2 ms
`+=` at module level (global variable)	about 6,800 ms

The last row is the interesting one. CPython has an optimization that lets s += x resize a string in place when nothing else references it, which makes the loop roughly linear inside a function. At module level, the global namespace dict holds a second reference, the optimization cannot apply, and each += copies the whole string — quadratic time. The same code, moved out of a function during a refactor or run in a notebook cell, went from tens of milliseconds to seven seconds. join (or writing lines directly to the file with writelines) does not depend on an interpreter implementation detail, so prefer it.

Multiprocessing: measured to be slower here

from multiprocessing import Pool
import numpy as np

def process_chunk(values):
    return values * K

if __name__ == "__main__":        # required with the spawn start method
    values = np.random.default_rng(0).uniform(0, 100, 1_000_000)
    with Pool(4) as pool:
        results = np.concatenate(pool.map(process_chunk, np.array_split(values, 4)))

For one million values, the vectorized single-process version took about 1.2 ms; the Pool(4) version took about 290 ms — roughly 250x slower. The work per chunk was tiny, and the cost was all overhead: starting four processes, pickling each chunk, sending it through a pipe, pickling results back. On Windows and macOS, the default spawn start method also starts a fresh interpreter per worker and re-imports your main module, which is why the if __name__ == "__main__": guard is mandatory there — without it each worker tries to create its own pool.

Multiprocessing pays off when each task does substantial CPU-bound Python work (hundreds of milliseconds or more) and the inputs and outputs are small relative to that work. It is also the right answer when the GIL is the bottleneck: threads do not speed up CPU-bound pure-Python code in standard CPython builds, though they are fine for I/O-bound work, and NumPy releases the GIL inside many of its operations. Free-threaded CPython builds (3.13+) change part of this picture, but they are still opt-in.

Cython and Numba: when the loop really cannot go away

If, after removing redundant work, you still have a numeric loop that cannot be vectorized — data-dependent branches, a recurrence where each step depends on the previous one — compiling it is the next step. I did not benchmark these for this article, so there are no numbers here, only what each involves.

Cython, with typed variables:

# calculate.pyx
cimport cython

@cython.boundscheck(False)
@cython.wraparound(False)
def calculate_cython(double value):
    cdef Py_ssize_t i, j
    cdef double total = 0.0
    for i in range(1000):
        for j in range(100):
            total += value * i * j
    return total

The cdef types are what make this fast; an untyped .pyx file compiles but runs close to Python speed. Note Py_ssize_t rather than long: long is 32 bits on Windows, which matters once loop counters get large. The cost is a build step (cythonize, a compiler toolchain on every platform you ship to) and code that is harder for the rest of the team to change. Also note that boundscheck(False) has no effect here — this function does not index any arrays; it matters for typed memoryview access.

Numba compiles a decorated Python function at first call:

from numba import njit

@njit
def calculate_numba(value):
    total = 0.0
    for i in range(1000):
        for j in range(100):
            total += value * i * j
    return total

No build step, but a compile delay on the first call, and only a subset of Python and NumPy is supported inside @njit functions. For this specific function, either tool would still lose to the closed form in section 4.

What each step bought, measured

Step	Change	Measured result
0	Original nested loop	13 ms per row
1	Hoist `float(row["value"])` out of the loop	4.7 ms per row
2	Replace the loop with its closed form	0.35 ms for all 2,000 rows
3	NumPy, including CSV load	5.9 ms for 2,000 rows (load-dominated)
4	`Pool(4)` over 1M values	about 290 ms vs 1.2 ms single process

The order matters as much as the techniques: measure, remove redundant work, fix the algorithm, vectorize, and only then consider compiled code or parallelism — measuring again after each step, because as step 4 shows, “more cores” can easily make things worse.