Parquet vs CSV for market data after work: stop shipping giant text files

Why this matters after work

You searched for parquet vs csv market data. After a full day of work, you do not need a 400-page manual—you need a why-first path that fits one evening: understand the angle (when bars and panels should leave CSV behind), run a tiny example, and know what not to claim.

This hub article is educational only. Not investment advice. It will not pick stocks, size positions, or promise edge.

The evening angle

Tool focus: parquet.

• CSV is fine for tiny samples and human edits
• Parquet wins on typed columns, compression, and partial reads
• market panels grow fast—measure file size and load time once
• keep a CSV export only for the slice you need to eyeball

Toy code (safe, synthetic)

import time
from pathlib import Path
import numpy as np
import pandas as pd
rng = np.random.default_rng(2)
n = 50_000
df = pd.DataFrame({
    "date": pd.date_range("2020-01-01", periods=n, freq="min"),
    "ticker": np.where(rng.random(n) > 0.5, "AAPL", "MSFT"),
    "close": 100 + rng.normal(0, 1, n).cumsum()/100,
})
csv_p, pq_p = Path("/tmp/toy_bars.csv"), Path("/tmp/toy_bars.parquet")
df.to_csv(csv_p, index=False); df.to_parquet(pq_p, index=False)
print("csv_mb", round(csv_p.stat().st_size/1e6, 3),
      "parquet_mb", round(pq_p.stat().st_size/1e6, 3))
t0 = time.perf_counter(); pd.read_csv(csv_p); t1 = time.perf_counter()
t2 = time.perf_counter(); pd.read_parquet(pq_p); t3 = time.perf_counter()
print("csv_s", round(t1-t0, 3), "parquet_s", round(t3-t2, 3))

Run it in a throwaway venv. If an import is missing, install only what you need for tonight—do not boil the ocean.

Practice loop (35–50 minutes)

1. Recreate the toy snippet from memory (or retype it—muscle memory beats copy-paste).
2. Swap synthetic numbers for a CSV you already have (public sample or your own research extract).
3. Write three bullet notes: what worked, what broke, what you will not automate yet.
4. Commit the script or notebook with a clear message—future you will thank present you.

Pitfalls

• Treating a tutorial metric as a tradable signal.
• Leaking future information into features or labels.
• Pasting API keys, customer data, or proprietary code into random tools.
• Upgrading five libraries at once when one change would isolate the failure.

What success looks like tonight

You can explain when bars and panels should leave CSV behind in two sentences, show one chart or printout, and list one risk. That is enough progress for an after-work session.

Next step

Tomorrow, connect this tool to the adjacent skill (data hygiene, validation, or a simple dashboard)—still educational, still not investment advice.

Educational only. Not investment advice.