Polars

If you list the tools that made Python feel fast lately, a pattern jumps out. The dataframe library everyone is switching to, Polars, is written in Rust. The linter that replaced flake8, Ruff, is Rust. The packaging tool eating pip and virtualenv, uv, is Rust. Pydantic v2 rewrote its core in Rust; Hugging Face’s tokenizers is Rust; orjson is Rust; even cryptography moved its guts to Rust years ago. Python isn’t going anywhere — it’s still the interface. But the engine underneath the performance-critical parts is quietly being rewritten in Rust. This post looks at that shift from two angles I’ve actually measured: Polars vs Pandas (the library you use) and PyO3 vs Cython (how you’d build such a library yourself). All numbers below I ran on one 8-core Linux x86-64 box. ...

When performance bottlenecks started affecting my production data pipeline, I decided to test whether Polars could deliver on its performance promises. This is what I learned from migrating a real production workload from Pandas to Polars. The Workload The application was a data aggregation service running as a Kubernetes pod with the following constraints: Resources: 2 CPUs, 3 GB RAM Execution frequency: Every 2-2.5 minutes Data volume: 5,000-7,000 rows × 100-150 columns per run Operations: Multiple database calls, API requests, DataFrame merges, arithmetic operations (additions, multiplications), and group-by aggregations Web server: FastAPI with Uvicorn handling production traffic All operations were properly vectorized — no row-by-row iteration. The pipeline combined data from various sources into a single DataFrame, transformed it, and output the results. ...