Go

I have spent the last weeks building ursus — a dataframe library for Go modelled on Polars. Lazy execution with a query optimizer, Arrow memory layout, SIMD kernels, and streaming execution that spills to disk instead of falling over. df, err := ursus.ScanParquet("events.parquet"). Filter(ursus.Col("price").Gt(5)). GroupBy(ursus.Col("region")). Agg( ursus.Col("price").Sum().Alias("revenue"), ursus.Col("qty").Mean().Alias("avg_qty"), ). Sort(ursus.Desc(ursus.Col("revenue"))). Collect(ctx) The projection reaches the Parquet reader, the filter becomes a row-group predicate, and the group-by runs on every core. None of that shows up in the query. ...

This is Part 2 of a series. Read Part 1: Pandas vs Polars in Production - Performance Comparison for the background on the Polars migration. After migrating from Pandas to Polars, CPU performance improved dramatically—but a memory problem persisted. Despite extensive debugging, I couldn’t identify the root cause. So I made a pragmatic decision: architect around it. This is the story of splitting a monolithic Python application into a Go orchestration service with Python workers, not because I fully understood the problem, but because I needed production to be stable. ...