pandas 3.1 Is Out: select(), filter() and a Faster read_csv

pandas 3.1 adds DataFrame.select(), row filtering with filter() and parallel read_csv. What's new, the behaviour changes that can shift results, and how to upgrade safely.

Divya NairDivya NairAuthor18 November 20264 min read 1 views
pandas 3.1 Is Out: select(), filter() and a Faster read_csv
In this article▾
  1. DataFrame.select(): pick and compute columns in one step
  2. filter() finally filters rows
  3. The missing-value trap
  4. A faster read_csv()
  5. Changes that can quietly shift your results
  6. Deprecations to fix while you're in there
  7. How to upgrade without surprises

pandas 3.1.0 came out on 22 October 2026, followed by a 3.1.1 bug-fix release on 4 November. If the 3.0 upgrade this autumn felt like a chore, 3.1 is the friendlier release: two new methods that make everyday column and row selection read more like SQL, and a read_csv() that's noticeably faster on big files. It also brings a few quiet behaviour changes that can shift your results without an error, and a long list of deprecations. Here's what's worth knowing, with small examples you can paste into a notebook.

DataFrame.select(): pick and compute columns in one step

Until now, choosing some columns and adding a calculated one usually meant two steps: index the columns, then assign(). The 3.1.0 release notes introduce DataFrame.select(), which does both. Take a small orders table:

import pandas as pd

orders = pd.DataFrame({
    "order_id": [101, 102, 103, 104, 105],
    "city": ["Pune", "Delhi", "Pune", "Kochi", "Delhi"],
    "qty": [2, 1, 5, None, 3],
    "price": [450.0, 1200.0, 99.0, 300.0, 650.0],
})

orders.select(
    "order_id",
    "city",
    revenue=pd.col("qty") * pd.col("price"),
    revenue_k=pd.col("revenue") / 1000,
)

The result has four columns: order ID, city, revenue (900 for the first Pune order, 1,950 for the last Delhi one) and revenue in thousands. Notice that revenue_k refers to revenue, which was created earlier in the same call. As with assign(), later arguments can use earlier ones. Order 104, with a missing quantity, gets a missing revenue, which is what you want: no silent zero.

If you've used Polars, pd.col() will feel familiar. It's an expression that refers to a column by name, so you don't have to repeat orders["qty"] everywhere.

filter() finally filters rows

For years, DataFrame.filter() confusingly selected labels, not rows. In 3.1 it accepts a boolean condition through a new cond keyword, or as an expression:

orders.filter(pd.col("qty") > 2)              # orders 103 and 105
orders.filter(cond=orders["city"] == "Pune")    # orders 101 and 103
orders.filter(items=["order_id", "price"])      # the old label selection

Passing a list of booleans positionally now raises a UserWarning; use items= for labels and cond= for masks so the intent is obvious.

The missing-value trap

This is the analytical mistake to watch for. The release notes say missing values in the mask are treated as False by default, with na=True to keep them and na="raise" to raise. But whether you even have missing values in the mask depends on the column's dtype.

In the example above, qty is an ordinary float column, so order 104's quantity is NaN. Comparing NaN < 10 gives plain False, not missing, so na=True makes no difference and the row is dropped either way. Switch the column to the nullable Int64 dtype and the comparison returns <NA> for that row. Now na=True keeps order 104, and na="raise" stops with "The mask contains missing values." A two-line test on your own data will show which case you're in.

For pipelines where a missing value would mean bad data, na="raise" on a nullable column is a cheap data-quality check.

A faster read_csv()

Large CSVs are now read in parallel across CPU cores by default, when the file is local, uncompressed and at least 5 MB, and you haven't passed options that need the whole file parsed as one. The result is identical to a serial read. The thread count comes from a new mode.max_threads option, which defaults to the CPUs available, capped at 6. The release notes also say single-threaded reads are typically two to three times faster than before.

On a shared server or a small cloud notebook, you may want to limit it:

with pd.option_context("mode.max_threads", 1):
    df = pd.read_csv("large.csv")

Keep the speed-up in proportion, though. Faster parsing doesn't change the fact that pandas holds everything in memory. A Plotly benchmark rerun on 3 November with pandas 3.1.0 on a 4-vCPU, 16 GB machine found pandas ran out of memory on both join queries at 119 million rows, while DuckDB and Polars finished. Think of 3.1 as a quicker front door, not a bigger house.

Changes that can quietly shift your results

These don't raise errors, which is exactly why they're worth a look:

  • Numeric timestamps now default to microseconds. pd.to_datetime([...], unit="s") returns datetime64[us], not seconds or nanoseconds. Use as_unit() if a downstream system expects a specific resolution.

  • Timedelta.total_seconds() now includes nanoseconds. 999 nanoseconds used to report 0.0 seconds; it now reports 9.99e-07. Tiny, but it can break an equality check in a test.

  • read_csv(dtype="category") infers numeric categories. A column of 1, 1, 2 now gets integer categories [1, 2] instead of strings. If you join that column to another table where the codes are strings, the join can come back empty. Read as str first and convert if you need string categories.

  • Index.values is read-only. Code that mutated it in place will now fail.

Deprecations to fix while you're in there

  • inplace=True is deprecated on methods that never really worked in place, including drop, rename, reset_index, set_index, dropna, drop_duplicates, sort_values and query. Write df = df.reset_index() instead.

  • Parquet: engine="fastparquet" and engine="auto" are deprecated; use PyArrow or leave the argument out.

  • Excel: the xlrd and pyxlsb engines are deprecated in favour of calamine.

  • Joins: lsuffix/rsuffix in DataFrame.join() give way to suffixes.

pandas 3.1 also requires NumPy 2.0.2 or newer, so an old pinned NumPy 1.x will block the install.

How to upgrade without surprises

  1. Install the bug-fix release: pip install "pandas>=3.1.1".

  2. Turn the new deprecation warnings into errors while you test, so nothing slips past:

import warnings
import pandas as pd

warnings.simplefilter("error", pd.errors.Pandas4Warning)
  1. Run your notebooks or tests, fix each inplace and engine warning, then remove the filter.

  2. Compare a few key outputs (row counts, date columns, any category joins) before and after.

If you skipped 3.0, start with the six pandas 3.0 changes that break old code; 3.1 builds on top of them. And if select() and pd.col() make you curious about the other side, our comparison of Polars 2.0 for pandas users shows how far the two APIs have converged. When a script is still slow after the upgrade, the Python 3.15 sampling profiler will tell you where the time actually goes.

Divya Nair

Written by

Divya Nair

Keep reading

More from Divya Nair

More from Divya Nair

More in Guides & Tutorials

Have a story of your own?

Publisha is free to start. Write with AI that keeps your voice, and publish in a click.