Datetime Selectors

We start our exploration of datetime columns in Polars by directly selecting them using the Polars selectors: select(cs.datetime(time_zone=“*“)). This enables us to effortlessly extract and manipulate datetime information from our dataset.

What are Polars selectors? Selectors in Polars provide an intuitive way to select columns from DataFrame or LazyFrame objects based on various criteria like column name, data type, or other properties. They enhance and unify the functionality offered by the col() expression, making column selection a breeze. What’s more, selectors even allow you to broadcast expressions over the selected columns, further expanding their versatility. With selectors, you have a powerful tool at your disposal to streamline column selection and manipulation in Polars.

To make use of this feature, ensure that you have Polars version 0.18.1 or above. If you’re using Poetry or Pipenv for dependency management, upgrading your dependencies to the latest versions will ensure you have access to all the new functionalities and enhancements.

Polars pl.date_range function

Let’s delve into the powerful date_range function provided by Polars. This function allows you to generate a sequence of dates within a specified range with incredible ease. By specifying the start and end dates, along with the desired frequency (such as daily, weekly, or monthly), you can quickly create a DataFrame that spans the entire range of dates.

The date_range function also provides flexibility in handling time zones. You can specify the time zone for the generated dates, ensuring consistency and accuracy in your time-related analyses across different region for a wide range of use cases, such as creating synthetic time-series data, filling gaps in existing data, or generating test datasets for time-related analyses.

Note that starting from previous versions of Polars, the date_range function now requires the eager=True parameter to be specified in order to expand the range over rows. This ensures that the date range is not consolidated into a single row if that is not the desired behavior.

Polars convert_time_zone() function

Datetime columns when any row contains time zone information, converts the column in TZ aware and ‘UTC’ as default, converting the existing timezones to UTC. If you want to maintain a custom tz for the column, you can use te schema / schema_override constructor arguments

Example: Select datetime columns matching their time zone and compute their offset with Polars Selectors

But let’s take it a step further and introduce an exciting new feature: generating row offsets based on datetime columns! First of all you would need a Polars version > 0.18.1. You should upgrade your poetry/pipenv dependencies.

poetry update polars # would pick up minor version updates too

And here you can find and example of how to select datetime columns and generate their row offset!

from datetime import date, datetime

import polars as pl
import polars.selectors as cs
import zoneinfo

import pytz

start = pl.datetime(2023, 9, 1)
stop = pl.datetime(2023, 12, 1)
df = pl.DataFrame(
    {"dt": pl.date_range(start=start, end=stop, interval="1mo", time_zone="UTC", eager=True)},
    schema={"dt": pl.Datetime("ns", "UTC")},
).with_columns(
    [
        pl.col("dt").dt.convert_time_zone("Europe/Berlin").suffix("_Berlin"),
        pl.col("dt").dt.convert_time_zone("Australia/Sydney").suffix("_Sydney"),
    ]
)

Select all datetime columns with timezone

df.select(cs.datetime(time_zone="*"))

Select the offsets that may differ by column but also by row as the daylight savings are not the same around the world.

df.select(cs.datetime(time_zone="*").dt.dst_offset())

Note how if any column metadata is provided, the timezone is infered from the first record, and the following rows are transported to that time zone.

Get ready for our next post, where we’ll unveil other datetime operations with Polars! From rolling windows to rounding dates, combining datetime values, and converting string columns seamlessly – we’ve got it all covered. Stay in the loop and discover the boundless possibilities of data manipulation with Polars in Python!