Skip to content

Data Wrangling·Lesson 32 of 67·intermediate

Time series basics

Parse timestamps properly, index by them, resample to a coarser grain, and use rolling windows to watch a model's latency and quality drift.


In this lesson · 18 min

  • →Parse timestamps and set a DatetimeIndex
  • →Slice by date with a string
  • →Pull parts of a timestamp out with the .dt accessor
  • →Resample to a coarser grain with a chosen aggregation
  • →Smooth a noisy metric with a rolling window, and measure change with shift and diff

Almost everything an AI system emits is a time series: requests per minute, latency, cost, error rate, the fraction of answers a judge accepts. pandas has a whole layer for this, and it removes most of the date arithmetic you would otherwise get wrong.

Timestamps and the index

PythonPython not started

Parsing real timestamps means being explicit, and letting bad rows fail visibly:

PythonPython not started

Setting the timestamp as the index unlocks the time-aware behaviour:

PythonPython not started

The .dt accessor

On an ordinary timestamp column, .dt reaches the parts:

PythonPython not started

Resampling

resample is a group-by where the groups are time buckets. It is how a stream of individual requests becomes a chart you can read:

PythonPython not started

The frequency strings you will use: min, h, D, W, ME (month end). Upsampling to a finer grain leaves gaps, which you then fill deliberately:

PythonPython not started

Rolling windows

A rolling window smooths a noisy metric so a trend becomes visible:

PythonPython not started

The first five rolling values are NaN because a six-wide window needs six observations. min_periods relaxes that when you would rather have an early, noisier number:

PythonPython not started

shift and diff

shift moves a column along the index, which is how you compare each point with the one before:

PythonPython not started

Try it yourself

Exercise 1

Resample the request log to hourly buckets in hourly, with requests (count) and median_latency columns.

Your turnPython not started
Exercise 2

Add a rolling_3d column — the three-day rolling mean of cost — and a change column holding the day-on-day difference.

Your turnPython not started

Check yourself

Why does a rolling(6) mean start with five NaN values?

Check yourself

You add a feature using .shift(-1) to a model trained on time-ordered data. What have you done?

Recap

  • Parse with an explicit format and utc=True; errors="coerce" keeps bad rows visible.
  • A DatetimeIndex lets you slice with a string: df.loc["2026-03-01"].
  • .dt reaches the parts of a timestamp column — hour, date, day_name, dayofweek.
  • resample("1h").agg(...) is a group-by over time buckets; ffill or interpolate fills the gaps upsampling leaves.
  • rolling(n) smooths, and needs n observations unless you set min_periods.
  • shift(1) and diff() compare a point with the one before. shift(-1) looks into the future — and leaks it into any model you train.