Lesson 3 - Differencing to Remove Trend

Welcome to Differencing to Remove Trend#

Lesson 2 confirmed formally that Cyclepath is non-stationary. Now you fix it — starting with differencing, the standard tool for removing trend. The idea is almost embarrassingly simple: instead of modeling the level of the series, model the change from one point to the next. A steadily climbing trend becomes, after differencing, a roughly constant number — no more climbing left to explain.

By the end of this lesson, you will be able to:

• Compute a first difference with pandas and explain why it removes a linear trend
• Confirm the fix by re-running the ADF test and reading the new p-value
• Interpret the mean of a differenced series as the trend’s per-step growth rate
• Recognize that passing the ADF test doesn’t mean all structure is gone

Let’s difference Cyclepath.

First Differencing#

import numpy as np, pandas as pd
from statsmodels.tsa.stattools import adfuller

def cyclepath():
    idx = pd.date_range("2016-01-01", periods=96, freq="MS")
    t = np.arange(96); rng = np.random.default_rng(42)
    trend = 9000 + 90*t; seasonal = 3200*np.sin(2*np.pi*(t-3)/12); noise = rng.normal(0,350,96)
    return pd.Series(np.round(trend+seasonal+noise).astype(int), index=idx, name="trips")

y = cyclepath()

d1 = y.diff().dropna()
print(len(d1))                # 95
print(round(d1.mean(), 2))    # 88.05

y.diff() computes y_t - y_{t-1} at every point, which is why the result has 95 values instead of 96 — the very first month has no prior value to subtract. The mean of the differenced series is about 88.05 — and that’s not a coincidence: Cyclepath’s trend was built with a slope of exactly 90 per month (trend = 9000 + 90*t), so differencing recovers something very close to that slope directly. This is the intuition behind why differencing works: subtracting consecutive values cancels out a constant level and leaves behind the constant rate of change — which is a much easier thing for a model to treat as “roughly constant” than an ever-rising level.

Confirm It: Re-run the ADF Test#

stat, pval, *_ = adfuller(d1, autolag="AIC")
print(round(stat, 3))   # -8.642
print(round(pval, 4))   # 0.0

The ADF statistic drops from -0.920 (raw series) to -8.642, and the p-value collapses from 0.7815 to effectively 0 — nowhere close to the 0.05 threshold, decisively rejecting the unit-root null. By the ADF test’s definition, first-differenced Cyclepath is stationary. One .diff() call and a trend that visibly climbed for eight years is gone.

The Catch: “Stationary” Isn’t “Structure-Free”#

Before treating this as finished, look at what’s still hiding in the differenced series. Autocorrelation at lag 12 asks: does this month’s change relate to the change from exactly one year ago?

from statsmodels.tsa.stattools import acf

a = acf(d1, nlags=13, fft=True)
print(round(a[1], 3))    # 0.746
print(round(a[6], 3))    # -0.853
print(round(a[12], 3))   # 0.801

That 0.801 at lag 12 is a big number — nearly as strong as a series correlated with itself. Differencing was built to cancel a trend; it was never built to touch seasonality, and this is the proof. Cyclepath’s differenced series passes the ADF test (no unit root, no drifting mean) while still carrying almost the entire seasonal rhythm Module 2 measured — the changes from one July to the next July, one January to the next January, are still strongly related, exactly as they were in the raw series. The lag-6 value of -0.853 tells the same story from the other direction: a change six months apart (roughly opposite points in the seasonal cycle) is strongly negatively related.

Practice Exercises#

Exercise 1: What would happen to a quadratic trend?#

If a series had a quadratic trend (curving, not a straight line) instead of Cyclepath’s linear one, would a single first difference (d=1) be enough to remove it?

Exercise 2: Interpret a differenced mean of zero#

Suppose a different series’ first difference has a mean very close to 0, unlike Cyclepath’s 88.05. What does that tell you about the original series?

Exercise 3: Predict the next step#

Given that differencing fixed the trend but left a lag-12 autocorrelation of 0.801, what would you try next, and why?

Summary#

First differencing (y.diff()) replaces each value with its change from the previous one, and on Cyclepath it works exactly as intended: the ADF p-value drops from 0.7815 to effectively 0, and the differenced mean of 88.05 closely matches the series’ true per-month trend slope of 90. The differenced series is, by the ADF test’s definition, stationary. But its autocorrelation at lag 12 is a striking 0.801 — proof that differencing, built specifically to cancel a linear trend, left the entire seasonal pattern untouched. Passing the ADF test is a real, necessary milestone, but for a seasonal series it is not the end of the stationarity story.

Key Concepts#

• First differencing — y_t - y_{t-1}, cancels a constant (linear) trend by construction.
• Differenced mean = trend slope — a first-differenced series’ mean approximates the original series’ per-step rate of change.
• ADF confirms, doesn’t diagnose everything — passing the test rules out a unit root, not seasonality or other structure.
• Lag-12 autocorrelation — a large value at the seasonal lag is the specific symptom of leftover seasonality after trend-only differencing.

Why This Matters#

It’s tempting to treat “the ADF test passed” as “I’m done” — but for any series with real seasonality, that’s premature, and Cyclepath’s lag-12 autocorrelation of 0.801 is exactly the kind of evidence that should stop you from moving on too early. The habit this lesson builds — check the test, then check what’s still left in the autocorrelation — is what keeps a differencing decision honest instead of just technically passing. Next, Lesson 4 introduces seasonal differencing to deal with that leftover structure directly, and discusses when a transformation like a log is the better tool instead.

Next Steps#

Continue Building Your Skills#

First differencing fixed Cyclepath’s trend and passed the ADF test — but that lag-12 autocorrelation of 0.801 is unfinished business. Next you’ll meet seasonal differencing, built specifically to target a relationship at a fixed seasonal lag instead of the one-step relationship regular differencing handles, and see how it compares.