Lesson 3 - Why Cron Isn't Enough

Welcome to Why Cron Isn’t Enough#

Lesson 2’s chain worked, end to end, on the real January data. That was the easy day. This lesson gives CityFlow’s pipeline a bad day — a single missing reference file on one ordinary Sunday — and traces, with real code and real output, everything that goes wrong because the only thing running this pipeline is && and a cron timer.

By the end of this lesson, you will be able to:

• Reproduce a real transient failure that kills a &&-chained pipeline partway through
• Show, with real files, that a stopped chain leaves downstream output silently stale
• Show that a stopped chain leaves no unified record of what ran and what didn’t
• Explain what “no backfill” costs a team in practice when a scheduled run is simply missed

Setting the Scene: Yesterday Succeeded#

Say CityFlow’s pipeline ran fine yesterday, 2024-01-20, and produced real output: 96,799 trips, $2,364,118.62 in revenue. Today, 2024-01-21, it’s cron’s turn to run it again. Start by seeding that real “yesterday” state on disk, so the failure below has something concrete to leave behind — or not:

import warnings
warnings.filterwarnings("ignore")
import os, sys, subprocess, tempfile

WORKDIR = os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")
os.makedirs(WORKDIR, exist_ok=True)

def p(name):
    return os.path.join(WORKDIR, name)

# clean slate, so this demo behaves the same whether it's the first run or the tenth
for name in ["cityflow_daily_summary.csv", "cityflow_pipeline_report.txt",
             "step1.log", "step2.log", "step3.log", "step4.log"]:
    if os.path.exists(p(name)):
        os.remove(p(name))

# seed "yesterday's" already-successful output: 2024-01-20's real numbers from Lesson 2
with open(p("cityflow_daily_summary.csv"), "w") as f:
    f.write("pickup_date,trips,avg_fare,avg_distance,total_revenue\n")
    f.write("2024-01-20,96799,16.5,2.83,2364118.62\n")
with open(p("cityflow_pipeline_report.txt"), "w") as f:
    f.write("CityFlow report for 2024-01-20: 96,799 trips, $2,364,118.62 revenue\n")

print("seeded yesterday's (2024-01-20) outputs in", WORKDIR)

seeded yesterday's (2024-01-20) outputs in /tmp/cityflow_cron_demo

Today’s job is a four-step chain: extract today’s trips, enrich them against a zone-name reference file, aggregate, and report. Write all four as real, separate scripts, exactly like Lesson 2:

STEP1 = """
import os, tempfile, datetime as dt
import pandas as pd

WORKDIR = os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")
TARGET_DATE = "2024-01-21"  # the day after the "yesterday" seeded above

def log(msg):
    with open(os.path.join(WORKDIR, "step1.log"), "a") as f:
        f.write(dt.datetime.now().isoformat() + " step1_extract_today: " + msg + "\\n")

log("started")
cols = ["tpep_pickup_datetime", "passenger_count", "trip_distance", "fare_amount", "total_amount"]
raw = pd.read_parquet("yellow_tripdata_2024-01.parquet", columns=cols)
is_target_day = raw["tpep_pickup_datetime"].dt.date.astype(str) == TARGET_DATE
valid = is_target_day & (raw["passenger_count"] > 0) & (raw["trip_distance"] > 0) & (raw["fare_amount"] > 0)
today = raw.loc[valid]
trips = len(today)
revenue = round(float(today["total_amount"].sum()), 2)

with open(os.path.join(WORKDIR, "today_valid_trips.txt"), "w") as f:
    f.write(TARGET_DATE + ": " + str(trips) + " valid trips, $" + format(revenue, ",.2f") + " revenue\\n")

log("finished ok, " + str(trips) + " valid trips found for " + TARGET_DATE)
print("step1_extract_today: ok -", format(trips, ","), "valid trips for", TARGET_DATE)
"""

STEP2 = """
import os, sys, tempfile, datetime as dt

WORKDIR = os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")

def log(msg):
    with open(os.path.join(WORKDIR, "step2.log"), "a") as f:
        f.write(dt.datetime.now().isoformat() + " step2_enrich_zone_reference: " + msg + "\\n")

log("started")
# Real work: enrich today's trips with a zone-name reference lookup.
# The reference file is missing today -- a transient upstream problem,
# exactly the kind of blip that would succeed if you just retried a minute later.
reference_path = os.path.join(WORKDIR, "zone_reference.csv")
if not os.path.exists(reference_path):
    log("ERROR: zone_reference.csv unavailable (transient)")
    print("step2_enrich_zone_reference: ERROR - zone reference lookup timed out", file=sys.stderr)
    sys.exit(1)
log("finished ok")
print("step2_enrich_zone_reference: ok")
"""

STEP3 = """
import os, tempfile, datetime as dt
import pandas as pd

WORKDIR = os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")
TARGET_DATE = "2024-01-21"

def log(msg):
    with open(os.path.join(WORKDIR, "step3.log"), "a") as f:
        f.write(dt.datetime.now().isoformat() + " step3_aggregate: " + msg + "\\n")

log("started")
cols = ["tpep_pickup_datetime", "passenger_count", "trip_distance", "fare_amount", "total_amount"]
raw = pd.read_parquet("yellow_tripdata_2024-01.parquet", columns=cols)
is_target_day = raw["tpep_pickup_datetime"].dt.date.astype(str) == TARGET_DATE
valid = is_target_day & (raw["passenger_count"] > 0) & (raw["trip_distance"] > 0) & (raw["fare_amount"] > 0)
today = raw.loc[valid]
trips = len(today)
avg_fare = round(float(today["fare_amount"].mean()), 2)
avg_distance = round(float(today["trip_distance"].mean()), 2)
revenue = round(float(today["total_amount"].sum()), 2)

with open(os.path.join(WORKDIR, "cityflow_daily_summary.csv"), "a") as f:
    f.write(",".join([TARGET_DATE, str(trips), str(avg_fare), str(avg_distance), str(revenue)]) + "\\n")

log("finished ok, appended " + TARGET_DATE + " to daily summary")
print("step3_aggregate: ok -", format(trips, ","), "trips")
"""

STEP4 = """
import os, tempfile, datetime as dt
import pandas as pd

WORKDIR = os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")

def log(msg):
    with open(os.path.join(WORKDIR, "step4.log"), "a") as f:
        f.write(dt.datetime.now().isoformat() + " step4_report: " + msg + "\\n")

log("started")
daily = pd.read_csv(os.path.join(WORKDIR, "cityflow_daily_summary.csv"))
last = daily.iloc[-1]
line = "CityFlow report for " + str(last["pickup_date"]) + ": " + format(int(last["trips"]), ",") + \\
       " trips, $" + format(last["total_revenue"], ",.2f") + " revenue\\n"
with open(os.path.join(WORKDIR, "cityflow_pipeline_report.txt"), "a") as f:
    f.write(line)
log("finished ok")
print("step4_report: ok")
"""

for name, src in [("step1_extract_today.py", STEP1), ("step2_enrich_zone_reference.py", STEP2),
                  ("step3_aggregate.py", STEP3), ("step4_report.py", STEP4)]:
    with open(p(name), "w") as f:
        f.write(src)

print("wrote 4 step scripts to", WORKDIR)

wrote 4 step scripts to /tmp/cityflow_cron_demo

Notice step2_enrich_zone_reference.py checks for a zone_reference.csv file that this demo deliberately never creates — a stand-in for any transient upstream problem: a reference file that hasn’t landed yet, a flaky network call, a service that’s briefly unavailable. It’s exactly the kind of failure that would succeed if you simply tried again a minute later.

Gaps 1 & 2: No Dependency Tracking, No Retry#

Chain the four scripts the way cron would, with &&, and watch what actually happens:

chain = ["step1_extract_today.py", "step2_enrich_zone_reference.py",
         "step3_aggregate.py", "step4_report.py"]
ran = []
for script in chain:
    result = subprocess.run([sys.executable, p(script)], capture_output=True, text=True)
    ran.append(script)
    print(script + ": exit code " + str(result.returncode) + " -> " + result.stdout.strip())
    if result.returncode != 0:
        print("  -> chain stopped here (cron's && semantics): " + result.stderr.strip().splitlines()[-1])
        break

never_ran = [s for s in chain if s not in ran]
print()
print("steps that actually ran:", ran)
print("steps that NEVER ran:   ", never_ran)

step1_extract_today.py: exit code 0 -> step1_extract_today: ok - 76,103 valid trips for 2024-01-21
step2_enrich_zone_reference.py: exit code 1 -> 
  -> chain stopped here (cron's && semantics): step2_enrich_zone_reference: ERROR - zone reference lookup timed out

steps that actually ran: ['step1_extract_today.py', 'step2_enrich_zone_reference.py']
steps that NEVER ran:    ['step3_aggregate.py', 'step4_report.py']

Step 1 found a real 76,103 valid trips for 2024-01-21 — the pipeline was doing genuinely correct work right up until the reference lookup blipped. Then && did exactly what && does: the instant step2 exited 1, the whole chain stopped. There’s no retry policy anywhere in this system — cron doesn’t know this failure was transient, doesn’t know it would have succeeded thirty seconds later, and doesn’t try again. It just stops, silently, in the middle of the night, and the next thing that happens is nothing, until a human notices.

Now look at what that leaves behind:

with open(p("cityflow_daily_summary.csv")) as f:
    summary_after = f.read()
print("cityflow_daily_summary.csv after the failed run:")
print(summary_after)
print("still only 2024-01-20 in there -- today's row was never added, and nothing on disk says so.")

cityflow_daily_summary.csv after the failed run:
pickup_date,trips,avg_fare,avg_distance,total_revenue
2024-01-20,96799,16.5,2.83,2364118.62

still only 2024-01-20 in there -- today's row was never added, and nothing on disk says so.

cityflow_daily_summary.csv is untouched — still showing 2024-01-20 as the latest day, because step3_aggregate.py (the step that would have appended 2024-01-21) never ran. If a dashboard or a teammate reads this file tomorrow morning, nothing tells them it’s a day stale. There’s no timestamp check, no “expected rows: 31, found: 30” alert, no dependency graph anywhere that knows step3 was supposed to run and didn’t. The file just looks like a file. This is “no dependency tracking” made concrete: cron enforces order (step2 after step1) but has zero concept of whether a downstream file is actually fresh, complete, or current.

Gap 3: No Single Place to See What Ran#

The four scripts each keep their own log, because that’s the only logging mechanism a hand-built chain has:

print("log files present after the run:")
for name in ["step1.log", "step2.log", "step3.log", "step4.log"]:
    print(" ", name + ":", "exists" if os.path.exists(p(name)) else "MISSING (step never ran)")

log files present after the run:
  step1.log: exists
  step2.log: exists
  step3.log: MISSING (step never ran)
  step4.log: MISSING (step never ran)

To reconstruct “what happened this morning,” someone has to know there are four log files, know where they live, open each one by hand, and notice that two of them simply don’t exist. There’s no single timeline that says “run started at 02:00, step1 ok, step2 failed, run aborted” — there’s just a scattering of files, some present, some not, with the absence of a file being the only signal that a step never even started. On a pipeline with ten steps instead of four, or one that’s failed on three different days for three different reasons, this stops being an inconvenience and becomes a real investigation every time.

Gap 4: No Backfill#

Suppose nobody notices until Wednesday that Monday’s run failed this way. CityFlow is now missing 2024-01-21 and whatever ran on the 22nd assumed the 21st already happened. Fixing this by hand means running the exact chain again, once per missed day, with no help from cron at all — cron only knows how to run the next scheduled invocation, not “catch me up on what I missed”:

# Manual backfill: cron has no concept of "run this for a date I missed,"
# so re-running for each missing day means invoking the chain by hand, once per day.
for date in 2024-01-21 2024-01-22; do
  echo "Backfilling $date..."
  TARGET_DATE="$date" python3 step1_extract_today.py \
    && TARGET_DATE="$date" python3 step2_enrich_zone_reference.py \
    && TARGET_DATE="$date" python3 step3_aggregate.py \
    && TARGET_DATE="$date" python3 step4_report.py
done

Nothing here is wrong, exactly — it would work, given the scripts were parameterized by date, which this lesson’s versions weren’t. That’s the point: making a hand-built pipeline backfillable takes deliberate, extra engineering (date parameters, a loop, a way to know exactly which days are missing), and cron gives you none of it for free. Every one of these four gaps — stale dependencies, no retry, scattered logs, manual backfill — comes from the same root cause: cron can start a process on a schedule, and that is the entirety of what it knows how to do.

Key Takeaways#

• No dependency tracking: && enforces order, not freshness — cityflow_daily_summary.csv silently stayed on 2024-01-20’s data after the chain stopped, with nothing on disk flagging it as stale.
• No retry: a transient failure (a missing zone_reference.csv) killed the whole chain at step2, even though the same command would likely succeed on a second try; cron never makes that second try.
• No single place to see what ran: step1.log and step2.log exist, step3.log and step4.log don’t — reconstructing what happened requires knowing to check four separate files.
• No backfill: catching up on a missed day requires a human to notice, then manually re-run the chain once per missing date — cron has no built-in notion of “days I owe you.”

Exercises#

Exercise 1 — Create zone_reference.csv (any content, even an empty file) inside the same working directory this lesson used (os.path.join(tempfile.gettempdir(), "cityflow_cron_demo")) before running the chain, then re-run the four-step chain. Confirm all four steps now report exit code 0, and that cityflow_daily_summary.csv gains a real 2024-01-21 row.

Exercise 2 — Modify step2_enrich_zone_reference.py so that instead of always failing, it fails only the first time it’s called (track that with a small marker file it creates on its first run) and succeeds on any later call. Show that a human re-running the chain by hand now gets a working pipeline — then explain in a comment why this is still not “retry,” since nothing runs step2 again automatically.

Exercise 3 — Extend the backfill loop to cover three missing days instead of two, and add a check after each day’s chain that prints whether that day succeeded or failed, based on the exit code of the last command in the chain. What does this tell you about how much custom bookkeeping a hand-built pipeline needs before it can even answer “which of the last N days actually succeeded?”