Dataset in, finished pages out
Short answer: A business publishing on a fixed schedule did every step by hand: reading source data, formatting pages, checking completeness, uploading. The failures were embarrassing rather than exotic - a missing image, a truncated description. The pipeline now runs those steps in order with a gate that refuses to publish an incomplete page, and the operator only sees exceptions. Handover included the gate rules and a log of every publish.
Checked on 2026-10-20
A business publishing content on a fixed schedule did every step by hand: reading the source data, formatting pages, checking they were complete, uploading them. The work was repetitive but not trivial, and the failures were embarrassing: a missing image or a dead link shipped to readers because nobody had time to check.
What shaped the work?
Three constraints shaped the build: a fixed publishing calendar, source data arriving in different shapes, and a failure mode that stayed invisible until a reader complained. Each one shows up in the design below.
- Editors keep editorial control: the pipeline may never decide what gets published.
- A release must be able to fail safely — half-published is worse than not published.
- The output had to be plain files that the existing host could serve, with no new monthly platform.
- Non-technical staff needed to re-run a single item without touching the database.
What does it run on?
Python · cron · static output · validation gates
What did we actually build?
The pipeline runs five steps in order and refuses to publish an incomplete page. What follows is what each step does, where it stops, and what the operator sees when it does.
- Source data in, pages out: templates plus per-item validation, run on a schedule.
- Quality gates before release: required fields, image presence, link resolution, duplicate slugs. Anything failing is held back and reported, not published.
- Failures are loud: the run leaves a log tail and an email, and the previously published version stays live.
- Single-item re-run, so an editor who fixes one record does not wait for tomorrow's batch.
- Static output is diffed before release, so a template change cannot alter hundreds of pages at once.
The choices that mattered
Gates block, humans decide
The pipeline never publishes around a failed check.
Static output
Cheap to host, trivial to back up, nothing to keep patched.
Re-run one item
The most-used feature in practice; without it, staff work around the system.
Report, do not retry blindly
A missing field is not a transient error. Retrying it just hides it three more times.
What we would do next
- Editorial preview links so a draft can be reviewed by someone outside the team before release.
- Scheduled release windows per section instead of one global batch.
What is not on this page
No client name, no screenshots of their data and no metrics: those are theirs. What is here is the engineering, which is the part a buyer can evaluate. On a call we will walk through the code.