Aviation Data · Retrospective · Nov 2026
Why I switched off the IST delay pipeline.
From May to October 2026 a small pipeline built a flight-delay dataset for Istanbul Airport — a paid backfill of May to July, then nightly collection — joined it with ADS-B tracks and weather, and published a delay dataset, a model and a live prediction board here. On 14 November 2026 I switched it off. It produced real results, but it had stopped producing trustworthy data two months before anyone noticed — including me. This page is what it found, what broke it, and why I'm putting the time back into mechanical engineering.
Timeline
Six months, three data sources, one person.
- May 1 – Jul 26 · paid backfill~1,500 flights a day. 96% of everything in the dataset comes from these 85 days.
- Jul 27 · free tierDown to ~100 flights a day — the free schedule API allows 100 requests a month. Enough for a live board, not for a full airport.
- Aug 17 – Sep 5 · three failures at onceOpenSky lost coverage of Istanbul, my OpenSky API account was disabled, and the token that publishes to this site expired. I fixed the account and the token by hand; the coverage came back on its own in early September.
- Sep 13 · the quiet oneThe schedule provider changed what its "delay" field measures. Nothing errored. The delay rate on this site went from about 13% to 66% overnight. Details below.
- Oct 11 · OpenSky stops answeringEvery request through the relay that lets GitHub's servers reach OpenSky started coming back as HTTP 502. Daily flight counts fell from ~100 to 0–2. Every run still reported success.
- Nov 14 · switched offWorkflows disabled, pages frozen on the clean period (May 1 – Sep 12), this write-up published.
The quiet failure
Istanbul didn't suddenly get five times later. The definition did.
The delay label came from the schedule provider, which reports a delay value only for some flights. I treated a blank as "on time" and checked that once against an independent source — OpenSky's ADS-B timestamps — and it held. Then, on 13 September, the provider started filling in a value for 98% of flights instead of ~30%, and the share of departures labelled 15+ min late jumped from about 13% to about 66%.
The independent timestamps show what actually changed. Before 13 September, the reported delay ended about 25 minutes before the aircraft was seen airborne — it measured the pushback from the gate, and taxiing came after. From 13 September it ended about 1.5 minutes before airborne — it now measured takeoff, so it included Istanbul's ~20-minute taxi. Meanwhile the time from schedule to airborne didn't move at all.
Show the numbers
| Week of | Departures | Delay value present | Reported 15+ late | Schedule→airborne (min) |
|---|---|---|---|---|
| 27 Apr | 532 | 24% | 9.0% | — |
| 4 May | 3,914 | 29% | 11.7% | — |
| 11 May | 4,377 | 38% | 17.2% | — |
| 18 May | 4,731 | 29% | 12.6% | — |
| 25 May | 4,956 | 26% | 10.3% | 22.4 |
| 1 Jun | 4,658 | 26% | 10.3% | 22.3 |
| 8 Jun | 2,688 | 32% | 12.9% | 23.1 |
| 15 Jun | 5,275 | 27% | 11.5% | 21.9 |
| 22 Jun | 5,264 | 25% | 11.4% | 22.4 |
| 29 Jun | 5,912 | 26% | 12.1% | 22.5 |
| 6 Jul | 5,543 | 30% | 12.6% | 22.4 |
| 13 Jul | 5,663 | 30% | 14.1% | 23.3 |
| 20 Jul | 5,668 | 43% | 19.5% | 24.4 |
| 27 Jul | 1,196 | 35% | 13.1% | 25.0 |
| 3 Aug | 736 | 31% | 12.0% | 21.0 |
| 10 Aug | 688 | 35% | 14.2% | 23.6 |
| 17 Aug | 568 | 42% | 16.5% | — |
| 24 Aug | 740 | 44% | 21.1% | — |
| 31 Aug | 323 | 40% | 13.9% | 19.8 |
| 7 Sep | 693 | 45% | 17.3% | 25.0 |
| 14 Sep | 644 | 99% | 61.6% | 19.7 |
| 21 Sep | 664 | 99% | 65.2% | 21.2 |
| 28 Sep | 640 | 99% | 74.2% | 22.6 |
| 5 Oct | 579 | 99% | 65.8% | 20.4 |
Why I'm stopping
Keeping it alive had become maintenance, not engineering.
A pipeline like this has to keep three outside services working together — a flight-tracking network, a commercial schedule API and a relay server — plus the tokens and accounts that connect them. Each one changes on its own schedule, without telling you: coverage disappears, an account gets disabled, a field quietly changes meaning, a server starts returning errors. None of that is under my control, and almost none of it shows up as a failed run. Keeping up with all of it is hard for one person — and the moment I stopped watching closely, it drifted for two months without anyone knowing.
Both failures that ended it were silent: the pipeline logged a warning and exited "successfully", so the site kept serving numbers that looked plausible. If I kept it running, the next fix would have to be monitoring — alerts on zero rows, checks that the label still means what it meant. That is real work, but it is not the work I want this portfolio to be about. I'm a mechanical engineering student, and that is where the time is going now.
What survived
The results from the clean period still stand.
How the live board's probabilities held up
Each flight was given a delay probability before it departed. Grouped by that probability, here is how often they really were late:
| Predicted risk | Flights | Average predicted | Actually late |
|---|---|---|---|
| 0–8% | 607 | 3.5% | 6.1% |
| 8–12% | 181 | 9.8% | 10.5% |
| 12–20% | 101 | 14.9% | 17.8% |
| 20–35% | 136 | 26.7% | 27.2% |
| 35%+ | 42 | 43.4% | 33.3% |
The middle of the range is almost exactly right. The "safe" flights ran late about twice as often as predicted — the same weakness the offline tests showed: a bad day hits normally punctual flights, and nothing in their history warns of it.
Lessons
What I'd do differently.
- A run that collects nothing is a failed run. Zero flights should turn the job red, not log a warning.
- A label is someone else's definition. Checking it once against an independent source isn't enough — check it every day, automatically.
- Count every dependency as a cost. Each outside service is a separate thing that can break on its own schedule.
- Plan how it ends. A live project with no end date eventually becomes a stale page that still says "live".
The archive