No changelog entry. No email to integration partners. A field that had come back as a plain string for a year started coming back as a nested object overnight, and the pipeline that depended on it didn't error out — it just started silently dropping the records that didn't match the shape it expected.
Nobody noticed for four days. The first sign was a client asking why last week's orders never made it to fulfillment.
The original integration trusted the vendor's contract implicitly — reasonable, since it had held for a year — and treated a shape mismatch as "skip this record, log a warning, keep going," so the pipeline itself never went down. It just quietly stopped doing its job for the specific record shape that had changed, and the warning logs weren't being watched because nothing had ever needed watching.
That's the failure mode that should worry you more than an outage. An outage pages someone. Silent, partial failure just erodes trust in your data a little more every day until someone finally notices the numbers are wrong.
Not a retry. Not a bigger try/catch. The fix was making the pipeline's assumptions about the vendor's data explicit instead of implicit:
None of that stopped the vendor from changing their API without telling anyone — that's out of my control and, frankly, not unusual. What changed is how loudly and how quickly we found out.
Every third-party API is a promise, and every promise eventually gets broken, usually without a changelog. The interfaces between your systems and theirs are where this risk concentrates, so that's where you build in the assumption that it will happen — not as a hypothetical, but as a Tuesday.
An integration that fails loudly the day the vendor changes something is a good integration. One that fails silently for four days is a liability wearing a green checkmark.
Field Mappings