By ByteBot
At 00:41 UTC today, Firebase’s iOS SDK began crashing every app that had Firebase Analytics enabled — on launch, for every session, with no workaround available to developers. The cause was a nil key in a server-side experiment payload served by Google’s own analytics backend, triggering an unhandled NSInvalidArgumentException inside GULMutableDictionary. SDK versions 10.24.0 through 12.14.0 were all affected. Developers shipped nothing. Their apps crashed anyway.
Google deployed a server-side fix approximately 2 hours and 11 minutes after the crash began. However, client-side caching meant affected apps continued crashing for up to 4 additional hours after the fix went live, pushing full recovery to approximately 23:52 UTC today.
What Actually Broke in Firebase iOS SDK
Firebase Analytics fetches experiment configuration from Google’s backend at every app launch via a POST request to https://app-analytics-services.com/sdk-exp. The payload served starting at 00:41 UTC today contained a nil value where a dictionary key was expected. When APMETaskManager processed this response, it attempted to set a nil key on a GULMutableDictionary — a fatal operation in Objective-C that the SDK did not guard against.
Indeed, the crash was remarkably consistent. Of 42 crash reports examined in the primary GitHub incident report, every single one showed the same stack trace, triggered immediately after receiving an HTTP 200 from the sdk-exp endpoint:
APMETaskManager processExperimentFetchResponse:
APMEExperiment initWithExperimentID:flags:
GULMutableDictionary setObject:forKeyedSubscript:
NSInvalidArgumentException: key cannot be nilNo defensive coding on the app developer’s side could prevent this crash. The exception fired deep inside Firebase’s utility layer, before any application code ran. Five separate GitHub issues (#16728, #16729, #16730, #16732, #16734) were opened within minutes, all documenting the same failure across different SDK versions and different app teams.
Version Pinning Did Not Help
The standard advice for managing third-party dependency risk is to pin your SDK versions. Teams running Firebase iOS SDK 10.24.0, 10.29.0, 12.10.0, and 12.14.0 were all affected equally. The crash was triggered by the payload Google’s server returned, not by any code differences between those versions. As one developer put it in the Hacker News discussion: “Pinning versions doesn’t save you when the config comes from their server.”
This is the architectural reality of any SDK that fetches live configuration at runtime. Firebase Analytics uses this pattern for A/B testing and feature flags. Consequently, every app that includes Firebase hands Google’s backend a conditional kill switch — unintentional this time, but structurally present. OWASP’s M2 (Inadequate Supply Chain Security) specifically calls out this pattern: third-party SDKs that fetch runtime configuration expand your trusted computing base beyond what your team controls.
Related: TeamPCP’s Supply Chain Worm Hit Bitwarden and 168 More — Act Now
The Fix Was Two Hours — But Users Crashed for Six
Google’s engineering team identified the malformed payload and rolled out a server-side correction at approximately 19:52 UTC, roughly 2 hours and 11 minutes after the incident started. No SDK update was required from developers. However, “fix deployed” is not the same as “users stop crashing.” Client-side caching of the experiment configuration meant apps that had already fetched the bad payload continued crashing for up to 4 more hours post-fix.
One developer noted on the GitHub issue: “Resolved, but can still affect customers for up to 4 hours.” That is not a footnote. It means the total user impact window stretched to roughly 6 hours. For any app with meaningful global traffic, that is a significant outage — and developers with alerts firing “crash spike, no deployment” had no recourse except to communicate with users and wait.
What Developers Can Do About Firebase iOS SDK Risk
Ultimately, Firebase is not leaving most apps. The question is how to reduce the blast radius when this happens again. Lazy initialization helps — deferring FirebaseApp.configure() until after the first user interaction, rather than in application(_:didFinishLaunchingWithOptions:), can prevent a crash from killing the cold start entirely. It narrows the impact window without removing the dependency. Adding a crash guard around SDK initialization and routing failures to a secondary reporting channel (not Firebase) gives your team visibility even when your primary telemetry is the thing that’s down.
More importantly, every team with Firebase should have a “crash without deployment” runbook. When crash rates spike and nothing in your CI/CD explains it, the first checklist item should be: which third-party SDKs are currently fetching remote configuration? Check the Firebase iOS SDK release notes for known issues and monitor the firebase-ios-sdk GitHub issues page during incidents. This Firebase crash will not be the last server-side SDK incident. It is a category of problem, not a one-off.
Key Takeaways
- Firebase iOS SDK crashed every app using Firebase Analytics starting at 00:41 UTC today — triggered by a nil key in a Google server-side experiment payload, not developer error
- Version pinning offers no protection when an SDK crash is caused by vendor-side configuration; Firebase SDK versions 10.24.0 through 12.14.0 were all affected equally
- Google fixed the server payload in roughly 2 hours, but client-side caching extended user impact to approximately 6 hours total
- Any SDK that fetches live configuration at runtime is a conditional single point of failure — lazy initialization and a “crash without deployment” runbook reduce your exposure
