Marketplace · Incident response
A 5-hour marketplace outage, traced and written up the same day
A marketplace platform was down for 5 hours. The same day, we found the cause, put a stopgap in place and wrote three separate explanations: one for the developer, one for operations and one for the owner.
The situation
A marketplace platform went down for 5 hours. For a marketplace, that means buyers cannot buy and sellers cannot sell, and everyone wants to know two things: will it happen again, and whose problem is it?
Restarting a server brings a site back, but it does not answer either question. Without a cause, the same outage tends to return at the worst possible time.
What we did
Find the cause, the same day. We traced the outage to a memory leak in the application. The application was using more and more memory until the server could no longer serve requests.
Put a stopgap in place. A real fix to a memory leak belongs in the application code and takes time to write and test. Until then, a stopgap keeps the leak from taking the platform down again.
Write it up for three audiences. One postmortem rarely works for everyone, so we wrote three:
- For the developer: where the leak shows up, the evidence, and what needs fixing in the code.
- For the operations team: what to watch, what the stopgap does and what to do if the symptoms come back.
- For the owner: what happened, what it cost in downtime, what is in place now and what still needs to be done, in plain language.
Outcome
By the end of the day, the platform was back, protected by a stopgap, and every party had a clear explanation and their own next step. The developer had what they needed to fix the leak properly, operations knew what to watch, and the owner knew where things stood without reading a stack trace.
A good postmortem does not look for someone to blame. It makes the next outage less likely and shorter if it happens.