A safe delivery pipeline for a 9-year-old marketplace, live in one day

The situation

A B2B marketplace for footwear and apparel wholesale has run on the same Laravel codebase since 2017. It has about 3,900 commits, Redis queues and one dedicated server.

The setup had grown the usual way:

• The code lived in an outside agency’s GitLab.
• Changes went straight to production. There was no staging copy and no check between a commit and live customers.
• The server was under memory pressure. Background workers had multiplied over the years, some of them duplicates and some running as root.

A database migration that failed halfway, or a worker that ate the last of the memory, would take the marketplace down for its buyers and suppliers.

What we did

Server tuning first. Before adding anything new, we made room on the existing machine.

• Background workers went from 49 to 9. Duplicates were removed, and none of them runs as root any more.
• Free RAM went from 2.5 GB to 10.3 GB.
• With that headroom, PHP-FPM workers went from 10 to 50, so the site can serve more requests at once.
• Root disk use dropped from 80% to 48%.

Backups that can rewind.

• The database goes to off-site storage 4 times a day.
• The binlog gives 7 days of point-in-time recovery, so we can restore to the minute before a bad change, not just to the last dump.
• Files are backed up daily and kept for 6 months.
• An hourly Git mirror keeps the code history even if someone force-pushes over it.

The pipeline.

• CI runs on the client’s own runners.
• Every new database migration is tested on a copy of the production schema: apply, roll back, apply again. A migration that cannot be undone is caught before it reaches production.
• Staging runs on a copy of the database. Production photos are mounted read-only, and email and notifications are switched off, so nobody gets a test message.
• Production deploys run from a button, through a gate on the server itself that a pull request cannot weaken.
• Before any migration, the database is dumped and its binlog position is recorded.

Outcome

The pipeline went live in a single day, across 39 pull requests. Some of those were real fixes, not only pipeline set-up.

The marketplace now has a staging copy to check changes on, migrations that are tested before they run, and a recovery point for every minute of the past week. The server has room to breathe, and the team can change a 9-year-old codebase without holding its breath.