503s on git.kolab.org

We regularly experience “503: Service Unavailable” on git.kolab.org as of late, even though the system is more up to date, and both fatter and leaner.

Fatter, because more resources are allocated to it. More CPU, more memory, larger APC, opcache, and database services got separated out from the system, and put on to a MariaDB Galera cluster (which, by the way, is pointing at MySQL laughing, while running circles around it).

Leaner, because of the slimmer database processing costs, better caching, more tmpfs, a slimmer version of Phabricator — all vendorized and otherwise unused code has been stripped, and the amount of cached op-code for PHP — in a more recent version of PHP as well, that we provide to our infrastructure precisely because of those enhancements.

Still, git.kolab.org gives you 503s — otherwise known as the incapable sys-admin giving you the finger. Mind you I used the word “in-capable”, not “incompetent” — you shall learn to appreciate the difference.

The cause of these temporary outages — for you, they may appear to repair themselves — is the system running Out Of Memory (or OOM).

It is now up to us (read: me) to find out what is causing the OOM events, and address it. This is not necessarily something I have a significant chunk of time for though, compared to all other things that also need happening.


One thought on “503s on git.kolab.org

Comments are closed.