How-To: Reboot on OOM

Ever had your linux box getting Out of Memory (OOM)? Cleaning up after the OOM killer kicked in to find out that even though OOM killer did a decent job at trying to kill the bad processes, your system ends up in an unknown state and you might be better of rebooting the host to make sure it comes back up in a state that you know is safe.

This is in no way a solution to your OOM problem and you will most likely need to troubleshoot further the cause of the OOM, nonetheless, this will make sure that when the issue happens (in the middle of the night?), your system gets back on track as fast as possible.

If you use this solution, you really need to make sure that when your system reboots, it is actually functional!

Warnings been said, let see how we can make our system reboot when the OOM killer gets in town.

Well, the trick is actually to panic the kernel on OOM and to tell the system to reboot on kernel panic.

This is done via procfs. You could either echo the values in /proc/sys/…, but this will not be kept upon reboot, OR set the values in sysctl file and activate them. I will go for the later solution.

Create a file called /etc/sysctl.d/oom_reboot.conf and enter the lines below:

# panic kernel on OOM
vm.panic_on_oom=1
# reboot after 10 sec on panic
kernel.panic=10

Then, activate the settings with:

# sysctl -p /etc/sysctl.d/oom_reboot.conf
vm.panic_on_oom = 1
kernel.panic = 10

And you should be good to go. You can tweak the number of seconds you want to wait before the system reboot to you need by replacing 10 with whatever value suits you.

If OOM killer was going to kick in again, you would get the same screen than the screenshot above.

If you want to confirm this works, you can run the program below on a test host (a VM on your workstation is a good candidate):

/**
oom.c
allocate chunks of 10MB of memory indefinitely until you run out of memory
*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define TEN_MB 10 * 1024 * 1024

int main(int argc, char **argv){
        int c = 0;
        while (1){
                char *b = malloc(TEN_MB);
                memset(b, TEN_MB, 0);
                printf("Allocated %d MB\n", (++c * 10));
        }
        return 0;
}

Then compile with:

$ gcc -o oom oom.c

And run it:

$ ./oom
Allocated 10 MB
Allocated 20 MB
...
...