When a program crashes, you’ll see error messages like Segmentation Fault. You might also see (core dumped) as well. I used to feel annoyed when I saw those words because the program crashed. When I was an intern at Tianrang Networking Inc. this summer, I watched my colleague Wengong Jin work on C++ programs. Then realized that I should be glad to see (core dumped) because the core dump file keeps all runtime information at the moment the program crashes, including data in memory, threads, stack traces, registers, and so on. It should be helpful for locating the bug.

Generate and Use Core Dump Files

On Ubuntu, to generate the core dump file when a program crashes, you need to run the following command in advance:

ulimit -c unlimited

After that, run the buggy program on the same shell session. The reason is that the size limit of core dump files is 0 by default, which prevents it from being created. This command sets it unlimited for the current shell session. After the buggy program crashes, you’ll find a file named core in the current working directory. You can load it with gdb, for example:

gdb ./a.out core

Then you’ll see everything just as if you attached to the program when it crashed.

Notice that the core file won’t be overwritten. So remember that before running the buggy program, delete the existing core file.

Trigger Core Dump in Program

When writing a program, we might want the program to terminate when some fatal events happen. Besides printing logs and calling exit(), we can also abort(). When abort() is called, the program exits immediately without cleaning up, and generates the core dump file.

Generate Core Dump Files inside Docker

We use nvidia-docker in our development environment. It took me quite some time to figure out how to generate core dump files inside Docker.

First, run the following command on the host:

echo '/tmp/core.%t.%e.%p' | sudo tee /proc/sys/kernel/core_pattern

The kernel generates the core dump file according to the settings in /proc/sys/kernel/core_pattern. The default value on Ubuntu is |/usr/share/apport/apport %p %s %c %P, i.e. passing to apport via the pipe. Docker containers share the same /proc as the host. However, apport is not likely to be installed inside the Docker container. Therefore, we change the setting on the host, asking the kernel to save core dump files directly to /tmp which is a directory that the Docker container also has.

Then, pass the following additional arguments to docker run:

--ulimit core=-1 --security-opt seccomp=unconfined

The first argument set the size limit of core dump files to unlimited. The second one allows ptrace APIs inside the Docker container so that gdb has enough previledges.

Reference