When a program crashes, you’ll see error messages like
Segmentation Fault. You might also see
(core dumped) as well. I used to feel annoyed when I saw those words because the program crashed. When I was an intern at Tianrang Networking Inc. this summer, I watched my colleague Wengong Jin work on C++ programs. Then realized that I should be glad to see
(core dumped) because the core dump file keeps all runtime information at the moment the program crashes, including data in memory, threads, stack traces, registers, and so on. It should be helpful for locating the bug.
Generate and Use Core Dump Files
On Ubuntu, to generate the core dump file when a program crashes, you need to run the following command in advance:
ulimit -c unlimited
After that, run the buggy program on the same shell session. The reason is that the size limit of core dump files is 0 by default, which prevents it from being created. This command sets it unlimited for the current shell session. After the buggy program crashes, you’ll find a file named
core in the current working directory. You can load it with
gdb, for example:
gdb ./a.out core
Then you’ll see everything just as if you attached to the program when it crashed.
Notice that the
core file won’t be overwritten. So remember that before running the buggy program, delete the existing
Trigger Core Dump in Program
When writing a program, we might want the program to terminate when some fatal events happen. Besides printing logs and calling
exit(), we can also
abort() is called, the program exits immediately without cleaning up, and generates the core dump file.
Generate Core Dump Files inside Docker
nvidia-docker in our development environment. It took me quite some time to figure out how to generate core dump files inside Docker.
First, run the following command on the host:
echo '/tmp/core.%t.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
The kernel generates the core dump file according to the settings in
/proc/sys/kernel/core_pattern. The default value on Ubuntu is
|/usr/share/apport/apport %p %s %c %P, i.e. passing to
apport via the pipe. Docker containers share the same
/proc as the host. However,
apport is not likely to be installed inside the Docker container. Therefore, we change the setting on the host, asking the kernel to save core dump files directly to
/tmp which is a directory that the Docker container also has.
Then, pass the following additional arguments to
--ulimit core=-1 --security-opt seccomp=unconfined
The first argument set the size limit of core dump files to unlimited. The second one allows
ptrace APIs inside the Docker container so that
gdb has enough previledges.