I compiled Turbo Bob (a build system) with a bug that prevented entering a build container, and since Turbo Bob is built with itself, I could not build a working binary again - chicken-egg problem.
(Of course I could download an old release, but I had some unreleased work I was benefiting from that the old binary included, and while Bob builds fine with just Go compiler, I didn’t want to install Go tools on my host system only to fix this minor issue. And besides, this was a fun challenge.)
I wanted to get the old and working binary back again.
Luckily I had the old working version of Turbo Bob’s binary processes still running for
other projects, and I knew I can dump running process’ binary from
First, identify PIDs for processes we’re interested in:
$ ps aux | grep bob vagrant 8141 0.0 0.0 711956 7708 pts/2 Sl+ Jul30 0:00 bob dev vagrant 14780 0.0 0.0 711956 7056 pts/3 Sl+ Jul31 0:00 bob dev
14780. The Linux proc filesystem contains process’s binary in
I thought I could just
$ cp /proc/14780/exe /usr/local/bin/bob and call it a day, but
exe entry is a symlink:
$ file /proc/14780/exe /proc/14780/exe: broken symbolic link to /usr/local/bin/bob (deleted)
I knew that since the binary is running, it has to keep handles somewhere for the old binary, because the process’ binary file is not physically removed from the filesystem until all references to it are closed.
Next I inspected the proc filesystem’s
maps (“A file containing the currently mapped
memory regions and their access permissions.”):
$ cat /proc/14780/maps 00400000-00894000 r-xp 00000000 fd:00 657514 /usr/local/bin/bob (deleted) 00894000-00d95000 r--p 00494000 fd:00 657514 /usr/local/bin/bob (deleted) 00d95000-00ddd000 rw-p 00995000 fd:00 657514 /usr/local/bin/bob (deleted) 00ddd000-00e0e000 rw-p 00000000 00:00 0 c000000000-c004000000 rw-p 00000000 00:00 0 7fe96e6a8000-7fe970a59000 rw-p 00000000 00:00 0
00400000-00894000 looks promising (notice it has an executable bit).
I could dump it out with sudo:
$ sudo cat /proc/14780/map_files/400000-894000 > maybe_bob $ file maybe_bob maybe_bob: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
Why did I use
maps instead of
fd (file descriptors)? While any open files by the process
should be found from
/proc/<id>/fd/..., I suspect the process’ binary is special
because technically it’s not a file opened by the process, since the binary must exist before
there even is a process that could open any files.