Symbolicating crash logs

Introduction

This post is devoted to a frequent routine of many iOS/Cocoa developers: symbolicating crash logs from users. If you're not interested in details and just want The Solution you can go directly to the section named "symbolicatecrash".

The Solution to what?

Well, according to Apple's Technical Note TN2151 one can symbolicate crash logs just dropping them at Device Logs section in XCode's organizer window. All what they ask you for is a dSYM and .app file somewhere on your disk.

In practice, that's not so smooth. As for me, it never worked as supposed to. Well, it works fine for the latest debug build. But never for any of old AdHoc builds. All you can get from XCode is just symbolication of stack frames related to system libraries, but frequently that's not enough to realize what went wrong.

Symbolication crash logs manually

In the follow chapter I'll try to explain how one can symbolicate a crash log without XCode. Our main tools here will be atos, otool and dwarfdump.

A few words about what is a crash log. That's basically just a plain text file. It contains three sections in essence: headers, stack traces for all threads and binary images information. Headers contain basic info about device, crashed process, versions, date/time and crash reason.

Stack traces look like this:

...
10  YandexMaps                     0x00003a22 0x1000 + 10786
...

First, a frame number goes, then binary image name, then an absolute frame address, finally the same address, but represented as a sum of binary image load address and offset. The process of symbolication is exactly about converting that frame address into a readable symbol name.

The end of file contains a list of all binary images loaded. The list consists of lines like this:

0x1000 -   0x1bbfff +YandexMaps armv7  <257b985709d23dc690477e12bc17fa48> /var/mobile/Applications/97CB76B8-19C5-4266-938E-26669B5CCAD3/YandexMaps.app/YandexMaps

First, image's address range goes. Then image name, architecture, image's UUID and a path, from where the image was loaded. The UUID is actually the most important info here — that's the key to finding a corresponding .app/dSYM file.

dwarfdump

If configured properly, XCode creates a dSYM file together with every app build. dSYM files contains debug information (sadly, for your app only) in a widely used DWARF format. So dwarfdump is a command line tool for dumping info from such files.

First thing you do is finding a corresponding dSYM file for your crash log. The UUID described in the previous section is the key. You can get the UUID from a dSYM file with the following command:

dwarfdump --arch <arch> --uuid <dsymname>

If you store all dSYMs in a specific place, you use grep to find the necessary one:

dwarfdump --arch <arch> --uuid *.dSYM | grep <first-digits>

Finally when you find the dSYM, you can get information about a frame address:

dwarfdump --arch <arch> --lookup <address> <dsymname>

Usually dwarfdump is enough to extract information about all your app's stack frames — you just go line by line. Unfortunately, that doesn't work if you're curious about frames belonging to system libraries. That's where we need the next two tools.

otool and atos

Surprise: there are no dSYM files for system libraries in the SDK. Good news are that we can extract all necessary information directly from binary images. We could do that with our app as well, but usually it's much easier to store just a dSYM file instead of a pair of the dSYM and .app.

Exactly as with dSYM, first we need to make sure that we have a binary image with matching UUID. We can read image's UUID with otool:

otool -arch <arch> -l <binary-image>

The output is quite long but we're only interested in a section like this:

Load command 7
    cmd LC_UUID
cmdsize 24
   uuid FD032EF9-2890-39ED-957A-A99CA51F120C

Forgot to say: you can easily find all necessary binary images in /Developer/Platforms/iPhoneOS.platform/DeviceSupport/.

Now when we have the image we can use it to convert numeric addresses to symbols. That's exactly what atos does. You do that this way:

atos -arch <arch> -o <binary-image> -l <image-load-address> <list-of-interesting-addresses>

The output is pretty straightforward and looks like this:

-[__NSOperationInternal start] (in Foundation) + 652
-[NSOperation start] (in Foundation) + 16
-[NSRunLoop(NSRunLoop) runUntilDate:] (in Foundation) + 56

-l option allows us to use exactly the same address as in the crash log without any additional math.

symbolicatecrash

Originally symbolicatecrash is a script shipped with the SDK by Apple. But exactly as XCode it is too capricious. Thanks to @nskboy, there is a working fork of it. It needs only a dSYM file anywhere on disk (versus dSYM+app required by the original version) and also fixes few nasty bugs. You can get it from GitHub.

The usage is very easy:

./symbolicatecrash -o symbolicated.crash <original-crash-log>

It works stably for me and I'm pretty sure it will for you.