Hunting down TestFlight bugs

Standard

Preambule

Recently, I went into one of the most annoying issues, that I’ve had on my work so far. Basically, problem was following: we’re about to release new application to the store. For this purpose, we prepared build for TestFlight and submitted it to for the external testing.

And then – KABOOM – testers (multiple of them) reported that the application is crashing literally on the startup. So, user launches the application and it instantly crashes and closes itself.

After almost week of error and trials, I managed to resolve issue. And purpose of this post is to share this experience, so, maybe, someone will resolve own issue quicker then I did.

Getting to work

So, first of all, – make sure that you’re not able to reproduce this crash itself:

  1. Edit scheme -> change configuration to “Release”
  2. Run application
  3. Try to find out crash

Ok, if you was able to reproduce the crash, great, you can fix this and skip rest of this post.

In my case, I wasn’t able to get away so easily.

Well, this didn’t work – what we’re going to do next? We’re going to find out crash logs from TestFlight.

Finding crashlogs from TestFlight

In case if iTunes Connect account is not yours and belongs to your company or client – grab credentials and add this account to your xCode:

  1. Click “Xcode” menu
  2. Choose “Preferences”
  3. Select tab “Accounts”
  4. Click “+” on the bottom
  5. Select “Add Apple ID”
  6. Enter credentials that you’ve grabbed

If everything is alright, you should be able to see crash logs from TestFlight now:

  1. Click menu “Window”
  2. Select “Organizer”
  3. Select tab “Crashes”
  4. Select build that was crashing
  5. Wait a second and list of crashes should appear

Ok, now you should have crashlogs. If you’re lucky enough – they’re already symbolicated for you (i.e. it’s not just addresses in memory, but actual names of functions in code).

If they’re actually not symbolicated you will see something like this:

edited_crash_log

I have removed name of project (since it’s project from my work), but it doesn’t really matter here. As you can see here, instead of function name and line in the code, there are only raw addresses in memory. It’s not symbolicated crash log and if you get something like this – next section will explain how to symbolicate it.

Symbolication

Well, I wasn’t that lucky, so I had to symbolicate it by myself. If you’re also not lucky, here how you can do it:

  1. Go to iTunes Connect
  2. Log in with credentials for application
  3. Click “My Apps”
  4. Select “TestFlight” tab
  5. Select “iOS” on left menu
  6. In list select build that was crashing
  7. Click “Download dSYM”

Ok, now you should have symbols that will help you to symbolicate crashlog. First of all, you have to get raw file of crash. In order to do so, do following:

  1. Return to Organizer to “Crashes” tab
  2. Right click on crash that you want to symbolicate
  3. Choose “Show in Finder”
  4. You should get Finder window, with file selected (extension: “xccrashpoint”)
  5. Right click on it and choose “Show Package Content”
  6. Navigate down on folders until you get to the folder containing “.crash” files

Basically, .crash file is a simple text file that contains all details related to crash. Copy any of this files to your working director. I would suggest to rename it into something more shorter, so it will be easier to write to Terminal. I used “temp.crash”.

Now you have to find matching dSYM for this crash. Open crash log and find following line: “Binary Images”. Right after this line you should have line with your application name in it and something like this “”. This is UUID which you can use to find out which dSYM corresponds to this crashlog. Image below depicts place (inside of black rectangle) where you can find it:

corrected-uuid

Navigate to folder with downloaded from iTunes Connect dSYMs and find here dSYM with UUID that was in crashlog. Note that names of dSYMs are in upper case and are separated by “-” (for image above correct dSYM file have name “503C13BD-BB1A-3885-ADB6-82CE15C90BF8.dSYM”). It might be hard to find out dSYM, but you should do it rather quickly. When you will find it – copy it to the working directory, where you copied .crash file previously. I would again suggest you to give it shorter name – I used “temp” for this.

Now the last step: symbolication itself.

First of all, you should find out exact path to “symbolicatecrash” on your system. You can do it in following way:

  1. Open Terminal
  2. Execute: “cd /Applications/Xcode.app”
  3. Execute: “find . -name symbolicatecrash”

You will get something like this: “./Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash”.
Note that it is relative path. To make it absolute just replace dot at the start with:
“/Applications/Xcode.app”
so you will get:
“/Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash”

Note that path may vary for different Xcode’s versions (path above for xCode 8) – so this method is more robust to find out exact path for your version.

Now with that path navigate to your working directory (where .crash and .dSYM files are located) and do following:

  1. First of all run: “export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer”
  2. Then run: “/Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash (or path that you get above by “find”) temp.crash temp.dSYM > report.txt”

If everything is ok, after few seconds you should get symbolicated crash log (in file called “report.txt” in this case, you can replace it with any other name). Check it, you are searching for crashed thread. Analyze crash log and try to figure out what exactly went wrong.

Unfortunately, for me it wasn’t enough.

Magic crashed line

After symbolicating of crash log, I realized that it’s pointing to line that is not present in code. So how we are supposed to resolve such issue?

Basically, such problem may be related to optimizations. Solution here is to turn it off:

genious

So, here what we’re going to do:

  1. Open your project settings in xCode (not target, top level)
  2. Open “Build Settings”
  3. Search for “Optimization”
  4. Turn to “None” Optimization Level for code generation (there are two places, for Swift compiler and Apple LLVM)
  5. Prepare new build for TestFlight and submit it
  6. Ask testers to reproduce crash

After testers will confirm that crash happened on new build – go to Organizer once again and find this new crash. Now, it should be way more verbose then in previous version. Symbolicate it and you should get all possible information related to crash.

Basically, turning off optimizations helped me to find out root cause of crash. Hopefully, it will do so for you.

Good luck with bug hunting!

References

  1. Tutorial by Christopher Hale on iOS crash symbolication