Debugging across the FFI boundary

Over the past few years, I've found myself working quite a lot with UniFFI, particularly in projects where Rust is used to power native mobile and desktop applications.

In this kind of architecture, Rust libraries often contain most of the application's domain logic, while the native clients are primarily responsible for the UI and integration with platform-specific APIs.

This setup works quite well, but it can make debugging a bit tricky, especially when you're working on the Rust side of things. Sometimes an issue only shows up when the SDK is used by a native client, meaning that reproducing it in isolation isn't always straightforward. Ideally you'd want to debug the Rust code directly while the application is running, just as you would with any other Rust application.

Turns out you can, otherwise I wouldn't have bothered writing this up.

Since the Rust library is compiled into native machine code and runs as part of the same process as the application, we can use LLDB to attach to that process, set breakpoints in Rust, and inspect what's actually going on.

A bit of context

A few days ago, I was looking into an issue in the Matrix Rust SDK related to how ignored users were being loaded. Basically, the Rust SDK was sending an empty list of users across the FFI boundary, when it should have been sending a populated one.

The first thing I did was build the Rust SDK, pick a client (Element X, in my case), and reproduce the issue. I set a couple of breakpoints in Xcode and confirmed that the observer was indeed receiving an empty list.

Looking at the Rust implementation, I noticed that the observer should only publish events when changes occur. At that point, I wasn't really expecting it to emit anything, so I wanted to figure out where this event was coming from.

Now, I could start adding logs throughout the Rust code, rebuild the SDK, run the client again, and hope I've logged enough information to figure out what's going on. But that gets tedious pretty quickly (been there, done that), especially when you're not entirely sure where the issue originates.

What I'd really like is to pause execution right before the event is sent, inspect the values, look at the call stack to understand how we got there, and do all of that without having to rebuild the SDK every time I want to inspect something else.

This is where LLDB comes in.

Attaching the debugger

First things first, make sure you know the process name or PID you want to debug. In my case, the process name is ElementX.

Next, let's launch lldb:

lldb

You can attach the debugger to a process that's already running. So if your app is already open, you can simply run:

(lldb) process attach --pid <PID>

Or you can use the process name, like me:

(lldb) process attach --name <process-name>

Note that you can only have one debugger attached to a process at a time. If you launched your application through Xcode, chances are its debugger is already attached.

You have two choices: use LLDB from within Xcode, or launch the application without Xcode's debugger and attach from the terminal.

Killing Xcode makes me happy, so that's what I do whenever I get the chance.

There's also a useful flag, that comes handy especially in my case. Since I'm investigating something that happens at startup, I'd like to attach as early as possible. This is where --waitfor comes in:

(lldb) process attach --name <process-name> --waitfor

This tells LLDB to wait for a process with that name to launch and attach as soon as it can. Once LLDB attaches, execution will be paused, which gives us a chance to set our breakpoints.

Time to launch your application in your simulator.

Setting breakpoints

Now for the interesting part. We want to set a breakpoint in the Rust code. In my case, I'm targeting the code responsible for sending updates to our ignored-users observer.

LLDB supports a few different ways of setting breakpoints, and the documentation is pretty good. I usually prefer specifying a source file and line number.

(lldb) breakpoint set --file /absolute/path/to/matrix-rust-sdk/bindings/matrix-sdk-ffi/src/client.rs --line 1940

If LLDB manages to resolve the breakpoint, we should see something along these lines:

Breakpoint 1: where = ..., address = 0x...

But since we're attaching very early in the application's startup, we might instead get something like:

Breakpoint 1: no locations (pending).
WARNING: Unable to resolve breakpoint to any actual locations.

Chill. This usually means LLDB hasn't loaded the image containing our Rust code, so it doesn't know where to place the breakpoint.

Just trust the process, if everything is set up correctly, LLDB should resolve the breakpoint once the relevant image is loaded.

One thing worth mentioning is that LLDB relies on debug information generated when compiling the Rust library to map source locations back to machine instructions. This is why building the SDK with debug information is important. If you encounter issue with your brakpoints and they are still pending after you resume execution, this is where I'd invest my time in, you're probably using an optimised build of the Rust library and missing some debug symbols.

Now we can resume execution:

(lldb) continue

In my case, LLDB resolved the pending breakpoint and stopped right where I wanted it:

Process 77914 stopped
* thread #29, name = 'Task 5', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
      ...
   1937	        let mut subscriber = self.inner.subscribe_to_ignore_user_list_changes();
   1938
   1939	        let user_ids = subscriber.next_now().into_iter().map(Into::into).collect();
-> 1940	        listener.call(user_ids);
   1941
   1942	        Arc::new(TaskHandle::new(get_runtime_handle().spawn(async move {
   1943	            while let Some(user_ids) = subscriber.next().await {
Target 0: (ElementX) stopped.

There you go. We've stopped inside the Rust SDK while it's running as part of an iOS application.

From here, we can start inspecting the values being sent, look at the call stack, and figure out what's actually happening.