Chromium Notes

Tracking down a mysterious Windows crash

2011-08-31T21:34:00Z

Today's post is a guest post from Eric Roman. He wrote a slightly more expletive-laden version of this post inside Google and I asked him if I could post it here. It serves as a good illustration of how deploying software on more than a hundred million different users' computers is a nearly-biological fight.

Understanding stability of Windows applications is really hard.

Recently, @apatrick has concluded a heroic investigation into one of Chrome's most mysterious top crashes for Windows. He just committed a "fix" for it on the canary channel, which seems to be working!

This debugging journey is pretty epic, so I'm giving a blow-by-blow summary of it below. If you want to jump straight to the conclusion, see r96807 for the spoiler (can also read comments in bug 81449 for the full novella).

Prelude

Our story begins a year ago, when we first began tracking crashes in a generic RunnableFunction<>::Run() location. These crashes had established themselves as a top browser crash for Windows Chrome. At the time, Huan and I unsuccessfully looked into the issue, but couldn't make heads or tails of it (bug 54307). The basic format of the crash looked something like this:

0377fd78 0377fdb0 0x0
0377fd7c 0254c3c3 0x377fdb0
0377fd84 021ba1ae chrome_1c30000!RunnableFunction<void (__cdecl*)(void *),Tuple1<void *> >::Run+0xc
0377fd8c 021baa21 chrome_1c30000!`anonymous namespace'::TaskClosureAdapter::Run+0xb
0377fdb0 021baaa6 chrome_1c30000!MessageLoop::RunTask+0x81
0377fdc0 021bae47 chrome_1c30000!MessageLoop::DeferOrRunPendingTask+0x28
0377fdf8 021d1b24 chrome_1c30000!MessageLoop::DoWork+0x80
0377fe24 021ba960 chrome_1c30000!base::MessagePumpDefault::Run+0xc2
0377fe30 021ba8e5 chrome_1c30000!MessageLoop::RunInternal+0x31
0377fe38 021ba7d9 chrome_1c30000!MessageLoop::RunHandler+0x17
0377fe58 021c6530 chrome_1c30000!MessageLoop::Run+0x15
0377fe5c 021c6643 chrome_1c30000!base::Thread::Run+0x9
0377ffa8 021c1a2f chrome_1c30000!base::Thread::ThreadMain+0xa1
0377ffb4 7c80b713 chrome_1c30000!base::`anonymous namespace'::ThreadFunc+0x16
0377ffec 00000000 kernel32!BaseThreadStart+0x37

Above we can see that the crash is due to jumping to instruction pointer of 0. And judging by the top frames, it looks like there was some stack corruption at work (notice how the first frame's alleged return address is actually a stack location).

The callstack itself isn't terribly helpful though, since we can't tell what code was actually running prior to the crash (it was gobbled up by the stack corruption).

Moreover, the source location of RunnableFunction<>::Run() doesn't help narrow things in the slightest, since pretty much all of Chrome's code runs through this path (Chrome relies heavily on message passing to post asynchronous tasks to another thread's message loop).

Really, all we know at this point is that "some task" got posted to "some thread", and then crashed at "some point" after running this task.

There is one interesting piece of information that can be inferred from the minidumps: based on the crashed thread's index, it is likely the crashing task was running on Chrome's "Child process launcher" thread. In fact, a subsequent instrumentation (r58786) confirmed that all of these crashes were happening on the child process launcher thread. Since there is very little code that legitimately runs on this thread, we did a full code-flow analysis of all the paths that could post tasks. But we did not discover any problematic codepaths (I was hoping to stumble across something bad like a use-after-free).

Without any extra leads, (as well as a temporary dip in the crash's frequency lowering its priority) the mystery bug got pushed onto the back-burner.

It lay dormant for the next 9 months, waiting for a new champion to take up arms.

Part II

In May, Al Patrick (an innocent bystander), is assigned the bug on suspicion that it is a regression. (At this point the callstack has morphed a bit due to various optimization ambiguities, but it is essentially the same bug I had failed to solve earlier).

Our new hero starts off by adding some instrumentation trying to see if a bad function pointer is ever being directly passed to a runnable function (r85359). This doesn't turn up anything salient.

Next he instruments posted tasks to retain the location where they got posted from (r85991). This change is absolutely brilliant! I have benefited from it many times since it was introduced, to help debug crash dumps.

The instrumentation is cheap yet effective: whenever posting a task, the FROM_HERE macro (that was being used in debug builds to save filename/line numbers) now saves the instruction pointer into the PendingTask. Later when the task is de-queued by the target thread's message loop, this same instruction pointer (i.e. the birthplace of the task) is pushed onto the stack prior to calling the task's virtual function. That way should a crash happen later, you can poach the address of the birthplace off the stack during postmortem dump analysis!

This instrumentation reveals that the problem tasks were posted by ChildProcessLauncher::Context::Terminate(), suggesting that the task itself was a runnable function on ChildProcessLauncher::Context::TerminateInternal().

So far so good, but we still have no idea why it is crashing.

Next, our hero instruments the base runnable function/method tasks in Chrome to detect use-after-frees (r86447), as well as other memory molestation (by preserving the value of the function pointer into the minidump). This instrumentation reveals that not only was the task alive and well at the time it was run, but the function pointer was also untouched. This is a major breakthrough in the investigation, since it tells us conclusively that whatever craziness has happened, it occurred while executing ChildProcessLauncher::Context::TerminateInternal().

This is where things start to get weird. Looking at TerminateInternal (the function that is blowing up) there is hardly any code:

chrome_1c30000!ChildProcessLauncher::Context::TerminateInternal:
025e0cea push ebp
025e0ceb mov ebp,esp
025e0ced mov eax,dword ptr [ebp+8]
025e0cf0 mov dword ptr [ebp+8],eax
025e0cf3 test eax,eax
025e0cf5 je chrome_1c30000!ChildProcessLauncher::Context::TerminateInternal+0x16 (025e0d00)
025e0cf7 push 0
025e0cf9 push eax
025e0cfa call dword ptr [chrome_1c30000!_imp__TerminateProcess (02ba56f4)]
025e0d00 push esi
025e0d01 lea esi,[ebp+8]
025e0d04 call chrome_1c30000!base::Process::Close (0289e055)
025e0d09 pop esi
025e0d0a pop ebp
025e0d0b ret

And the code for base::Process::Close() which it references is also pretty simple:

chrome_1c30000!base::Process::Close:
0289e055 cmp dword ptr [esi],0
0289e058 je chrome_1c30000!base::Process::Close+0x1d (0289e072)
0289e05a push edi
0289e05b mov edi,dword ptr [esi]
0289e05d call dword ptr [chrome_1c30000!_imp__GetCurrentProcess (02ba56f0)]
0289e063 cmp edi,eax
0289e065 je chrome_1c30000!base::Process::Close+0x19 (0289e06e)
0289e067 push edi
0289e068 call dword ptr [chrome_1c30000!_imp__CloseHandle (02ba561c)]
0289e06e and dword ptr [esi],0
0289e071 pop edi
0289e072 ret

So basically all that we are doing is killing the process, by calling a handful of win32 API functions. Hmmm.

Al theorizes that someone may be hooking one of the winapi calls (perhaps kernel32!TerminateProcess), and that whatever code it is running in response to that call is responsible for the stack corruption. For instance if the hooker used the wrong calling convention (not stdcall), that could be corrupting our stack upon return!

So how is this guy "hooking" the API call?

There are a lot of different ways you could intercept Windows API calls, and I am definitely no expert to explain them all. You could for instance do things like directly patch the code in the system DLL (in user land, or even on disk). Or re-write the binary to substitute all the target function calls with your new one. But definitely the simplest and most intuitive way to hook is to just patch the import address table. (You can see how that works in the code above -- our compiled code doesn't call directly into kernel32, but rather it jumps to the address listed from the import table table... that is the address you would be patching if you wanted to intercept the call).

To this end, Al adds yet more instrumentation, this time to try and detect if TerminateProcess is being hooked via the import address table (r96266). Unfortunately this instrumentation doesn't reveal any smoking gun yet.

Ricardo makes a good observation -- a hook on CloseHandle() is perhaps more likely than a hook on TerminateProcess, based on the position of the 0 that appears on the stack (bug 81449). Plus, there is probably more value in hooking CloseHandle over TerminateProcess.

Finally, Al commits a changelist that bypasses the address table altogether for both CloseHandle and TerminateProcess, and instead calls the underlying implementation in ntdll.dll directly: r96807. This is not quite as efficient, but not a huge deal either due to the low frequency of these calls.

This workaround appears to have thwarted the bad hooker, and so far there hasn't been a single crash of this sort in the Windows Chrome canary!

It remains unclear which of the two was being hooked or why. The workaround is a pretty ridiculous thing to have to do, but the bypass could shave off as much as 10% of our Windows browser crashes (yes, this crash really was that high in some releases)! It is likely the hooker is malware -- we could copy a fragment of the hooked code into our minidump to try and learn more.

Ideally we want to alert the user about these sorts of problems, rather than papering-over them with workarounds (since they indicate a real problem with their underlying system). But we don't have a good mechanism to do that yet (see bug 72239 for some proposals).

In summary, kudos to @apatrick for his excellent and persistent debugging investigation over the past three months.

Also this shows how powerful Chrome's Canary channel is, since it allows you do this style of experimental debugging with very quick turn-arounds (mere days).

Static initializers

2011-08-29T19:42:00Z

Globals and singletons are already well-known as a design antipattern, but they have an interesting additional cost. Consider a global (I include file-level static in this category) value that has initialization code. That code must be run at startup (which leads to the static initialization order fiasco, though that is not the point of this post).

Because this initialization code is run at startup, before even main() is entered, it is in the critical path for startup. It turns out that even simple code must be paged in off disk, which can lead to disk seeks, and disk seeks murder your startup performance.

This is not hypothetical: with ChromeOS we found that innocuous-seeming static initializers in Chrome were actually affecting the bottom line of startup performance. (Note: that observation comes from a coworker; I'm not sure whether he was using a non-SSD machine at the time or if it also happens on SSDs. Just guessing, but paging in more code, especially code that is non-contiguous, must have some non-zero cost even on the SSDs that ChromeOS relies upon.)

Because of this cost we attempt to track static initialization on our performance bots and prevent new checkins from adding more. (Ideally we'd remove them all but progress is slow.) I recently looked into how this works and I thought it'd be useful to write it down before I forget.

How constructors are implemented

The compiler creates, for each object file, a function that contains the constructors for the file. Pointers to these functions are collected in a table at link time. At startup, __do_global_ctors_aux iterates through the table and calls each function. (Here's a nice page that walks through the disassembly.) Conceptually, to judge the cost of all static constructors you might want to do something like sum the size of all of these functions, but for our purposes we care about disk seeks; even doing more work in a single static constructor is fine if we reduce the total number of functions paged in, which means the size of the constructor table is the statistic of interest.

The table of functions shows up as the .ctors section of the executable. You can dump table via commands like (note that the first entry is -1, the rest are addresses):

$ objdump --full-content --section=.ctors path/to/binary

or in gdb,

(gdb) x/1000xg &__CTOR_LIST__

The gdb output is perhaps useful since it will decode little-endian for you. (N.b. that "g" trailing the "x" command prints 64-bit pointers; adjust as necessary locally.)

For a Chrome binary I glanced at the ctor list appears to be in pointer order, which means you can see how much of the resulting binary they span by subtracting the last entry from the first. From my random debugging build: 30mb, not good.

Constructors versus static initialization

Note that data that is initialized to a constant is implemented in a different way: the constant value can just be placed in the right place at compile time, so there is no cost. In contrast, C++ objects that have constructors involve code and must be computed at runtime. You'll also sometimes encounter code that initializes variables with function calls (like we did with the mysterious IcedTea crash).

You might also notice that static data can be shared between multiple instances of the same executable, while initialized memory is private; see my post about how memory works for more on that.

I noticed with some interest that the Go programming language, designed in part by compiler hackers, neatly sidesteps some of the above problems: by defining initialization order carefully ("The importing of packages, by construction, guarantees that there can be no cyclic dependencies in initialization.") and by only allowing simple values as constant initializers. See their manual for more.

What to do about it

Mozilla hackers have found that Linux is pathologically bad in how it runs the resulting ctor list, and it looks like they have at least considered fixing that manually. We have chatted about doing the same, but fundamentally I believe the way to keep startup fast is to do less. See also my earlier post about performance.

It appears that the generated functions that run these constructors get names starting with _GLOBAL__I_. This means a call like

$ nm out/Debug/chrome | grep _GLOBAL__I

will dump a list of all files that have a global constructor. Go delete some code!

The zygote process and software updates

2011-08-03T17:59:00Z

When you make a new tab Chrome (usually) starts a new process for that tab. How is this done? It would seem natural to just fork(), but fork can't be used safely in the presence of threads. fork only forks the current thread but other threads may be holding locks (including e.g. inside glibc or in the allocator) which would never be released after the fork.

If you are careful to not touch anything after a fork, it can be safe to immediately exec. This matches the process launching model on Windows (no fork, only fork+exec), with the negative that it forces the overhead of startup again on each new process. (Code reference: LaunchProcess(), which also knows to e.g. use _exit instead of exit.)

Forking and execing ourselves is how we spawn subprocesses on Mac (I believe; there may be some trickery related to how app bundles work that complicate this). On Linux it is unfortunately more complicated. Updates on Linux are managed by a systemwide package manager that runs independently of other software, which effectively means at any point any file you rely upon can be silently clobbered. (This problem even affects single-process apps like Firefox; an update will clobber some JavaScript used in the UI and suddenly things will either crash or get weird.) In Chrome's case, if Chrome binary itself is updated while the browser is running, processes spawned by the running Chrome would be the newer Chrome, which may have made an incompatible change to the interface between Chrome processes.

Instead, at startup, before we spawn any threads, we fork off a helper process. This process opens every file we might use and then waits for commands from the main process. When it's time to make a new subprocess we ask the helper, which forks itself again as the new child. By virtue of always forking from the same initial process, we guarantee that we are always running the same code; even if the files we opened are replaced by a system update our handle on them is the handle for the previous file. (That works as long as nobody overwrites the contents of the file we have open; thankfully, package updates write a new file and rename it over the old name, leaving our open copy the only remaining reference to the old file.)

(Code reference: ChildProcessLauncher's LaunchInternal(), the gory ifdef soup used when launching a subprocess. Truly some ugly code.)

This solution is both clever and an ugly hack. Any time someone adds code to Chrome that interacts with a file on disk they either need to be aware that they need to preemptively open it or they will produce mysterious failures across updates (in practice, usually the latter; e.g. bug 35793: Devtools stop working when chrome gets updated). An interesting question to ask is: why is this not a problem on Windows and Mac?

On Windows, files are locked if any process is using them, which forces a design where updates install into a separate directory. But -- annoyingness of locking aside -- in fact I think that design is preferable. To start with, a given version of Chrome will know its files will remain unmolested by updates. Furthermore, when an update happens, the updater can write out a separate "update succeeded" sentinel after writing all the files out, making impossible for an aborted update to leave both the previous and next version in a half-working state. (On Mac, we take a similar approach; I don't know enough about Macs to know whether the versioned directories within bundles make this magically work.)

With all this in mind you might reasonably ask why Linux needs to be special: why we waste memory on this zygote process launcher and have extra buggy codepaths just to support an inferior update model. (Note that by using .deb files we also lose our tiny incremental updates.)

And to that I can only answer the thinking we had at the time: one, we wanted to be good citizens on Linux; one distinction between "lame port of a Windows app" and "real Linux software" is exactly whether you distribute as a tarball or as a package. Secondly, and more importantly, we knew that regardless of what we did for Google Chrome the Linux distros would attempt to stuff Chromium into their package manager even when they know it breaks the app, much like they've done to Firefox. Now that I've summarized it in these terms it sounds a little depressing, but there it is; with ChromeOS where we control the stack we have more intelligent updates.

Data visualization and d3

2011-07-29T23:32:00Z

Lately I've been learning the awesome d3 library for data visualization. I usually only publish my toys internally, but only because it's convenient; I'd rather put them online so others can play with them.

First up: lines spent.

RTL titles

2011-04-19T18:26:00Z

(Here's a post from some months ago. I think I'm not writing new posts because I've been sitting on this one for so long, so perhaps it's for the best that I just publish it.)

I've been away for a while. Part of my travels involved a hackfest in Tel Aviv for right-to-left text in WebKit. My RTL knowledge of WebKit is minor -- I have done a decent bit of hacking on the Linux Chrome rendering code for complex text in Webkit -- but as a language enthusiast (I like to tell people: by college degrees, I am as qualified a linguist as I am a computer scientist!), I am already familiar with the bidi algorithm and I've studied some Arabic.

(I know some properties of Hebrew but not the alphabet, but in Tel Aviv all the street signs are Rosetta stones of Hebrew/Arabic/Roman scripts; by the end of my week I proudly identified and boarded the shared taxi to Jerusalem by reading the Hebrew sign in the window.)

Upon arriving in Tel Aviv, we assigned the important bugs to the more talented WebKit hackers, while I picked a relatively minor bug in the hope that I could churn through a few of them. By the end of the week I didn't even fix that one bug. I did, however, make two refactoring pre-changes and got another one in just at the finish line that touched 53 files. Somehow with me it is always a yak shave.

The bug seemed pretty simple: WebKit doesn't understand <title dir="rtl">RTL titles</title>.

To start with, what does this actually mean? Here's an attempt to explain as briefly as possible, glossing over details; bidirectional text is actually pretty gnarly.

First, some background for bidi beginners. You should know that text is always stored in its logical order: the first letter of an in-memory string of Hebrew is the first letter of where you'd start reading, on the right. The same is true for the order of the characters as written in an HTML document. When rendering a string that contains both right-to-left (RTL) and left-to-right (LTR) text, you end up "reversing" bits of it. The algorithm for this reversal is called the bidi (bidirectional) algorithm, and it is complex and interesting but out of scope for this post.

In discussions of bidi, the convention is to write the characters that should be RTL (representing a language like Hebrew) in uppercase, and the LTR characters in lowercase. So, for example, the in-memory string foo BAR XYZ should appear as foo ZYX RAB. Critically, note that the last word of the source string ends up in the middle of the rendered string -- ommitting many details, at a conceptual level you're laying out a string of chunks left to right and the BAR XYZ chunk should be rendered right to left as part of that.

I've already made an assumption there, though: I wrote that you're laying out the chunks left to right, but in a right to left document the overall layout order goes the other way. (The document starts from the right, after all.) foo BAR XYZ in a Hebrew document should render as ZYX RAB foo. This extra bit of of metadata about the text -- inventing a term, the direction context of the string -- is just what the dir attribute of the title is for.

WebKit can display plenty of RTL sites just fine, so it necessarily gets the all of the above details correct in web content. The problem that my bug was about is that WebKit generally isn't responsible for rendering title tags -- they're handled by the browser.

Aside: amusingly, a stylesheet containing
head { display: block } title { display: block }
will cause the title to display in the page. It sorta makes sense as soon as you see it.

For the browser to render the title correctly, it needs the same information that WebKit has -- the text and its direction. (We had many discussions about the proper way to display an RTL title in an LTR browser -- e.g., do you move the favicon too?) To fix this bug I "just" needed to (1) get the direction as specified on the tag (or any parent of the tag, or CSS, or ...); (2) plumb that extra metadata out to the browser; (3) make use of that extra metadata browser-side (e.g. flip the text layout direction when appropriate).

Unfortunately, titles are used in a variety of places within WebCore -- as an attribute of documents, of course, but also in data structures related to loading pages, history, and in APIs used to communicate state information up to the hosting environment. Touching all of these resulted in a larger patch than I'd anticipated.

I first landed some managable chunks of it: a small refactoring and then a larger one; it's good I landed them separately because I got the logic in the latter wrong and needed to quick-fix it (in my defense, part of the problem is that the code has a related bug).

Then comes the monster change-the-world patch where I swap out the type of a core object; despite my best efforts at modifying nine WebKit ports simultaneously I managed to break the GTK build and Qt build and the Qt build a second time.

That got me to the point where the data was exposed to the WebKit-internal platform layer, but not through any public APIs; so next was exposing that through the Chromium WebKit API, which is also the layer the testing interface uses so it allowed me to write a test. And finally, I screwed up that patch too, necessitating another quick fix.

And with all that in place, I then turned to the Chrome-side implementation...

...and discovered the chicken-and-egg problem of new HTML specs: because nobody implements <title dir>, few sites made use of it. In fact, I did find a site that frequently had titles that mixed LTR and RTL text, exactly the sort that would benefit from my change, and the site did use the dir attribute on a <title> tag, despite it not having any effect in browsrs -- but the site used it in such a way that was exactly backwards; my locally patched browser that obeyed the attribute made the site strictly worse. The site? Google Israel.

And with no better conclusion than that, this post has sadly languished unpublished on my laptop and the work is in a similar state. Now that I look, it seems my bug report about fixing Google Israel was fixed, perhaps it's worth again trying to land my patch. But at this point I am suspicious of the sunk cost fallacy: I had picked this bug because it was so minor that we had guessed it would be easy, and it's perhaps not worth much more time when I have a limitless stream of more important bugs.

Your reward for reading this long and anticlimactic post is an example of one of the many ways software can get RTL wrong. In this image, I constructed a page with a specially-crafted title, which Chrome naively formats as "$PAGE_TITLE - Google Chrome" and then hands it on to the OS. The image shows what happens when I alt-tab.