Translating Dropbox

¡Hola, mundo! Welcome to our new engineering blog. We'd like to start with a post on i18n, because aside from being exceedingly fresh in our minds right now, hearing our take on it might be useful to other startups. Going global can be an intimidating prospect, especially if you're like me and illiterate in all the new languages. I'll focus today on assembling a translation team because it's one of the trickier challenges.

Back in March 2008, Facebook's community famously translated the site into French in just 24 hours. Community translation, done right, often offers the highest quality because users know your product inside-out and naturally use the right tone and formality level. Community translation also makes it possible to go from a handful of languages to over 100.

While intriguing, we're holding off on community-led translation for now and going with a hybrid approach instead. The main reason is the substantial engineering investment. We'd need to:

• Build a translation mode that allows volunteers to translate text in-context, as it appears on the screen. This is especially hard for everything not on the website: our Windows, Mac, and Linux desktop apps, mobile apps, emails, etc.
• Determine, at all times, which text needs translation, and which needs to be re-reviewed. This is tricky because if a translated string is reused, but in a different context, it still needs to be reviewed.
• Build a voting mechanism to select the best translations, including an ability to flag offensive translations.
• Prevent large-scale voter conspiracy so that incidents like these don't happen.
• Import/Export translations from/to a bunch of different formats (gettext PO, iOS and Android XML, property lists, static HTML, etc.) Groan.

So it requires a big initial effort. Facebook had a staff of about 500 in March 2008, for example, whereas we have 50. By contrast, professional translators are incredibly convenient and barrier-free. We can send them the latest version of 12 files in 12 different formats, they'll determine the difference from the last version, and send us back the translations, ready to go.

Here's how our approach works: we hire a firm to translate everything beforehand, then ask our generous userbase to review and send us their corrections, iteratively. Our firm reads through every suggestion per English string, one string at a time, and makes the most popular correction. Before sending a feedback round to translators, we quickly skim it on our own first through a special admin page, to do three things:

• Fix bugs, such as untranslated text or incorrectly formatted numbers.
• Evaluate our translators: are people reporting typos, grammar mistakes, and gross mistranslations? Or is all the feedback subjective?
• Confirm that previously reported corrections get incorporated: if our translators don't fix a problem, and it continues to get reported, our admin page will throw a tantrum.

Here it is in action. Say I'd like to report a dubious French translation. First I click the green tab to the left of the screen:

Then I type or paste a few letters of the translation I'd like to improve. It pops up a list of matches:

After selecting the translation, the original English text shows up underneath. I type in my improved translation and the reason I like it better. Done.

This feature also exists on experimental builds of our desktop app, released on the forums.

Grouping suggestions by English string is key. It offers good usability: people can type just a few letters, autocomplete the translation in question, and view it alongside the original English string. Further, it organizes everything for easy translator review. One issue complicates things: strings with placeholder variables. For example:

Hello, %(first_name)s!

We want users to be able to autocomplete text as it appears on the page: "¡Hola, Dan!", not "¡Hola, %(first_name)s", and at the same time, internally group all instances of this placeholder string together. You might think it would be as easy as wrapping the gettext _() function, but consider typical gettext code:

greeting = _('Hello, %(first_name)s') % {'first_name' : user.first_name}

That is, placeholder substitution happens outside the _() translation call. Our solution: on the server, we subclass python's unicode builtin type, overriding __mod__ to remember its filled-in placeholders. We then wrap the gettext _() function to do two things:

• Return this subclassed string type.
• Keep a response-wide list of all such returned strings.

At the very end of the response, we take the list of translated strings and serialize into JSON, ready to go for browser-side javascript autocompleter code. For each string, this list includes the placeholder form "¡Hola, %(fname)s" (for bookkeeping), and filled-in form "¡Hola, Dan" (for autocompletion). The reason we keep autocompletion entirely browser-side is that it makes the experience feel instant. Another nice result is the autocompletion menu is always narrowed down to text that appears on the page -- less choices to wade through.

One last complicating detail: AJAX. Some of our AJAX requests, for example in the events feed, contain translated display text. We want users to be able to select text no matter where it came from. To solve, we pack a list of new translations within each AJAX request, similar to the list from the initial request.

• Excellent background on Facebook's system. They use professionals too, more than most people realize.
• Check out Smartling if you're interested in community translation, especially if your content is web-only. They're making community translation easier on multiple levels.

For anyone about to do similar work, I thought I'd give a quick summary of some of the other challenges we needed to cover for today's launch. Let us know in the comments if you'd like to hear more about any of these topics; we'd be happy to expand in future posts.

We encountered some typical i18n problems:

• Translating text inside Python/Javascript/Objective-C/Java, templates, database tables (where our emails live), and static docs like our terms of service. Keeping translations synced with evolving English. Covering string substitutions and plurals.
• Translating Dropbox-specific terminology ahead of time. Everyone loves a good translation blunder. One of my favorites: "Jolly Green Giant," the mascot for Green Giant frozen vegetables, being translated into Arabic as "Intimidating Green Ogre." Here's another disaster. We carefully translated Dropbox terminology before everything else -- phrases like "selective sync," "Dropbox guru," and "Packrat" -- because these are the hardest to get right.
• Supporting both the perverse American date format 4/22/1970 and much more logical German 22.04.1970. Correctly formatting times, numbers, percents, and currency. Thanks babel!
• Formatting people's names correctly. For example, Japanese names are family-name-first, with a space in between if the name is ASCII (Takahashi Yukihiro), and no space for East Asian characters (高橋幸宏). Similarly, for our signup form, we put family name first for Japanese.
• Displaying translated country lists, correctly sorted, in our address form. We use IP geolocation to guess the right country and put it at the top of the list.
• Translating images with embedded text, overdubbing videos.
• Fixing one million layout problems resulting from "overfitting" layouts to fixed English text. Watching long translations completely destroy one's fragile, inelastic layouts.
• Making sure various locale settings -- user preference, cookie, web request, Accept-Language header, desktop app settings -- all work together. For example, if a user changes their locale from the desktop app, and then clicks a web link from within the app, we make sure the website matches the new locale. On the other hand, it would be confusing, and difficult to undo, if changing the website's locale affected the desktop app.

And a few not-so-typical:

• For the desktop app, overriding a class in Python's gettext module to pack translations inside python bytecode (instead of outside the executable, for example /usr/share/locale on Linux). This eases cross-platform concerns and keeps our app stand-alone. In the future, as we add more languages, we might have the server beam down text to the client.
• Building a lightweight gettext-like lib in javascript for browser-side translations.
• Extending the parser for Cheetah, our templating language, to allow English string extraction.
• Word wrapping Japanese on the desktop app. The algorithm is slightly different and has a couple of hard to detect special cases, more info here.
• Collation. It is a hard lesson in life that for most of the world, alphabetical != ASCIIbetical. For example, in German, ä is sorted as if it is two letters, ae, while in Swedish, it comes after z. For the most part, sorting a list of strings is as easy as calling a library. Thanks ICU and PyICU language bindings! Unfortunately, we don't have ICU in browser-side javascript. Supporting locale-sensitive sorting and resorting on the website's file browser, with dynamic insertions for new files and directories, and making it fast, was an interesting challenge. Similarly, we had to manually implement Japanese collation, with its multiple scripts, for iPhone and iPad, and build our own Japanese browser widget to match Apple's -- something missing from the iOS SDK.

This is a totally new world for us, so if you think of any improvements we can make or areas we missed, we'd love to hear about it!