Dropbox owes a large share of its success to Python, a language that enabled us to iterate and develop quickly. However, as our infrastructure matures to support our ever growing user base, we started exploring ways to scale our systems in a more efficient manner. About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed. This was a massive effort–around 200,000 lines of Go code–undertaken by a small team of engineers. At this point, we have successfully moved major parts of our infrastructure to Go.
One recurring theme that hindered our development progress was the lack of robust libraries needed for building large systems. This is not surprising since Go is still a very young language. To address this issue, our team started building various libraries to provide better abstractions, such as connection management and a memcache client. We are very excited to announce that we are open sourcing these libraries to help the broader community build large scale production systems.
We have included several libraries to kickstart this effort. To highlight a few:
- caching: provides a general abstraction for building caching layers.
- errors: enhances the standard error interface and exposes stack trace information.
- database/sqlbuilder: allows developers to generate sql statements programmatically.
- memcache: implements a fully featured memcache client library which supports connection pooling and flexible sharding.
- net2: adds functionality for connection management.
- (and my personal favorite, hash2, which includes a space efficient permutation-based consistent hash algorithm)
We will continue to expand the set of libraries in the repository:
https://github.com/dropbox/godropbox
To make sure that we continue to invest in this open source effort, we are committed to using the public version of this repository internally. We are migrating our internal systems to use the libraries directly from this repository. This ensures all fixes and improvements are applied publicly before they are pulled back internally.
We hope you will join our community and help make these libraries better!
