How the data center site selection process works at Dropbox

For more than a decade, Dropbox has operated its own world-class, exabyte-scale storage system—a multi-metro, hybrid-cloud architecture that spans the globe. But to get to this point, we’ve had to learn a lot about what makes a good data center, and how to pick the perfect site. 

Early on, we leveraged industry relationships, then specialized real estate brokers. As we matured, we brought the site selection process in house. Similar to what you’d see in a construction or supply chain environment, our process mimics a competitive RFP. In more recent years, sustainability has also become key, and our commitment to our environmental goals is now an important factor when making these decisions.

In this story, you’ll learn about the approach we take to data center site selection, and how we balance cost and reliability with our company values. In our last three selection processes, we successfully used this approach to negotiate best in class rates and reliability. Whether you’re an expert at evaluating data center facilities or about to go through the process for the first time, we hope this glimpse into how we work is helpful for your search.

Before we can select a new data center location, we have to define our resource requirements. This is where our capacity engineering team comes in. 

As a first step, the capacity engineering team assesses the types of services we need to support and quantifies the cabinet counts required. From there, the datacenter engineering team defines the following requirements:

• Power. This is based on historical usage by existing hardware, as well as projected usage by future equipment. Usually, this will be expressed in a kW or MW allocation, and includes the total expected IT load of both cabinets and network equipment. 
• Space. Cabinets can vary in dimension, and there should be enough space to store and support a variety of configurations using industry best practices (4-foot cold aisles, 6-foot hot aisles, etc.) 
• Time. The target date by which the team requires capacity to be online and available, commonly referred to as the lease commencement date.

Once we’ve established our capacity requirements, the data center engineering team begins their search by identifying the locations and datacenter facilities that can can support our requirements. We leverage our existing network of providers to identify which are active within the desired market. This is then typically met with a response confirming availability or indicating an alternative power threshold or timeline requirement. (For example, a provider could meet our power requirement, but not until the month or quarter after our required date.)

Depending on the responses we receive and our flexibility with power, space, and/or time requirements, we can then make a decision as to who to include—or exclude—in the next stage of the process.

After the initial round of responses, Dropbox will issue a full RFP document which highlights our facility-level requirements. Here are some of the physical parameters we look for:

• Supporting infrastructure design. Dropbox follows the Uptime Institute’s guidelines for Tier III facility standards. This is an industry-wide guidance for facility design best practices in cooling, power, maintenance activities, and fault tolerances. If deviation occurs, we request the landlord highlight the deviation for further evaluation.
• Expected cabinet weight with dimensions and expected quantity. Equipment is heavy and often exceeds the design thresholds of facilities. The flooring system needs to support both the cabinet weight while moving to the suite, and the weight once deployed in groups on the data center floor. 
• Network design. At this stage of our process, we are most interested in understanding the ingress and egress of traffic. For example, does the provider have our preferred carrier? Are there nearby carriers we could leverage if not? And can the connection be delivered to our space with the appropriate level of redundancy?

We also look at commercial parameters such as rent, utilities, and incentives (more on these later). Identifying these figures as early as possible help us set internal expectations, and allows us to track progress throughout our negotiations.

In recent years, Dropbox has taken a tougher stance on one parameter in particular: power usage effectiveness, or PUE. It is not lost on us that our business runs by consuming electricity—which, depending on the source, can pollute the world around us. PUE is a measure of a how efficient a facility is at supplying power to our equipment while minimizing the amount of electricity required to support it (eg. cooling costs, electrical losses, etc). In negotiations, PUE is represented as a cost multiplier the provider applies to the amount of power we consume; the lower the multiplier, the more efficient the usage. As much as possible, we have endeavored to promote, encourage, and prioritize facilities which utilize a greater amount of renewable energy and create an environment that encourages or mandates best efficiency practices.

An example of this would be ensuring facility providers use eco-friendly cooling units, organize their customers such that hot and cold aisles are strictly respected, and ensure proper airflow containment methods are deployed throughout the facility. While some efforts are more productive than others, ignoring them outright is not tolerable for our company. As such, we leverage the PUE figure as a tool to drive action by requiring a lower PUE to be baked into our contract. This usually comes with stipulations around customer installed containment and electrical consumption thresholds, but this mutual agreement is best for everyone involved. 

Once all responses to our RFP have been received, we complete a bid leveling process in which we assess the cost of each proposal. As part of this process, we typically send out a technical questionnaire. This document poses a series of in-depth questions about the space, power design, cooling design, network design, historical information, site environmental risks, facility security, operational specifics, and staffing. The answers to this questionnaire are critical to understanding what is really being offered. Only then can we conduct a complete analysis of each facility, and identify potential gaps or unique design features within the offer.

Some examples of potential issues we’ve identified based on the answers to our questionnaire include:

• Reduced UPS redundancy due to an alternate technology being deployed. In the event of a power outage, one provider could only give us 30 seconds of emergency power—versus five minutes with a standard UPS setup—because they used an inertia wheel. This technology spins a large metal wheel that produces enough electrical current to support the facility, but only for a very short duration, drastically increasing the importance of a quick generator startup during utility failure events. 
• Increased risk due to construction delays. This is a very common industry hurdle. As with all construction timelines, no one can be certain that all components will be available and installed in a timely manner. If your need-by date is immediately after power is available, a construction delay could mean the difference between meeting your deadlines or missing them entirely. 
• Inadequate monitoring programs, which would not have provided the necessary facility alerts. Part of our selection process is ensuring that we have visibility into facility-level alerts (eg. generators turning on, UPS losing utility power, air handler units becoming unavailable). The general preference is to have an automated alerting system, rather than rely on humans to raise issues, which can introduce delays or even errors in how we respond.

After our questionnaires have been completed, our team selects 4-6 providers we think are the most commercially and technically viable participants. We then validate these providers in-person by completing a site walk evaluation.

During these evaluations you may see construction in progress, which will allow you to probe into the order, shipment, delivery and installation timelines associated with any pending equipment. In situations where you are evaluating a completed facility, it may be helpful to validate that the equipment presented in the questionnaire is the same that is in place. We have uncovered many less than desirable situations which would have gone unnoticed had we not been on site to physically validate. 

In one recent example, a provider told us their dock could receive a full-sized 53-foot trailer, but turned out to require a forklift or tailgate. In another case, while visiting a facility that fairly represented its physical parameters, we found exposed fiber optics on the exterior of the building and—believe it or not—wildlife actually living inside the building.

After completing our in-person visits, the team will stack rank each facility on individual parameters—a deceptively simple process that actually involves finely-tuned formulas and a careful system of weights. While all facility elements are important, giving adequate weight to nonnegotiable or highly critical items ensures we prioritize our most desired design requirements. As part of this process, we give each provider a numerical score in the following areas:

• Space. Will the space fit the intended rack count, support the anticipated cabinet loads, have a suitable loading dock, require lifts or elevators, etc.?
• Power. Does the electrical offering provide the desired amount of redundancy, include an acceptable grounding system, sufficient upstream redundancy, adhere to the desired efficiency levels, etc.?
• Cooling. Does the mechanical offering provide the desired amount of redundancy, operate at an acceptable level of efficiency, operate without the consumption of water, etc.?
• Network. Does the network infrastructure provide the means to maintain network redundancy at both a campus and building level, adhere to building best practices through physical separation and appropriate pathway routing, etc.?
• Security. Will the facility adequately secure our equipment by deploying a sufficient number of cameras, maintaining proper security protocols, retaining historical video and log data, etc.?
• Site hazards. Is the facility located within a flood zone, susceptible to seismic activity, located within a flight path, etc.?
• Operations and engineering. Does the site use a DCIM product for monitoring/alerting, have 24/7 on-site engineering, have sufficient SLA times in place with critical equipment manufacturers, etc.?
• Logistics. How the does facility handle regular shipments to customers, unscheduled shipments, carrier pickups, etc.?

Beyond a facility’s physical specifications and design execution, there are also other factors to consider that are specific to Dropbox infrastructure design—for example, the distance of fiber optic pathways that carry traffic to and from our POPs and the proposed facility, proximity to our existing data center facilities, and staffing considerations for ongoing support. 

In the case of our external fiber (outside plant) pathways, we partner with the Dropbox network team to evaluate network providers already present in the facility (on-net) and nearby providers (near-net) the facility can offer to clients. We then solicit the vendors available to us for circuit pathways between our desired locations and review the proposed routes to identify any complications. Often times, we identify pathways which have been used for other facilities (shared fate) or areas that collapse two circuits together which could compromise facility or metro level redundancy in the event of a single fiber cut (single point of failure, or SPOF). In some cases, you may receive a route different than what you agreed to; diligence and discipline when reviewing options is mandatory here. 

Using this ranking system lets us assign a numerical value to each potential site and highlight areas of excellence or deficiencies as they relate to our requirements. This enables us to narrow our choices down to two—or sometimes, three—possible sites, which will be chosen to receive a counter proposal.

When negotiating a lease we want to show each provider that we have done our homework. Based on the commercial details provided above, here are some of things we consider:

• Rental rate. The cost of power and space, measured in $/kW/month. This rate is generally set by the facility provider, and is influenced by the provider’s priorities (business objectives, revenue targets, rate projections, market alignment, etc). Understanding these priorities can help us determine where rate reductions can happen.
• Utility rate. The cost of electricity, measured in $/kWh.
• Rental escalator. The annual percentage by which the cost of rent will increase. This figure is influenced, at least to some extent, by macroeconomic trends.
• Rent ramp. This is the rate at which our contractual power will increase over time. It should be known that there is usually quite a bit of flexibility here. Recently, we have introduced the idea of a ramp down at the ends of our lease to better align contracted dollars to our production use. 
• Power usage effectiveness (PUE). This is the cost multiplier a provider will charge for the cooling and operational costs beyond a tenant’s base electrical consumption. This figure is somewhat negotiable, but is also dependent on the overall efficiency of the facility itself in conjunction with how willing your team is to participate in efficiency best practices. A commitment from the tenant can go a long way here.
• Incentives. These are extra items that go above and beyond the data center facility itself. This might include rent abatement (first three months free), a tenant improvement allowance (the landlord offsetting some installation costs), or free office/storage space (dedicated spaces provided to your operational staff, a common need). Depending on the market trends, you may see these offered initially. 

Once we’ve sent our counter, we wait for their reply—and then decide whether or not to move ahead. If the commercial details look good, all that’s left is to sign the contractural agreement (ideally, using Dropbox Sign). Then it’s time to move in!

And that’s our process. To recap how we got here…

1. Identify what you need early.
2. Understand what’s being offered.
3. Validate the technical details.
4. Physically verify each proposal.
5. Negotiate.

From start to finish, our team has been able to leverage this process to ensure we are entering into a mutually beneficial agreement while working to minimize associated costs. Looking at the past three selection processes, we have been able to negotiate best in class rates compared to the market—in addition to promoting increased efficiency and contract flexibility.

While it’s ultimately up to you to identify what works for you and your business, this is a process that has worked for us. As daunting as the site selection process might seem, we hope this can be helpful for others—especially first-timers—attempting to navigate these waters themselves. 

~ ~ ~

If building innovative products, experiences, and infrastructure excites you, come build the future with us! Visit dropbox.com/jobs to see our open roles, and follow @LifeInsideDropbox on Instagram and Facebook to see what it's like to create a more enlightened way of working.