Your System is not a Sports Team

It’s the responsibility of an engineering team to do what’s right for the company, not to advocate for the system they own. Engineering teams need to be oriented around a mission not a system to avoid narrow-minded decision-making.

Do you have a team at your company called the Kafka Team, or the HBase Team, or the Docker Team? Hmm, you may have screwed up. Don’t worry, there’s time to fix this.

Years ago at Dropbox we started the Magic Pocket team to design and build the block storage system of the same name. It was a silly internal codename that somehow ended up sticking. The Magic Pocket team had a lot of successes but as soon as the system was stable and launched in production I had the team go through all the effort of rebranding to the Storage Infrastructure Team. This was a huge pain and prompted a lot of eye-rolling from folks who saw this as a meaningless management gesture. At least in this one occasion there was some method to this madness.

The method here is that this is the team of block storage domain experts at the company. If the system ever ceases to meet our needs and we need to pursue an alternative, this is the team that needs to advocate for that strategy. Magic Pocket is not a sport team and and our people not a bunch of zealous fans. They’re highly skilled engineers with a responsibility to design, build, and operate the best storage system in the world. If we can’t trust that team to drive this strategy who can we trust?

Orienting a team around a mission and not a specific system is critical to ensure that their priorities are aligned with what’s best for the company.

The Storage Infrastructure team mission was brought into sharp focus when our lead engineer proactively killed a cold-storage project that he was in charge of. The system was “his baby,” but he felt like it turned out being excessively complex and not in the long-term interest of the company. People who make these kinds of decisions are the ones you want to celebrate.

As a fortunate twist he came up with a far simpler and better design a few days later. We end up launching it ahead of schedule. It’s nice when things work out like that.

It’s natural to be biased towards a system you’ve spent countless hours building or operating. We’ve all seen examples of “system bias” gone wrong. This can sometimes be painfully obvious: the team spending six months to improve performance by 10% when it was completely fine to begin with; the team trying to desperately force their tooling on clients who are better off without it; the team riding their outdated system to the grave like the captain going down with the Titanic.

System bias can also show up in far more insidious ways. If a team views their responsibility as owning and advocating for a system then they’ll find creative ways to fill up sprint plans with work on that system, whether or not that actually matters. The same problem happens within a team when an engineer is a DRI (Directly Responsible Individual) for a specific system or sub-component. You’ll start seeing features get developed that are good in theory, but which we’d be fine without.

There’s always something to work on on a given system, you can always polish that gemstone a little more, but that doesn’t mean it’s worthwhile doing so. Many engineers have a tendency to focus inwardly on improving their pet system, unless encouraged to have a broader outlook by virtue of a mission that’s aligned with company value.

One question I like to ask engineers is:

If we could be spending these resources working on  any project at the company right now, would this still be the best use of our time?

If the answer to this question is “no” then you might have scoped your mission too tightly. Either that or you’re over-staffed! (You’re probably not over-staffed.)

If system bias is a natural consequence of a sense of ownership, we’re certainly not doing the team any favors by burdening them further with a job description that codifies this bias.

Orient your team around the problem you solve, not the tools you use to do it.

Someone else can probably give you better advice on how to define a mission than me. It should be focused enough to give a sense of team identity but broad enough so as to not codify any implementation decisions. It should also be short — one or two sentences. Start by asking yourself “what problem do we solve?” That’s probably your mission.

Organizational structures can lead to excessive constraints on the mission of a team. Sometimes this is just an inevitable consequence of a large company. The more lightweight these structures are the more engineers will be empowered to focus on broad company impact in their decision-making, rather than staring at their shoes and only thinking about their own system.

For a tech lead it’s usually easier to reason about system bias within a team. If a team is split up into too many small silos the engineers within a silo will be forced to prioritize their tiny domain. Formal structures like tech leads can serve to make this even worse if the domains are particularly small. I tend to prefer establishing DRIs for sub-priorities with a team. The DRI is responsible for a sub-priority or sub-component but it’s not their identity; they’re still expected to work on other stuff on the team and to focus their energies on whatever matters most to the team as a whole.