On Messing Up Your Remote Team—And Then Doing It Right

How Honeycomb built a world-class remote team

Charity Majors is the CTO and founder of Honeycomb, a platform offering full stack observability for event driven debugging. Below is her story how Honeycomb built a fully distributed team of over 30—and how they got it wrong on their first try.

I knew from the very start that I wanted Honeycomb to be geographically distributed. Long before we had any thoughts of hiring anyone, I knew it was the right thing to do and the direction I wanted to take.

I also knew you have to bake remote-awareness into your culture early—it is very hard to tack it on to an existing culture later in a company’s development.

But things got complicated. Because there was also the the inconvenient reality that we (the early team of four) deeply wanted to sit in a room together every day. We liked seeing each other’s dumb faces every day, we liked eating lunch together. We liked having a place to go to every day apart from family and personal life. Having separate physical space helps delineate work from personal time, which was good for workaholics like us.

Most importantly: we had only the barest fuzzy idea of what we were actually trying to build.

Many, many days we spent more time talking to each other about what we might build and why than we did actually building—and it was impossible to predict in advance when those conversations would arise, when we’d need to grab a whiteboard, or how long they’d be. We were all deeply in tune with each other’s offhand comments, what each other was doing—there was a high level of ambient awareness which helped us achieve terrific spontaneous creative work as a unit.

Squaring that circle has taken quite a few false starts and wrong steps.

The moral (and practical) call to distribute your team

My attachment to distributed teams has taken a while for me to unpack. I think it mostly comes down to:

  1. I am increasingly of the opinion that spreading the hiring net more widely is a moral issue. I live in San Francisco, and lots of people cannot afford to live here. Particularly workers with families, those from less-wealthy backgrounds, and anyone who didn’t move here before 2009. It feels morally wrong to limit hiring to the fortunate few who have already won the bay area housing lottery.
  2. Talent. There is so much talent outside the bay area, and so much less competition for it! So many wonderful senior contributors have had to move away to raise families, in particular.
  3. Our product is designed for communication and collaboration between teammates, so it’s a natural fit (and we believe in heavily dogfooding).

I have had plenty of experience with distributed teams in the past. I worked at Linden Lab on Second Life for five and a half years—another tool used by distributed teams to collaborate. My first ever management experience was actually building out a globally distributed team called the Linden Lab “DNOC,” or distributed NOC (Network Operations Center)—a tier-one group of what we would now call site reliability engineers. And at my last job, at Facebook, Parse had a team of engineers in London assigned to us who we collaborated with closely.

So I had plenty of experience running, hiring, building, and debugging distributed teams. Meaning this should have gone well. Right? In theory I knew what I was doing—no sweat.

...And that’s about as much thought or preparation as we put into it. We wanted it, it was the right thing to do, so it would work out. Right? We flung ourselves into the first experiment with no preparation and little more than happy confidence that our best intentions would be plenty.

The importance of planning, or: when good intentions are not enough

And then our first three remote hires did not work out, in ways that ranged from “just didn’t gel” to “actually quite catastrophic.” We had sailed into them optimistically, devoted to doing whatever it took to help them succeed, but we somehow didn’t realize what that would take, or that we might be incapable of it.

There were lots of reasons for this, but one big one overshadowed the rest: fundamentally too much of what we were doing was brand spanking new—to us as well as everyone else. We were in the middle of working out what observability even was! I would come back excited from a conversation with a prospect and tell everyone in the room about it—often times via an ad hoc white boarding session—inadvertently leaving out the remote person. What we were doing or building or thinking seemed to change by the hour or the day, leaving the remote person trailing in the dust and eternally frustrated.

We tried setting up bidirectional webcams. We tried daily syncs. We tried pairing 100% of the time, despite the serious hit to our productivity (with four people, it meant sacrificing 25% of our output). We tried flying the person down half the time. Nothing seemed to help, and all three of them left within a month or two of their hiring.

After that, we regrouped and decided to approach things with a bit more structure and caution. We resolved that we were still devoted to distributed teams, but:

  1. It would need to be with people who were already experienced remote workers.
  2. It would need to be a well-defined role.
  3. Team members would need to be within our time zone +/- one hour.

Finally, we decided we weren’t going to try again until we actually had formal management structure.

Honeycomb needed to outgrow that stage where even the founders didn’t know what we were building or how to talk about it. In retrospect, it was incredibly silly to think that remote folks would be able to follow along the daily lurchings as we tried to figure out who we even were or what we were doing. It was just me, Christine, Ben, and Toshok (and sometimes Ian)—and I had known all of them for between five and 15 years, so they had a high ability to tolerate my lurching and fumbling.

... we grew up

We were in no shape to hire real professional employees until there was significantly less uncertainty. I’m not even sure we could have made it work with local employees.

We needed to get to the stage where we could tolerate more structure and a slightly more sedate pace. Back then, we had no project management to speak of. Once a quarter or so I pulled the team together to debrief where we were going and our goals for the quarter. But we had no users, we had no management, we had nothing but a vague conviction and a general direction for our nose to be pointed in, as we built the distributed column store and query engine underlying the entire product apparatus.

Distributed workers rely on far more structure than local workers do, because they don’t have all the informal cues that let a small team of four to five close friends row in the same direction.

Employee number 10 was our next fully remote hire, and to my great relief, our distributed hires have gone well since that point. We now have six full time remote employees, including one department head and one exec-level, with one as far away as the east coast. Some of them make a point of visiting once a month, others don’t.

Some types of work are much easier to do remotely than others. For example, heads down work, like writing code after the designs have been hammered out, when you can put on your headphones and just disappear into code for a few days. For roles that are involved in planning, however, in-person visits still seem necessary to maintain the relationships and have those sorts of strategic conversations.

Distribute your team. Just do it.

I think more companies should open their doors up to remote/distributed workers. It is a HUGE advantage when hiring; and not just because of the expanded hiring pool, either. Even many people who want to go to the office every day (like me) prefer companies who invest in distributed teams, because of the cultural implications, as well as the social justice impact.

It turns out that when you make the workplace safe for remote workers, it’s actually better for everyone. Once you have decoupled “doing work” from “being in a specific place at a specific time,” everybody gets more flexibility. It’s awesome for local parents—they can take meetings from their phone or laptop wherever they happen to be. They can go home at 4 p.m. every day, or work from home for a week without it being a big deal or something where special arrangements have to be made.

I personally usually work from home in the morning, then come in around lunch time. I need to record a lot of podcasts and webinars and take phone calls, and it’s quieter and easier to do that from home. When you have invested in a distributed culture, everybody can create the kind of schedule and routine that works best for them and their team.

It also forces managers to be more outcome oriented when evaluating employees’ productivity. There’s no such thing as watching people’s butts-in-seats time. Any time someone starts complaining about a person not being around enough, or not working hard enough, I will cut that line of complaining off and redirect the question to ask if the person is getting their work done; because that’s all that matters. The few times that I myself have found myself getting cranky about a person’s hours, I cut myself off and realized that I was actually not happy with their output, and that was the better and more productive conversation to have.

To make distributed teams work, you need clear communication, clear direction, an investment in some hardware/software A/V systems, good management, and some investment in processes and structure. I think of it as forcing you to grow up a little bit faster than you otherwise would have. And for most of us, that’s a huge net benefit in and of itself.

It’s a better, fairer way to hire people, it opens up new worlds of talent to hire from, and it creates a better workplace environment for you and everyone else around you. What’s not to love? Go distributed today.