What I Learned from Being Distributed at Honeycomb

Improving distributed work is an ongoing, intentional part of our company culture.

On being distributed optimized

Danyel Fisher is the Principal Design Researcher of Honeycomb, a platform offering full-stack observability for event-driven debugging. Below is his story about lessons from his remote work at Honeycomb.

Recently, Charity put together a fascinating blog post on being distributed at Honeycomb. In it, she lays out the challenges — and some of the opportunities — of leading a distributed company.

I’m in a great position to reflect on this. When I joined Honeycomb, the company was still forming its plan for being distributed. My role was going to be to figure out how to advance Honeycomb’s data analytics to the next level.

Joining Honeycomb as a distributed member was risky — but that was the least of my concerns. I was jumping from working for a stable position in a large corporation to an early-stage startup. In exchange for that risk, I was going to get to work on tremendously interesting technology, and to support some of the most interesting users, I’d ever encountered.

It was risky for Honeycomb, too: they (justifiably!) asked whether a researcher with largely academic experience could ship product to users. But what I had to offer was also unique: background in understanding how data analysts approach complex problems, and a human-centered approach to building sophisticated visualizations that would support those problems. We believed that my background could help Honeycomb differentiate itself from our competitors.

Compared to those bigger leaps, remote work was almost an afterthought in our conversations.

That said, I was under no illusions about remote work. It’s not trivial, and is dependent on a lot of different forces. To learn more about what could make it work, I turned to lessons from large distributed corporations, and from the academic literature.

Corporate history and distributed work

I had come from Microsoft, a company with an intensive in-person meeting culture. Almost all buildings had fully-booked conference rooms, all the time — meetings simply needed to be face to face, in a conference room. I would regularly cross campus to collaborate with product teams; I’d flown to Asia and Europe to kick off projects; colleagues from remote offices would frequently come to Redmond.

In contrast, colleagues at Boeing talked about having much more flexible collaborations, with frequent networked and remote meetings. I was incredulous when I heard about a meeting where everyone was on the same floor of the same building — but didn't see a need to find a room. They were happy to meet online.

The difference might be history. Microsoft had purchased small tech startups, and brought them to Redmond, reinforcing the centralization. In contrast, Boeing had bought up other aerospace companies from around the country — companies like Rockwell and McDonnell Douglas, in southern California and Missouri — which couldn’t simply be picked up and moved. Hardware has a cost, and that had driven them to figure out how to work remotely.

This reassured me that it was possible to build a remote-friendly culture.

What others say

Researchers in fields like Computer Supported Collaborative Work (CSCW) have studied the challenges of remote collaboration. Judy and Gary Olson’s “Distance Matters” (2000) talks about the social and cultural challenges that come from remote collaboration. I keep coming back to the image of a team working through a meeting — on one side, Americans are chatting amiably on a mid-morning call; on the other, a Dutch group has stayed into the evening, and is eager to finish up and go celebrate a national holiday.

In 2014, the same authors wrote “How to Make Distance Work Work.” They note that remote workers can often be “blind and invisible,” unable to get local cues from colleagues on the other side of the network. They suggest that organizations assign “independent work modules to locations so they don’t have to communicate much.” That’s worrisome for a person who wants to help shape product collaboratively!

Fortunately, the paper suggests tangible steps: team members should strive to build strong connections between each other; individuals should strive to be fluent in distributed technologies; managers should assemble distributed teams with an eye toward ensuring they have shared goals and a feeling of sharing a common fate.

Preparation for remote work

With this in mind, we started planning how my remote collaboration would work. Most of the time, I’d work out of a co-working space near my home, which would also give me access to touch-down space and meeting rooms for colleagues passing through town.

We also knew I needed a solid, in-person foundation with my colleagues. I didn’t have a background in SRE, and so there would inevitably be a ramp up process of learning the tools I’d be building and the customers I’d be supporting. I spent my first three weeks at Honeycomb in-person. During that time, I had coffee with every employee, trying to make sure that I’d built a solid foundation with as many as possible. I made sure I’d checked in and out code; could run the system; and could generally technically fend for myself.

I’d fly down once a month for a few days at a time. We figured that over time, those meetings would get less frequent. In fact, that became my test for successfully building my remote connections: did those meetings become “boring”? That is — could I do my job as effectively remotely, or were the in-person meetings vital? What would happen if I didn’t go?

I’m very aware that this model takes advantage of a lot of privilege: I have a home which I can leave for a few weeks with minimal stress. My door-to-door commute is just over four hours; I can arrive in San Francisco with an overnight bag, and stay in a different AirBnB every visit. This is not a great fit for everyone; I would hope this model is not a prerequisite for remote collaboration.

Mixed Results

A year and a half later — with about sixteen roughly-monthly trips — the experiment had mixed results. On the positive side, we’ve been very successful working remotely. I led the design of BubbleUp remotely, working with an engineering team in San Francisco. In-person visits can be even more effective: during one banner week, Liz visited Seattle; she and I designed the core of Honeycomb’s SLO feature during two days of whiteboarding; the next day, Charity and Pie joined us to kick off planning the Observability Maturity Model.

I can skip a visit to California, although I began to feel very disconnected from my collaborators. Frustratingly, there’s a level of side chatter that I miss out on. A few days ago, someone mentioned that they’d fixed “that problem we were talking about at lunch.” In person, there are far more times when someone randomly wanders by and says “oh, I meant to discuss this with you.” Every trip back still leads to a meaningful initiative or exciting idea getting spun up.

Supporting Remote Workers with the Right Technology

It’s clear that the choice of technology stack can make a huge difference to remote work.

It’s absolutely critical to have every meeting room prepared to be remote-friendly. It’s the difference between seamlessly joining meetings, and paying a “meeting tax” of five to ten minutes per meeting. This was underscored recently when Honeycomb moved offices. The previous one was configured with meeting rooms that were set for one-click connection to Google Meet; the new one is still coming up to speed. We’re still figuring out how to place microphones, configure calendar invites, and get reliable wifi. Until we get that configured, we’re paying that tax.

We communicate internally through Slack; most (but not all) employees have gotten into the habit of making sure conversations are in public as much as possible. There are definite tradeoffs — a public question can feel like a rebuke — but biasing conversations towards being shared helps keep ambient knowledge floating around.

We use a document sharing tool, Quip, that often feels like a less-good Google Docs. Quip has one feature, though, that makes me a fierce loyalist: it generates a “recently edited” list. Glancing through that list can give a sense of the heartbeat of the company.

The biggest technical gap is that we still haven’t decided how to simulate whiteboards remotely. I usually use real whiteboards and send photos; Liz likes to use Google’s Jamboard on her tablet.

The second biggest is that it’s a lot harder to enforce attention during a distributed meeting: it’s far too easy to check Slack notifications, dash off an email, or tune half-way out. (For me, personally, the way around is to take notes—having something else to do with my fingers forces me to pay attention keeps me on track.)

Shaping the Product for Remoteness

We might have some advantage from the fact that the product we build is designed with remote workers in mind. When the system goes down at two in the morning and a pager goes off, chances are that the Ops person on call is not in an office. Because all engineers at Honeycomb rotate through on-call shifts, they all have the equipment to be able to work from home. Some Honeycomb features — like the ability to share query links via Slack and search each others’ histories— have emerged as ways to help enable distributed collaboration.

Building a Culture of Remote-Friendliness

Honeycomb is slowly working its way toward being more friendly toward distributed workers, and one delightful part of the process has been a series of experiments in how to be better distributed.

Some of these entail using technology to its best effect. We’re growing standards around being Good Remote Participants. Mostly, for example, people do a pretty good job of keeping webcams on during chats. (A disembodied voice over a static portrait just doesn’t give the same visible cues as a phone call; relatedly, some of our colleagues are trying to shake the habit of sitting outside the camera range in meeting rooms.)

We send out shared meeting notes to an email list; it’s incredibly useful to glance through and see notes from a Sales enablement meeting, a Product planning session, and an Engineering status report.

We have a slack channel, #distributed, to discuss issues of interest to supporting distributed work.

We’re also building new rituals around distributed work. Alyson, one of our engineers, went on a long trip over the summer; working remotely, she suggested that we start a Distributed Coffee. Once a week some of us gather around webcams to chat. Some local, some remote. There’s no particular agenda; no particular theme. Sometimes we get lots of people; others, quieter. These weekly informal meetings seem to be a useful touchpoint.

(I’d encourage the reader to compare the distributed handbook from Truss, an organization that’s trying to be all-remote.)

Experiments That Didn’t Work As Well

Of course, not every experiment succeeds.

We wanted to see if we could move away from the centralized Honeycomb model. Several people have flown up to Seattle for “in-person” remote weeks, to see if we could move away from Honeycomb HQ. The most effective of those trips start with a specific agenda and goal; merely being in each others’ presence wasn’t enough.

We had two “distributed weeks;” during them, employees were asked to work remotely all week. This turned out to be helpful in reminding people what it’s like to attend meetings remotely — but it didn’t teach us much about how to better manage mixed in-person/remote meetings. Unfortunately, it didn’t really teach us how to socialize remotely: most people simply postponed social interaction, and several people reported they were lonely during the week.

We’ve tried having everyone in an in-person meeting wearing a headset. While it vastly improves remote sound quality, local participants get really annoyed by hearing their collaborators voices twice, sometimes with a substantial lag. We’ve tried to hold our weekly all-hands as a fully-distributed meeting. Local members really missed out on having the moment of being together.

Honeycomb grows up

One thing that's made a big difference has been in Honeycomb's own organic growth.

When I first joined Honeycomb, the only remote engineer explained his work to me simply: “Every few weeks, I drive into San Francisco. I spend a day, and pick up my next assignment. Then I drive back to Tahoe.”

Since then, our Tahoe team has grown to three people. We’ve brought on Liz, who works from all around the world (but is based in New York); additional team members are now spread across the Pacific Northwest. Amongst other things, this has helped distribute the labor of reminding locals to use microphones to wait for remote participation, and to adjust cameras, and to check each other’s calendars for working hours.

Honeycomb also had a major office move. Like all office moves, it was a mess. There were three weeks when we didn’t have an office at all. That period was intensely disruptive for local employees, who had to scramble to find places to work, or provision home offices.

It was also educational: we all began to figure out how to work remotely — and to adapt remotely to in-person. We’re actually learning ways that remote can be better than in-person! (Most engineers join our morning standup remotely.)

The future

When I started at Honeycomb, we’d hoped that we would get to a place where there was no difference between being in Seattle or New York as it was to be in San Francisco. I think we’ve admitted defeat on that front: there is a beating heart in San Francisco.

I don’t think Honeycomb will ever be a fully distributed company. (I’m not sure that it wants to be!). I like the fact that we’re trying to grow the ways we work together from a distance. We’re continuing to level the playing field; to look for ways that our local colleagues can be as aware of their remote colleagues as they are of each other. Improving distributed work is an ongoing, intentional part of our company culture.

We’re building a culture of distance collaboration, while still knowing that being in the same place helps.