October 2, 2022
This post describes a two-part framework on how to run organizations that are outliers in terms of productivity and creativity. We'll focus especially on software engineering and related disciplines, as they are large industries that well-satisfy the criteria of the framework, but these ideas likely extend naturally to other fields such as aerospace, finance, pharmaceuticals, and even the broader context of academia.
Some of the methods described here will come with a significant cost, so it is worth taking a brief moment to consider why outlier productivity is important. In some industries, such as those that produce commodities, returns on additional productivity are fairly linear, whereas costs may grow superlinearly; it follows that these industries fall into a stable equilibrium of productivity. However, many modern industries are not like this. In software, because the marginal cost of production is essentially zero, a product that beats the competition can rapidly scale to millions of users, and these tendencies are compounded by modern flywheels centering data, marketplaces, and networks, producing quasi-monopolistic winner-take-all markets. Analogously, in aerospace, a more fuel-efficient jet might cost twice as much to research but yield ten times the sales; in finance, a good trading strategy can often be rapidly scaled to the vast pools of capital under management; in pharmaceuticals, the right drug that makes it through clinical trials and garners regulatory approval might generate billions in sales, whereas the wrong drug can only burn money. We will expand considerably on these ideas in future work, but for now let us proceed under the assumption that these industries, and others, can often reward relatively little marginal productivity with vast success.
This leads naturally to the question of how to generate this productivity, a topic for which considerable research effort has been applied. Certainly, people must be motivated to do their work and to do it well. For sure, smart people who know their trade will do better than those lacking. A diversity of opinion and background is probably helpful. For these, competent leadership and recruiting is essential, but these are not the main topic of this post. Put simply, people work effectively when they have everything they need to do their work, and this is a hard problem in a complicated world.
Here, we'll consider two opposite approaches for managing this complexity: specialization and the hive mind, the first which is realized quite commonly and the latter quite rarely. The main claim of this essay is that the hive mind is an underutilized tool of project management.
Specialization is a natural tool for managing complexity in an organization, for which the main idea is that not everyone needs to know everything, so the organization will construct interfaces between its employees so that each can focus on their individual roles. For example, when building a rocket, the propulsion team can be kept reasonably separate from the avionics team, so long as they agree on the interface by which the avionics throttle and gimbal the engines. Hedge funds often separate the modelers designing the trading strategy from the modelers designing the execution strategy.
Specialization has many benefits. Here are five: efficiency, scalability, interoperability, robustness, and focus.
First, often there is a truly natural division of responsibilities and a natural interface between them; in these cases, it often really is irrelevant how the other side of the interface operates so long as it does, and it would be a genuine waste of time to force these groups to deeply understand each other beyond their interface. In short, it is simply more efficient for these two groups to concentrate on their own work.
Second, specialization is highly scalable. Many projects really are too complicated for any one person to fully hold in their head. Google's monorepository, google3, has billions of lines of code. A Boeing 787 has well over two million parts. These sorts of projects are truly intractable for any individual; the only solution is for the individual to be given a bite-sized chunk, and the coordination of these bite-sized chunks is the vital role of the project manager.
Third, specialization yields interoperability of labor. By defining specific roles for employees, labor becomes more fungible, because the organization is aware of the extent of the knowledge and responsibilities of each employee. The actual width of these responsibilities might vary considerably, and the narrower they are the more fungible. Narrowing also allows the hiring of less expensive labor, which both increases the available pool of applicants and reduces costs. Nonetheless, the role of specialization is not just in creating narrow roles but also in the organizational management of responsibilities and accumulated knowledge.
From this interoperability comes organizational robustness. Without specialization, if a key employee is hit by a bus, there may be literally nobody in the world who can replace them. By abstracting the operations of the organization, no employee becomes irreplaceable, ensuring stability for the rest of the organization.
A final benefit of specialization is that the focus it provides benefits both the training and the recognition of talent. By defining specific roles, an employee has more time to get extremely good at what they do. Furthermore, it provides the organization a relatively low-stakes environment to vet employees for being a good match for the organization, and if they do well, to train and promote them. In short: specialization is a good match for the modern corporate environment.
With these benefits, one might come to believe that no organization can survive without specialization. Indeed I think this is the case; effective specialization is perhaps the vital tool for managing organizations. Yet I have also come to believe that no organization can long survive with only specialization; specialization is a tool of the bureaucrat, and the unchecked bureaucrat's organization is stale and sterile; a zombie organization. For, specialization has many wearing aspects. Three are its inefficiency, its interoperability, and its focus.
First, specialization can also create great inefficiency in organizations. Most organizational interfaces are not as clean as the examples given at the start of this section and can create tremendous friction and bottlenecks between different subunits, preventing people from getting what they need to do their work, and therefore from being effective. Even if these failures are detected, their rearchitecting is a complex, expensive, and delicate operation.
Second, the same interoperability discussed earlier is also one of the key weaknesses of specialization as a method of project management: it encourages employees, both explicitly and implicitly, to be cogs. For example, Amazon has hundreds of thousands of employees staffing its warehouses and trucks, most of whom are probably quite creative and passionate. Yet because of the specialization Amazon enforces, it doesn't particularly need to demand creativity or passion from these employees. Instead, it just demands labor. While this provides many advantages to the company -- productivity and ease of hiring, to name two -- the costs are real: high turnover, poor PR, let alone the happiness of its hundreds of thousands of hardworking employees. In other domains, the costs are far higher still. In creative roles, organizations depend on the imagination of their people and the emergent strengths of their subunits. Making creative employees cogs ensures they will put in exactly enough effort to avoid being fired and will equally reliably deter top talent.
Finally, the focus provided by specialization also constrains creativity to the framework provided by its interfaces. When an employee is forced to operate narrowly within the interfaces provided to her, she becomes unlikely to challenge the layout and broader approach of the organization. Yet, this is where the value of talented employees is often best realized: in the translating of experiences on the ground to re-architected approaches and investment. This is the work that allows big organizations to feel small, and because specialization tends to quash it, even small organizations which overly rely on specialization feel big.
The antidote to the excesses of specialization is the close-knit team, in which everyone knows a good deal about everything else happening on the team. The hive mind is the close-knit team taken to its logical extreme. In the hive mind, multiple people act as one: they talk through all decisions together and do all work together. In doing this, the hive mind functions as a virtual employee with abilities exceeding that of any normal employee. The hive mind is usually realized pairwise, although this may simply be because the necessary effort is rarely applied to produce hive minds of more than two people.
The benefits of the hive mind are highly contextual (and certainly not universal!), but a clear example of its potential impact emerges when we consider the outsized impact of the engineer Jeff Dean at Google. In software engineering, it is generally well-understood that better engineers are vastly more valuable than regular engineers, spurring the notion of the "10x" engineer, who is twice as skilled and, due to the superlinearity of rewards in software, produces ten times the value. Jeff Dean is a rare example of the "100x" engineer, who architected much of the major systems and infrastructure that made Google the trillion-dollar behemoth it is today. (Even today, derivatives of his work, such as Apache Hadoop, power much of the scientific computing of the world.) What is less well-known is that many of Jeff Dean's accomplishments were achieved in the context of a two-person hive mind. Jeff and his key collaborator Sanjay Ghemawat partnered on virtually all aspects of their work -- whiteboarding, pair programming, and co-authoring.
In fact, Jeff and Sanjay's outsized impact is probably explained as follows. First, both are brilliant programmers who would be at least "10x" engineers on their own. Yet through their collaboration, they became a virtual entity (usually manifested externally under Jeff's name; Sanjay tended not to prefer the spotlight) with abilities surpassing their own individual abilities. Finally, they were provided the necessary autonomy to build the "right" answer without the constraints of existing frameworks, and the highly leveraged environment such that their work could immediately impact millions of users and hundreds (and then thousands, and tens of thousands, and hundreds of thousands) of programmers. This is the recipe for the "100x" engineer.
I have also been fortunate to have once worked on an approximate hive mind. In my sophomore year of high school, my close friend Michael Truell and I built a programming competition called Halite which was eventually picked up by Two Sigma. Even in our first year running the competition, we attracted thousands of users and delivered an experience that I believe was superior to that of any other competition at the time. How was it that two fifteen-year-olds could build a complex system of databases, autoscaling servers, (reasonably) secure arbitrary code execution, a game engine, game visualizer, website, forums, and associated documentation? Certainly, I do attribute our success to Michael's excellence as a systems programmer. Equally important, though, was the manner in which we worked together. We sat next to each other and could get each other's feedback at a moment's notice. We both took the time to understand the full system we were building together, even if our responsibilities were often separate. We had a clear objective: to deliver the best experience we possibly could, and all of our decisions were in service of that objective (even if, as inexperienced 15-year-olds, we weren't always right.) As good friends, if we were frustrated with some problem or couldn't resolve a disagreement, we could work it out over ping-pong in the break room and then get back to work. And, we were given the autonomy we needed by Two Sigma to do the great work we wanted to do, which motivated us to work hard and to work late.
Indeed, the hive mind is not something to be taken lightly; two strangers cannot be reliably cordoned off into one in the hopes that they will achieve outlier results. The hive mind is better described as a special and intense working relationship between friends. It requires both tremendous trust as well as tremendous knowledge of the other person(s) involved.
The hive mind is at its best when a small number of highly impactful decisions need to be made which require both creativity and precision in execution, and the hive mind is given both resources and autonomy. It's at its worst when the goal is quantity over quality, when the operating frameworks come pre-formed, and when there is neither time nor space for creative and innovative work.
Finally, the hive mind, as a limiting example of a working relationship, implicitly defines a spectrum of other possible relationships, too. There are good reasons people may not want to be a full hive mind. Indeed, with Halite it was probably better that Michael and I didn't pair-program everything because we got more done; the approximate hive mind was a better fit for the project than a full hive mind would have been.
In this post, we began by considering the applications for the pursuit of outlier productivity. We then consider two approaches to collaborative work: specialization and the hive mind. Specialization benefits from its efficiency, scalability, interoperability, robustness, and focus, but this same efficiency, interoperability, and focus can lead to organizations that lack creativity and struggle to attract top talent. The hive mind, in which two people collaborate as one on their work, is a powerful and underutilized tool for highly leveraged work requiring creativity and innovation.
Given the benefits of the hive mind, a natural question is why it is so underutilized in practice. We speculate there are two key reasons for this. First, the virtues of specialization have achieved cultural dominance in society ever since Adam Smith posited it as a key contributor to growth. Consequently, society is organized around an individual-centric notion of specialization, from schooling to the selection of a particular job to the ubiquitous question "what do you do?" While there is a good chance that hive minds could also be trained systematically, the historical formalization and institutionalization of specialization assure its monopolization compared to the delicate and often esoteric operation of hive minds. Second, the kind of leveraged and creative work which best fits the hive mind has come into the mainstream only relatively recently. Indeed, historical examples of hive minds usually originate in the academic contexts where the creativity and productivity provided by the hive mind were most valued: the economists Daniel Kahneman and Amos Tversky, Albert Einstein and Michele Besso (primarily in 1905), James Watson and Franklin Crick, among others. Yet as specialized labor continues to be automated, the hive mind may come to be seen as a critical instrument in project management.
In any case, it would be a mistake to see these methods as either-or; these are tools in the toolbox. For any large organization, a mixture of specialization and hive mind-esque collaboration (as well as the broad spectrum lying between the two) is likely to be the right answer. The art of the effective manager is knowing which is appropriate where, and then defining the interfaces which are both appropriate and flexible between small hive minds.
Smith, A. (1776). An Inquiry Into the Nature and Causes of the Wealth of Nations.
Somers, J. (2018). The friendship that made google huge. The New Yorker. Retrieved September 2022, from https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge.