Learning or Earning

Google has a legendary and fearsome proprietary software stack, designed for massive scale. Distributed serving, distributed load balancing, distributed processing, distributed locking, distributed file system, distributed structured storage. Many of the principles are public by now— GFS, Bigtable, Map Reduce, Chubby are some examples—but the Google implementations are most certainly not in the open. Plus there are large horizontal pieces of Google technology which aren't even well-known publicly and about which I can't even write.

It's all enormously clever stuff. Seriously.

And when I joined Google this clever stuff was an impressive competitive advantage. It allowed Google to operate at an unprecedented scale, with huge reliability as well as at lower unit operating cost than competitors. And it still runs like this, and they're still way out front with this technology as far as I can tell.

Over time, though, such an advantage naturally erodes in relative terms. The open source stack grows ever thicker and by now includes pieces of technology like Cassandra, ZooKeeper, HDFS and Pig (and indeed the Hadoop project in general). The principles of huge-scale computing on commodity hardware are being better understood, and the open stack becomes ever-more viable for real-world work.

As the gap between commodity and proprietary narrows, the downsides of a homegrown stack become increasingly palpable. It takes longer to migrate acquired companies to your platform. There's no liquid talent market into which to tap when hiring. Maintaining a custom toolchain becomes burdensome. You risk making your engineers feel like outsiders in the broader tech community—ironically despite the hyper-advanced technology with which they work. Your existing employees may even resist or resent developing skills which aren't marketable elsewhere.

This last point is key. For a long time my facile philosophy on my accidental career was "be in a job where you're either earning big or learning big". Lately, though, I've begun to think a little deeper about the trade-off. It's important not just that one is learning but also what one is learning. On the technical side the argument is clear, I hope (see above). It's recently occurred to me that there's an argument directly parallel from an organizational perspective.

That's for next time.

Here's a polaroid from my commute to Twitter yesterday: