by Raymie Stata
When I started telling people about our Hadoop-as-a-Service startup, I was surprised by how many asked: Does the world need yet another Hadoop startup from a bunch of former Yahoos?
The answer, of course, is yes! Before Altiscale, there hasn’t been a way for most companies to run big-data analysis with anything approaching the scale and reliability of the big Internet companies. Altiscale’s mission is to change that. And in this mission, we’ve been joined by some ex-Googlers too, including Ricardo Jenez, our VPE, and Sequoia’s Bill Coughran, on our Board of Directors, who ran much of engineering at Google for years.
We at Altiscale share an experience few others have had. We’ve worked in environments where all the company’s data assets, plus tens-of-thousands of CPUs, were at our fingertips. You can wake up one weekend morning, curious about something that would require insane amounts of resources to explore, fire off a job, and have an interesting result before you start work on Monday. Once you’ve worked in such an environment, it’s painful to leave.
When I started looking at Hadoop usage outside Yahoo, I saw folks running 20-, 50-, sometimes 100-node clusters. What’s more, these clusters were often designated for “production,” with data scientists relegated to some neglected “research cluster” of four outdated nodes.
There is a night-and-day difference between a capacity-constrained Hadoop environment run by an overtaxed ops team busy keeping mission-critical serving workloads running, versus a practically limitless Hadoop environment, run by people who jump out of bed at 3am on Saturday to address the tiniest glitch in Hadoop, and who believe that data science is the raison d’être of the environment that they run. We sometimes call the latter Big Data Dial Tone.
Big Data infrastructure is going to transform companies and industries. But this transformation will require Big Data Dial Tone, that is, the kind of infrastructure that Facebook, Google, and Yahoo have built and made available, as a service, to their employees. When you look at it the way we do, Altiscale isn’t another Hadoop startup. To us, Hadoop is just a means to an end. Rather, Altiscale’s mission is to bring Big Data Dial Tone to the vast majority of companies who will not be able to build and run it on their own, so that they too can participate in the transformations ahead.