Berkeley Xin started his work on the Spark open source project while he was a doctoral candidate at the AMPLab at the
University of California, Berkeley. He received his Ph.D. in computer science from Berkeley, where his advisors were
Michael J. Franklin and
Ion Stoica. The first research project, Shark, created a system that was able to efficiently execute SQL and advanced analytics workloads at scale. Shark won Best Demo Award at
SIGMOD 2012. Shark was one of the first open source interactive SQL on Hadoop systems, with claims that it was between 10 and 100 times faster than
Apache Hive. Shark was used by technology companies such as Yahoo, although it was replaced by a newer system called Spark SQL in 2014. The second research project, GraphX, created a graph processing system on top of Spark, a general data-parallel system. GraphX at the same challenged the notion that specialized systems are necessary for graph computation. GraphX was released as an open source project and merged into Spark in 2014, as the graph processing library on Spark.
Databricks In 2013, along with
Matei Zaharia and other key Spark contributors, Xin co-founded
Databricks, a venture-backed company based in San Francisco that offers data platform as a service, based on Spark. In 2014, Xin led a team of engineers from Databricks to compete in the Sort Benchmark and won the 2014 world record in Daytona GraySort using Spark, beating the previous record held by
Apache Hadoop by 30 times. Xin claimed that Spark was the fastest open source engine for sorting a petabyte of data. While at Databricks, he also started the DataFrames project, Project Tungsten, and Structured Streaming. DataFrames has become the foundational API while Tungsten has become the new execution engine. ==References==