From Spark SQL to Declarative Pipelines at Databricks

On his first day at Databricks in 2013, Michael Armbrust — employee No. 9, — began coding Spark SQL.

Twelve years later, Armbrust, now a distinguished engineer, announced at the Databricks annual Data + AI Summit in June that the company had open sourced two of its platform technologies to Apache Spark. The news demonstrates Databricks’ continued focus on building out Spark, a project that has served as the company’s playbook since its inception.

Databricks CTO Matei Zaharia created Spark in 2009 at the University of California,…