We had the pleasure of backing Scott and Joe years ago at Greenplum, and heard their long term vision for turning an MPP analytic database into an open source, multi-cloud platform-as-a-service that would enable digital transformations at the world’s largest companies, and then, post an acquisition by EMC, watched them do exactly that. But it does beg an equally important follow-up question – does Astronomer have the right people in engineering and mission control to turn high potential into reality? That’s where Scott Yara and Joe Otto come in. So, ok, the answer to the “what could go right” question for Astronomer is intoxicating indeed. I think you might want to spend some money on Astronomer helping them. And your data engineer is the one keeping track of the nuclear launch codes. Yep, this is existential for your business. No longer does ETL stand for extract, transform and load – it stands for exist, thrive, and last. Put simply, the ability of a cloud scale company to generate insights from their data – to truly become “data driven” – is a direct reflection of their maturity in operationalizing the flow of their data assets. What was once possible to accomplish using ad hoc custom scripts now requires declarative data pipelines and workflows. These brave souls have an unenviable task – they must satisfy their colleagues’ voracious desire for data, by supplying all parts of the business with the exact right data in the right format at the right time, and do all this against a never ending rising tide in data volumes. With the release of Airflow 2.0 with a much better UI and a much faster and reliable scheduler, the general availability of Astronomer’s Astro cloud service and the acquisition of Datakin, the leader in real time, operational data lineage solutions, Astronomer is extremely well positioned to become the essential data processing orchestration company, providing complete data awareness and control to any organization: Airflow as the data plane, Astro as the control plane.Īnd why are we convinced this future state is so near and will be so meaningful? Look no further than the emergence of data engineers, arguably the fastest growing job title in the tech world today. There is a crying need for a solution to manage and power the flow of data within an organization. What could Astronomer become in the near future? Well, one of the benefits of the modern data stack is the ability to combine best of breed solutions into your optimal individualized data architecture, but the truth is interoperability is and will always be a difficult challenge to solve. And to do that as a managed service that is easy to use, flexible, powerful enough to handle complex DAGs, cloud-agnostic, and integrates seamlessly with the rest of your data stack, you need Astronomer. You can easily visualize your data pipelines’ dependencies, monitor the progress of each task and troubleshoot issues. To programmatically author, schedule and monitor your workflows, and to be able to do all that in a dynamic and scalable fashion, you want Airflow. Modern applications are collections of many loosely coupled services, and a workflow is the proper orderly execution of these services. Today, Astronomer helps enterprises optimally use Apache Airflow, the leading open source workflow management platform for scheduling data engineering pipelines. We invested in Astronomer for two simple reasons – what it is today, and what it has the potential to become in the not too distant future.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |