The Apache Software Foundation Announces Apache® Airflow™ as a Top-Level Project

The Apache Software Foundation Announces Apache® Airflow™ as a Top-Level Project

The, Apache, Airflow, Data, From, Software, Our, Use, More, Said, Big, Community, Asf, Are, That, Foundation, Project, Workflow, Cloud, Its, All, Has

[this announcement is available online at ]Open Source Big Data workflow management system in

[this announcement is available online at ]
Open Source Big Data workflow management system in use at Adobe, Airbnb, Etsy, Google, ING,
Lyft, PayPal, Reddit, Square, Twitter, and United Airlines, among others.
Wakefield, MA —8 January 2019— The Apache Software Foundation (ASF), the all-volunteer
developers, stewards, and incubators of more than 350 Open Source projects and initiatives,
announced today Apache® Airflow™ as a Top-Level Project (TLP).
Apache Airflow is a flexible, scalable workflow automation and scheduling system for authoring
and managing Big Data processing pipelines of hundreds of petabytes. Graduation from the Apache
Incubator as a Top-Level Project signifies that the Apache Airflow community and products
have been well-governed under the ASF's meritocratic process and principles.
"Since its inception, Apache Airflow has quickly become the de-facto standard for workflow
orchestration," said Bolke de Bruin, Vice President of Apache Airflow. "Airflow has gained
adoption among developers and data scientists alike thanks to its focus on configuration-as-code.
That has gained us a community during incubation at the ASF that not only uses Apache Airflow
but also contributes back. This reflects Airflow’s ease of use, scalability, and power of
our diverse community; that it is embraced by enterprises and start-ups alike, allows us to
now graduate to a Top-Level Project."
Apache Airflow is used to easily orchestrate complex computational workflows. Through smart
scheduling, database and dependency management, error handling and logging, Airflow automates
resource management, from single servers to large-scale clusters. Written in Python, the project
is highly extensible and able to run tasks written in other languages, allowing integration
with commonly used architectures and projects such as AWS S3, Docker, Apache Hadoop HDFS,
Apache Hive, Kubernetes, MySQL, Postgres, Apache Zeppelin, and more. Airflow originated at
Airbnb in 2014 and was submitted to the Apache Incubator March 2016.
Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer,
Etsy, Google, ING, Lyft, NYC City Planning, Paypal, Polidea, Qubole, Quizlet, Reddit, Reply,
Solita, Square, Twitter, and United Airlines, among others. A list of known users can be found
"Adobe Experience Platform is built on cloud infrastructure leveraging open source technologies
such as Apache Spark, Kafka, Hadoop, Storm, and more," said Hitesh Shah, Principal Architect
of Adobe Experience Platform. "Apache Airflow is a great new addition to the ecosystem of
orchestration engines for Big Data processing pipelines. We have been leveraging Airflow for
various use cases in Adobe Experience Cloud and will soon be looking to share the results
of our experiments of running Airflow on Kubernetes."
"Our clients just love Apache Airflow. Airflow has been a part of all our Data pipelines created
in past 2 years acting as the ring-master and taming our Machine Learning and ETL Pipelines,"
said Kaxil Naik, Data Engineer at Data Reply. "It has helped us create a Single View for our
client's entire data ecosystem. Airflow's Data-aware scheduling and error-handling helped
automate entire report generation process reliably without any human-intervention. It easily
integrates with Google Cloud (and other major cloud providers) as well and allows non-technical
personnel to use it without a steep learning curve because of Airflow’s configuration-as-a-code
"With over 250 PB of data under management, PayPal relies on workflow schedulers such as Apache
Airflow to manage its data movement needs reliably," said Sid Anand, Chief Data Engineer at
PayPal. "Additionally, Airflow is used for a range of system orchestration needs across many
of our distributed systems: needs include self-healing, autoscaling, and reliable [re-]provisioning."
"Since our offering of Apache Airflow as a service in Sept 2016, a lot of big and small enterprises
have successfully shifted all of their workflow needs to Airflow," said Sumit Maheshwari,
Engineering Manager at Qubole. "At Qubole, not only are we a provider, but also a big consumer
of Airflow as well. For example, our whole Insight and Recommendations platform is built around
Airflow only, where we process billions of events every month from hundreds of enterprises
and generate insights for them on big data solutions like Apache Hadoop, Apache Spark, and
Presto. We are very impressed by the simplicity of Airflow and ease at which it can be integrated
with other solutions like clouds, monitoring systems or various data sources."
"At ING, we use Apache Airflow to orchestrate our core processes, transforming billions of
records from across the globe each day," said Rob Keevil, Data Analytics Platform Lead at
ING WB Advanced Analytics. "Its feature set, Open Source heritage and extensibility make it
well suited to coordinate the wide variety of batch processes we operate, including ETL workflows,
model training, integration scripting, data integrity testing, and alerting. We have played
an active role in Airflow development from the onset, having submitted hundreds of pull requests
to ensure that the community benefits from the Airflow improvements created at ING. We are
delighted to see Airflow graduate from the Apache Incubator, and look forward to see where
this exciting project will be taken in future!"
"We saw immediately the value of Apache Airflow as an orchestrator when we started contributing
and using it," said Jarek Potiuk, Principal Software Engineer at Polidea. "Being able to develop
and maintain the whole workflow by engineers is usually a challenge when you have a huge configuration
to maintain. Airflow allows your DevOps to have a lot of fun and still use the standard coding
tools to evolve your infrastructure. This is 'infrastructure as a code' at its best."
"Workflow orchestration is essential to the (big) data era that we live in," added de Bruin.
"The field is evolving quite fast and the new data thinking is just starting to make an impact.
Apache Airflow is a child of the data era and therefore very well positioned, and is also
young so a lot of development can still happen. Airflow can use bright minds from scientific
computing, enterprises, and start-ups to further improve it. Join the community, it is easy
to hop on!"
Availability and Oversight
Apache Airflow software is released under the Apache License v2.0 and is overseen by a self-selected
team of active contributors to the project. A Project Management Committee (PMC) guides the
Project's day-to-day operations, including community development and product releases. For
downloads, documentation, and ways to become involved with Apache Airflow, visit
About The Apache Software Foundation (ASF)
Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source
projects, including Apache HTTP Server --the world's most popular Web server software. Through
the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members
and 7,000 Committers across six continents successfully collaborate to develop freely available
enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions
are distributed under the Apache License; and the community actively participates in ASF mailing
lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings,
and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations
and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg,
Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks,
Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private
Internet Access, Red Hat, Target, Tencent, and Union Investment. For more information, visit and
© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", and "ApacheCon"
are registered trademarks or trademarks of the Apache Software Foundation in the United States
and/or other countries. All other brands and trademarks are the property of their respective
# # #
NOTE: you are receiving this message because you are subscribed to the
distribution list. To unsubscribe, send email from the recipient account to
with the word "Unsubscribe" in the subject line.

Zdroj: Apache

2019-01-08 12:00:52 352 100Facebook | Twitter | Google+ | LinkedIn
Web PHP developer Milan Jankovec

<Milan Jankovec> Full Stack Web Developer

Vytváram webové stránky a aplikácie na mieru. Čistý a validný HTML5 kód na každej platforme, moderné animované CSS3 štýly a neuveriteľné JavaScript kúzla. Bezpečný a rýchly PHP kód bez obmedzení s geniálnou architektúrou databáz.

Najčítanejšie články

What's New in MySQL 5.6 Release Candidate

The MySQL 5.6 Release Candidate is an aggregation of the previous MySQL 5.6 Development Milestone Releases (DMR) and provides a true functio... čítať viac

MySQL 5.6: Improvements in Thread Pool

MySQL Thread Pool has now been updated for the MySQL 5.6 version. Obviously, with the much higher concurrency of the MySQL Server in 5.6 its... čítať viac

DBA and Developer Guide to MySQL 5.6

MySQL is the most trusted and depended-on open source database platform in use today. As such, 9 out of the top 10 most popular and highly-t... čítať viac

MySQL Applier for Hadoop

To support the growing emphasis on real-time operations, MySQL is releasing a new MySQL Applier for Hadoop to enable the replication of even... čítať viac

MySQL Cluster 7.3 GA: Increasing Developer Flexibility and Simplicity

The MySQL team at Oracle are excited to announce the immediate availability of the MySQL Cluster 7.3 Development Milestone Release GA releas... čítať viac

MySQL Cluster Auto-Installer: Video Tutorial

Learn how easy it is to deploy a MySQL Cluster database that has been configured to best meet your applications needs within your environmen... čítať viac

MySQL Replication Utilities: Video Tutorial

Learn how to use MySQL 5.6 and the MySQL Utilities to setup, monitor and manage your MySQL replication topology. See how a single command ca... čítať viac

MySQL 5.6 Replication - Enabling the Next Generation of Web & Cloud Services

The new MySQL 5.6.5 Development Milestone Release (DMR) introduces a much anticipated feature - Global Transaction Identifiers (GTIDs) for M... čítať viac

Dnešný výber

Release Apache RocketMQ 4.1.0 (incubating)

Hi all,The Apache RocketMQ team would like to announce the release of ApacheRocketMQ 4.1.0 (incubating).Apache RocketMQ... čítať viac

Apache Jackrabbit Oak 1.7.6 released

The Apache Jackrabbit community is pleased to announce the release ofApache Jackrabbit Oak 1.7.6. The release is availab... čítať viac

Apache Knox 0.9.0 Release

The Apache Knox team is proud to announce the release of Apache Knox 0.9.0!Apache Knox is a REST API Gateway for providi... čítať viac

W3C Invites Implementations of Web Authentication: An API for accessing Public Key Credentials Level 1

The Web Authentication Working Group invites implementations of an updated Candidate Recommendation of Web Authenticatio... čítať viac

Christmas Snowfall (Joomla)

Joomla extension Christmas Snowfall (Joomla). Easy to use beautiful module to celebrate the joyful Christmas Snowfall.FEATURESOption to defi... čítať viac

Ubuntu - LXD weekly status #37

Introduction So this past week was rather intense, in a nutshell, weve: Merged LXD clustering support Split python3-lxc,... čítať viac

[Announce] Apache Kylin 2.6.0 released

The Apache Kylin team is pleased to announce the immediate availability ofthe 2.6.0 release.This is a major release afte... čítať viac

The Apache News Round-up: week ending 5 June 2015

this announcement is available online at Weve kicked the week off with the release of our first official annual repo... čítať viac