Posted On: Nov 24, 2020

Amazon Managed Workflows is a new managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows”.

Rv Airflow

Organizations want to leverage massive amounts of data for analytics and insights. To effectively use this data, customers often need to first build a workflow that defines the series of sequential tasks that prepare and process the data. Many customers use Apache Airflow to take advantage of its open source community and large library of pre-built integrations to third-party data processing tools such as Apache Spark and Hadoop. With Apache Airflow, workflows are created as Python scripts that are familiar to data engineers and data scientists. However, data engineers need to install, maintain, scale, and secure Airflow themselves, which adds cost and operational complexity.

Apache Airflow is an open-source workflow management platform.It started at Airbnb in October 2014 as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and monitor them via the built-in Airflow user in.

Airflow’s aircraft and services provide for the rapid movement of goods throughout a region, enabling widely expanded and low-cost same-day delivery. The need for rapid middle-mile logistics is growing due to e-commerce growth and increasing road congestion. Airflo is a leading fly fishing brand, providing high-quality fly fishing tackle, fly rods, fly reels, fly lines, leaders and tippets, and a range of fishing clothing and bags.

Now, data engineers and data scientists can use the same open source Airflow platform and Python language to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Managed Workflows automatically scales its workflow execution capacity to meet your needs, and is integrated with AWS security services to enable fast and secure access to data.

There is no up-front cost commitment or minimum fee for using Managed Workflows , and you pay only for what you use.

You can get started with Managed Workflows using the AWS Management Console, Command Line Interface (CLI), AWS CloudFormation or SDK. You can create an account and begin deploying Directed Accyclic Graphs (DAGs) to your Airflow environment immediately without reliance on development resources or provisioning infrastructure.

Managed Workflows is generally available today in the following regions: US East (Ohio and N. Virginia), US West (Oregon), EU (Stockholm, Ireland, and Frankfurt), and Asia Pacific (Tokyo, Singapore, and Sydney), with more regions to follow.

To learn more about Managed Workflows:

• Read Jeff Barr's blog post
• Visit the Amazon Managed Workflows product page
• Read the Amazon Managed Workflows User Guide
  

What if we say it's not like the others?

Airflow is different We're not cutting any corners. This is not yet another FFmpeg wrapper like you might have seen elsewhere. Don't get us wrong, we love FFmpeg and use many of its parts under the hood, but our custom built video processing pipeline goes way beyond wrapping FFmpeg and calling it a day. We've been working on it for years it and it lets us do things that other similar software simply can't.

It's a bold claim for sure, so here are just a few examples:

• AirPlay HEVC videos to Apple TV without transcoding
• Streaming to AirPlay 2 enabled TVs
• Adaptive audio volume, spatial headphone downmix
• Lossless audio transcoding when streaming to Apple TV (FLAC codec, requires tvOS 14)
• High quality audio transcoding when streaming to Chromecast (Opus codec)
• OCR (text recognition) for DVD/Bluray/Vobsub subtitles

...with a very particular set of skills...

Airflow is a razor sharp focused software. It supports specific set of devices and it will pull every trick in the book to get the best possible results on these devices. It may not stream video to your smart fridge, but it will gladly push your Chromecast, Apple TV and AirPlay 2 TVs to their limits.

And yes, Airflow can handle pretty much any video format and codec you throw at it.

Pixels, pixels everywhere!

Airflow can stream full 4K HDR HEVC files to Chromecast Ultra, Built-in, Apple TV 4K and AirPlay 2 enabled TVs. It will go out of its way not to touch the original video stream unless absolutely needed for compatibility reasons, ensuring best possible video quality with lowest CPU load (your computer fans will thank you). As far as we can tell, Airflow is still the only desktop software that can natively stream HEVC videos to Apple TV and AirPlay 2 TVs.

Airflow Ds

And for those pesky videos that are incompatible with your device - Airflow will handle that tranparently, with hardware accelerated transcoding if your computer supports it.

Audio pipeline that goes to eleven

Full multichannel support including DD+ passthrough with Dolby Atmos? Of course.

Airflow Systems

Advanced adaptive volume booster + limiter for late night watching when you don't want to disturb your neighbours with loud scenes but still want to hear the dialogue clearly? Check.

Spatial headphone downmix for surround sound videos? Also check.

Detailed A/V sync adjustment where you can compensate for the delay of individual devices like bluetooth headphones? Airflow has it.

And subtitle support to match it

Www.airflowresearch.com

For both embedded and external subtitles. It's a bit of a secret that pretty much every other streaming software needs to extract embedded subtitle tracks before playing the video. That involves reading the entire file upfront! Crazy, right? Airflow needs no such crude tricks. Embedded or external, for our playback pipeline it's all the same. All widely used subtitle formats are supported, now including vobsub. Integrated opensubtitles.org search is a cherry on top.

...with real time text recognition

Some subtitles (DVD, Vobsub, Bluray) are stored as pictures. This means that the only way to render them when streaming is to burn them in the video. That's inconvenient to say the least. It massively increases CPU load (think fan noise and heat) and it's completely infeasible to do for 4K videos.

Enter our new realtime subtitle text recognition (OCR). During playback Airflow will transparently extract the text from picture subtitles and render it on target device just like it would with regular text subtitles.

But wait, there's more!

The 'small' things, like the scrubbing preview, beautiful polished user interface, multiple playlists support, meticulous last position tracking, or the integrated Speed Test for Chromecast, which is invaluable when dealing with network connection issues. The list goes on.

Airflow Operators

Did we mention the remote control companion app for Android and iPhone? No? Well, it's pretty cool. It lets you control all Airflow features from the comfort of your couch. And it's completely free!