Workflow orchestration
tldr: A fancy cron runner with advanced workflow management, status monitoring, alerting, etc…
Cyber security use cases:
- Monitoring tasks
- Scheduled offensive/defensive scan triggers
- Daily Enumeration workflows
- data pipelines
- and more…
Prefect
Prefect offers modern orchestration solutions to empower data teams to build resilient workflows that scale with versatile resource needs.
Apache Airflow
Apache Airflow® is a platform created by the community to programmatically author, schedule and monitor workflows.
Difference between CI-CD runners and Workflow orchestrators
Difference between CI-CD runners like Github actions and Workflow orchestrators like Airflow and Perfect?
As to the differences, GitHub actions is generally a “on commit, do something”, like build an image, run tests, linting, build documentation, etc. It’s a tool generally to automate actions on your repository.
Airflow and Prefect are orchestration systems. They’re not tied to a repo and can do a wider variety of things. For example, every day at 4am run a daily report for the prior business day. Every weekend retain our ML models by spinning up some Amazon ECS tasks. Every half an hour hit an external API and digest it’s data into our data lake.
[…] GitHub actions… Can be shoehorned to do orchestration in a limited fashion, but it’s not quite the intent.
Prefect and airflow are, however, explicitly designed to do it. For Prefect, that means I dictate how things are run (containers, dask workers, other), I can specific what to happen on failure (retry, pagerduty alert, slack warnings, etc), and many other things that in general make my life easier bridging the proof of concept to fully productionised system easier.
Ref: https://www.reddit.com/r/mlops/comments/13ulfux/airflow_vs_github_actions_workflows/