Saturday, February 23, 2019

Open Source Workflow Engines

Workflow Processing Engine Overview 2018: Airflow vs Azkaban vs Conductor vs Oozie vs Amazon Step Functions - Shawn's Pitstop

AWS Step Functions + Simple Workflow Service (SWF)

"When using Amazon SWF, you implement workers to perform tasks. These workers can run either on cloud infrastructure, such as Amazon Elastic Compute Cloud (Amazon EC2), or on your own premises. You can create tasks that are long-running, or that may fail, time out, or require restarts—or that may complete with varying throughput and latency. Amazon SWF stores tasks and assigns them to workers when they are ready, tracks their progress, and maintains their state, including details on their completion. To coordinate tasks, you write a program that gets the latest state of each task from Amazon SWF and uses it to initiate subsequent tasks. Amazon SWF maintains an application's execution state durably so that the application is resilient to failures in individual components. With Amazon SWF, you can implement, deploy, scale, and modify these application components independently."

               The different entities or "actors" in an Amazon SWF workflow.

"What is SWF not?

  • SWF does not execute any code.
  • SWF does not contain the logic of the workflow.
  • SWF does not allow you to draw a workflow or a state machine

How does it work?

SWF is based on polling. Your code runs on your machines on AWS or on-premises – it doesn’t matter. Your code is polling for tasks from the SWF API (where they wait in queues), receives a task, executes it, and sends the result back to the SWF API.
SWF then issues new tasks to your code, and keeps the history of the workflow (state)."

AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. Using Step Functions, you can design and run workflows that stitch together services such as AWS Lambda and Amazon ECS into feature-rich applications. Workflows are made up of a series of steps, with the output of one step acting as input into the next. Application development is simpler and more intuitive using Step Functions, because it translates your workflow into a state machine diagram that is easy to understand, easy to explain to others, and easy to change. You can monitor each step of execution as it happens, which means you can identify and fix problems quickly. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected."


              Visual Workflow.
"You should consider using AWS Step Functions for all your new applications, since it provides a more productive and agile approach to coordinating application components using visual workflows. If you require external signals to intervene in your processes, or you would like to launch child processes that return a result to a parent, then you should consider Amazon Simple Workflow Service (Amazon SWF). With Amazon SWF, instead of writing state machines in declarative JSON, you write a decider program to separate activity steps from decision steps. This provides you complete control over your orchestration logic, but increases the complexity of developing applications. You may write decider programs in the programming language of your choice, or you may use the Flow framework to use programming constructs that structure asynchronous interactions for you."

{ "Comment": "A Hello World example of the Amazon States Language using a Pass state", "StartAt": "HelloWorld", "States": { "HelloWorld": { "Type": "Pass", "Result": "Hello World!", "End": true } } }

ASL: Amazon States Language - AWS Step Functions

{ "Comment": "An example of the Amazon States Language using a choice state.", "StartAt": "FirstState", "States": { "FirstState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME", "Next": "ChoiceState" }, "ChoiceState": { "Type" : "Choice", "Choices": [ { "Variable": "$.foo", "NumericEquals": 1, "Next": "FirstMatchState" }, { "Variable": "$.foo", "NumericEquals": 2, "Next": "SecondMatchState" } ], "Default": "DefaultState" }, "FirstMatchState": { "Type" : "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:OnFirstMatch", "Next": "NextState" }, "SecondMatchState": { "Type" : "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:OnSecondMatch", "Next": "NextState" }, "DefaultState": { "Type": "Fail", "Error": "DefaultStateError", "Cause": "No Matches!" }, "NextState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME", "End": true } } }

State Types:
  • Common State Fields
  • Pass: passes input to output
  • Task: single unit of work; main building block for state machines; properties
    • "Type": "Task",
    • "Resource": "arn:aws:lambda:..."
    • "TimeoutSeconds": 300,  // optional
    • "HearthbeatSeconds": 60 // optional
    • "Next": "NextState1"
    • "Retry": [ { "ErrorEquals": ["CustomError"]. "IntervalSeconds": 1, "MaxAttempts": 5 }]
    • "Catch": [ { ... }
  • Choice: conditional branching logic ("if statement", "switch", create loops)
    • { "Choices": [ { "Variable": "$.foo", "NumericEquals": 1, "Next", "MatchOne" } ... 
    • "And": [...], "Not": [...], "Or": [...]
  • Wait: delay execution
    • "Seconds": 10  ==> wait 10 seconds before next state
    • "Timestamp": "2020-01-01T00:00:00Z" // wait until specified time
    • "SecondsPath": "$.waitseconds" // use input variable
    • "TimestampPath": "$.expirydate" // use input var
  • Succeed: end state
    • "SuccessState": { Type": "Succeed" }
  • Fail: end state
    • "FailState": { "Type": "Fail", "Cause": "Data error", "Error": "inventory low" }
  • Parallel: parallel execution
    • "StateP1": { "Type": "Parallel", "Next": "FinalState",
      "Branches" : [ { "StartAt": "Wait 20s", " States: { "Wait 20s" { ... } } }, 
Input and Output Processing for State/Task; using "$" with field names;
using "path" to filter portion of input for processing and output; example

{ "numbers": [2, 3], "sum": 5 }

{ "Type": "Task", "Resource": "arn:aws:lambda:us-east1:123:function:Add", "Next", "NextStep",
"InputPath": "$.numbers", "ResultPath": "$.sum", "OutputPath": "$.sum" }

State Machine Data - AWS Step Functions

"You can now use a local version of AWS Step Functions to develop and test your workflows."

course: AWS Developer: Lambda Deep Dive @ Pluralsight
Using Step Functions to Control Flow

Azure Logic Apps + Microsoft Flow

workflow cloud platforms

Logic App Service | Microsoft Azure

What are Microsoft Flow, Logic Apps, Functions, and WebJobs? - Azure | Microsoft Docs

"Microsoft Flow and Logic Apps are both designer-first integration services that can create workflows. Both services integrate with various SaaS and enterprise applications.
Microsoft Flow is built on top of Logic Apps. They share the same workflow designer and the same connectors."

"Functions and Logic Apps are Azure services that enable serverless workloads. Azure Functions is a serverless compute service, whereas Azure Logic Apps provides serverless workflows. Both can create complex orchestrations. An orchestration is a collection of functions or steps, called actions in Logic Apps, that are executed to accomplish a complex task. For example, to process a batch of orders, you might execute many instances of a function in parallel, wait for all instances to finish, and then execute a function that computes a result on the aggregate."
Azure Logic Apps Documentation - Tutorials, API Reference | Microsoft Docs

"Azure Logic Apps is a cloud service that helps you automate and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations. Logic Apps simplifies how you design and build scalable solutions for app integration, data integration, system integration, enterprise application integration (EAI), and business-to-business (B2B) communication, whether in the cloud, on premises, or both.

To build enterprise integration solutions with Azure Logic Apps, you can choose from a growing gallery with 200+ connectors, which include services such as Azure Service Bus, Functions, and Storage; SQL, Office 365, Dynamics, Salesforce, BizTalk, SAP, Oracle DB, file shares, and more. Connectors provide triggers, actions, or both for creating logic apps that securely access and process data in real time."
Logic Apps Designer - example logic app

"Every logic app workflow starts with a trigger, which fires when a specific event happens, or when new available data meets specific criteria. Many triggers include basic scheduling capabilities so that you can specify how regularly your workloads run. For more custom scheduling scenarios, start your workflows with the Schedule trigger. Learn more about how to build schedule-based workflows.

Each time that the trigger fires, the Logic Apps engine creates a logic app instance that runs the actions in the workflow. These actions can also include data conversions and flow controls, such as conditional statements, switch statements, loops, and branching."

Quickstart - Create and automate your first workflow - Azure Logic Apps | Microsoft Docs

Select trigger: "RSS - When a feed item is published"

Set up trigger with RSS feed, frequency, and interval

Select this action: "Office 365 Outlook - Send an email"

Add contents for email body

Enterprise integration with Azure Logic Apps | Microsoft Docs

Microsoft/vscode-azurelogicapps: Azure Logic Apps extension for VS Code @ GitHub

for low code / no code apps:

Automate processes + tasks | Microsoft Flow

"Turn repetitive tasks into multistep workflows. For example, with a few clicks capture tweets and add them as leads in Dynamics 365, subscribers in Mailchimp, and more...
Start with a template..."
A multistep workflow that adds a record to Dynamics365 and MailChimp when a user with more than 100 followers posts a tweet.

"Microsoft Azure LogicApps Components:
  • Workflow – is used to define the different steps of your business process, using a graphical user interface.
  • Triggers - initiate a workflow, and create a new instance of the workflow. A trigger can be an arrival of a file on FTP site or receiving an email. Types of triggers:
    • Poll triggers: These triggers poll your service at a specified frequency to check for new data. When new data is available, a new instance of your logicapp runs with the data as input.
    • Push triggers: (Webhooks) These triggers listen for data on an endpoint, or for an event to happen, then triggers a new instance of your logicapp.
    • Recurrence trigger: This trigger instantiates an instance of your logicapp on a prescribed schedule.
    • Event-based triggers: i.e. message from service bus queue
  • Actions - An action represents a step in the workflow.
    It can invoke an operation on your API.
  • Connectors - are used to connect to different data sources and services.
    Azure LogicApps provide an in-built set of managed connectors that cover different areas like social media, file FTP and many more.
    • Enterprise Integration Pack (EIP) Connectors - provide functionality like BizTalk.
      These connectors are used for complex enterprise integration scenarios such as XML messaging and Validation. It supports exchange messages through industry-standard protocols, including AS2, X12, and EDIFACT. It requires an "Integration Accounts" to store different artifacts, like schemas, partners, certificates, maps and agreements

Online training classes:

Azure Logic Apps: Getting Started | Pluralsight

Azure Logic Apps: Getting Started : Building Blocks

  • "Connectors" are pre-built wrappers around systems (web service APIs); there are about 200 built-in connects, and you can build your own (that is just an API)
  • "Triggers": to initiate a Logic App; could be "reactive" and "proactive" (see above list); could leverage "trigger templates" and "pre-built templates"; Not all connectors could be triggers
  • "Actions": building blocks of Logic Apps; most of Actions are also Connectors.
    Actions is just "next step" in a logic app. 
  • "Flow Control": sequence of logic in Logic App
    • "if-then", "switch", "for-each", "do-until", "scope"

Microsoft Azure Developer: Creating Enterprise Logic Apps | Pluralsight

Azure Logic Apps: Fundamentals | Pluralsight

Data operation samples - Azure Logic Apps | Microsoft Docs

"actions": {
   "Initialize_variable_-_integer_array": {
      "type": "InitializeVariable",
      "inputs": {
         "variables": [ 
               "name": "myIntegerArray",
               "type": "Array",
               "value": [
      "runAfter": {}
   "Select": {
      "type": "Select",
      "inputs": {
         "from": "@variables('myIntegerArray')",
         "select": {
            "Product_ID": "@item()"
      "runAfter": {
         "Initialize_variable_-_integer_array": [

Schedule and run automated tasks and workflows with Azure Logic Apps | Microsoft Docs
"triggers": {
   "Recurrence": {
      "type": "Recurrence",
      "recurrence": {
         "frequency": "Week",
         "interval": 1,
         "schedule": {
            "hours": [
            "minutes": [
            "weekDays": [
         "startTime": "2017-09-07T14:00:00",
         "timeZone": "Pacific Standard Time"
azure-quickstart-templates/101-function-app-create-dynamic at master · Azure/azure-quickstart-templates @ GitHub