The article provides a general overview of the Workflow Engine architecture and explains the common principles of how the designed Workflows are executed.
Once a Workflow is designed in the Workflow Studio, it is stored in the Workflow Repository, in the Production database, and is ready for execution. The Workflow Engine is a special module of the Workspace Management application which is responsible for managing the execution of workflows and handling such tasks as starting, resuming, terminating Workflows, monitoring and persisting the Workflow instance state.
For analysis of how the Workflow instance is being executed (or has been executed), the System provides the Visual Tracking action available in the Workflow Studio. To feed this module with data, enough for understanding of how each workflow activity has been executed, which input arguments it has received, and what the Activity result is, as a value of output arguments, the Workflow Engine catches the Workflow Instance events and persists them in the Monitoring database (by default, the name of the monitoring database is "M42Monitoring"). The System keeps all Workflow Instance events in the database until the workflow instance is completed, and for some time afterward.
A workflow is a long-running process which in many cases requires human interactions, which means the time between the start and finish of the Workflow Instance could be years. Obviously, in such a timeframe the Application is restarted many times. For that reason, the Workflow Engine handles the saving of the Workflow Instance state to storage any time it changed. The System uses the Instance Store database for storing the serialized Workflow Instance (by default, the database name is "M42InstanceStore"). The instances are stored in the database only until the moment they are completed.
The Workspace Management application supports two alternative implementations of the Workflow Engine, the basic one which is based on Microsoft AppFabric, and a new one, based on using Workflow Workers.
The first version of the Workflow Engine is fully based on the Microsoft AppFabric module which out-of-the-box provides an implementation of all the basic tasks of the Workflow Engine, like hosting Workflows, monitoring, and persistence. The specifics of the AppFabric Engine is a way how the prepared Workflows are deployed and hosted. Using the "Publish" or "Publish Repository" action available either in Matrix42 Software Asset and Service Management or in the Workflow Studio, the prepared Workflows are deployed to the Application Server, to folder "/svc/WF/", and the System dynamically creates Web Services endpoints for each deployed Workflow version. In the end, each version of the Workflow represents the Workflow Service.
Workflow Workers [Technical Preview]
Due to a couple of downsides of the previous Workflow Engine implementation based on AppFabric, such as a problem with performance, using enormous system resources and a problem with the horizontal scaling of the Workflow execution, the new concept of Workflow execution called Workflow Workers has been introduced.
Workflow Workers engine is designed to fully replace AppFabric engine in upcoming releases. The Worker roll-out strategy includes the gradual replacement of AppFabric components from version to version, with always available option to fallback to AppFabric when something goes wrong.
The Workflow Worker is a Windows Service which could be installed as on Application Server as on any other computer in an Organization. The Workflow Worker Engine is based on the Message Queue Architecture, which means all the commands for doing something with Workflows, like starts a new one, or resume the Workflow, first are added to the Message Queue. Workflow Workers continuously watch the Queue and poll the messages from it one by one. Such an approach solves the following challenges:
If the current number of active Workers are not coping with the Workflow commands processing, and the Queue starts growing, just additional Worker instance needs to be added to the Cluster. Using such solutions as Kubernetes allows automatic scaling, when additional instances of Workers created or disposed automatically, depending on the workload
The Queue has high availability, which guarantees the Workflow command is store in Queue. The messages stay in Queue until they are successfully processed.
In contrast to AppFabric execution, the Workflow Worker is an external Windows process and all Business Components and other resources hosted on Application Server are not accessible on Worker. That limitation puts additional requirements to Workflow Activities running on Worker. For example, if the Activity requires executing Business Logic, it needs to be triggered over the Web Service. The standard Workflow Activities delivered with Product which considers Worker specific and could be executed on Worker, marked as Worker Compatible. If the Workflow uses only Workflow activities compatible with Worker, then this workflow can be executed on the Workflow Worker.
The System allows installing an unlimited number of Workers. In Administration, application open the management area "Services & Processes > Workflows > Workers".
- Run "Download Worker" action.
The System automatically generates a new archive which already includes the basic configuration settings like Application Server URL and name of the Service Account.
- Unpack the received archive file on the computer where you want to install the Workflow Worker.
- Run "InstallWorker.cmd".
The batch file runs the signed Powershell script. Depending on the Domain Policies, the execution of the Powershell file can be rejected due to unknown publisher. In this case, make sure the certificate used by Configure.ps1 is installed into the Trusted Publishers Certificate Store on the Local Machine.
- Proceed with the installation steps:
1) Prove the Server URL is correct, otherwise, provide the URL of the Application Server.
2) The Worker requires the valid API token to authenticate with the Server. Please provide the valid token for the user with the Administrative rights. If you do not have prepared token, it can be generated. See "Generate API Token" for more details.
3) The System checks the validity of the provided credentials and set up the connection with the Server.
If you have a connectivity problem and got a message the server cannot be contacted and the trusted SSL/TLS secure channel cannot be established, then please check that the Application Server certificate is installed on the local machine to Trusted Root Authorities.
4) Set the Service Account.
The Workflow Worker uses Integrated Security to work with the Product databases. Therefore the Service account should have the "db_owner" permissions to databases. By default, the installation suggests the Service account which is used on Application Server, but in case of need, it can be changed.
- The installation registers and starts windows service "Matrix42 Workflow Worker".
Manage Workflow Workers
Once the Workflow Worker is installed it automatically registers itself in the Application. All registered Workflow Workers, as well as their current statuses, can be found in Administration application in the management area "Services & Processes > Workflows > Workers".
Setup System Workflow Engine
The System settings define which Workflow Engine is used for the execution of a Workflow Instance. The setting can be found in Global System Setting dialog in Administration area, in the Workflows tabulator:
Use legacy Workflow Engine (AppFabric)
The same as in the previous product versions, the System keeps using the AppFabric for processing all kind of Workflow commands.
[TECHNICAL PREVIEW] Use Workflow Worker together with Legacy Workflow Engine (AppFabric)
The System uses Workflow Workers for starting and processing of all Workflows marked as “Worker Compatible” (see Set Execution Engine for Workflow ). Workflows which are either not compatible with the Worker or have already been started on legacy Workflow Engine will keep using AppFabric for execution.
If this option is not selected the Workflows, even marked as "Execute on Workflow Worker", keep using AppFabric for execution.
Use Workflow Worker
The option is disabled for the latest Product version and will be enabled when the Workflow Worker engine will be in a productive state, and most of the Workflow Activities will be compatible with the Workflow Worker
Set Execution Engine for the Workflow
The Workflow Workers is an application which runs on an independent process, either on the Application Server or on any other computer. This, compared to the classic AppFabric implementation, brings extra tasks for migration, as previously all Workflow Activities have been executed in-process of the Application and had direct access to all Business Components.
Workflow Activity Compatibility
To be able to be executed on the Workflow Worker, the Activity has to be compatible with it, which means the Activity either has all the required modules for the execution deployed on the Workflow Worker process, or the Activity is able to delegate the execution back to the Application Server via a Web Service method call. To signal the System uses the Workflow Activity property PLSLBinaryComponentClassBase.WorkerCompatible to signal that the Workflow Activity can be executed on the Workflow Worker.
In Workflow Studio, the Activities compatible with the Workflow Worker are highlighted to provide better transparency to the Author, which blocks the Workflow from being executed in the Workflow Worker.
Action "Set Execution Engine"
By default, all the Workflows present in the Workflow Repository run on the AppFabric engine. But an Administrator is able to define the execution for the specific Workflow(s). In the workflow management area, select one or several workflow definitions and start the "Set Execution Engine" action.
In case the "Workflow Worker" engine is set, the System checks the selected Workflows and proves that all the workflow activities used in Workflows are compatible with the Workflow Worker. In case at least one Activity is incompatible, the set engine operations for such workflows are rejected. In this case, the mentioned activities need to be reworked, and the compatible flag for the activity set.