Recovery & Failover
Obsidian ensures continuity in the execution of your scheduled jobs by providing a variety of built-in recovery and failover mechanisms:
- Multiple concurrent hosts support
- Instance outage recovery
- Configurable recovery options by job
- Resubmission of abnormally terminating jobs
- Standby Nodes
Multiple Concurrent Hosts
Obsidian can be run out-of-the-box with multiple hosts via clustering. As long as at least one host is running and the jobs are not constrained to specific hosts, any server failures will not prevent jobs from being run on schedule. Any jobs running when a server fails will be recovered as defined in Instance Outage Recovery. No special configuration is required to run multiple hosts, but you do need paid licenses after the first free node. Just start up a node and it joins the available service pool.
Instance Outage Recovery
When a given server fails, any jobs that were in the midst of running cannot be completed normally. Any other running hosts will discern that the jobs have not had any activity and they will be marked as Died. This will allow other hosts to run the job for any subsequently scheduled times. And since no special configuration is required to run multiple hosts, when the issue with the server failure is resolved, simply start it up again and it joins the pool. A given instance will reuse its license as long as it has not been claimed by another host.
Configurable Job Recovery
Obsidian supports various recovery mechanisms.
Recovery modes by job type can be specified to determine behaviour when no instance is available for executing scheduled jobs. When an instance finally starts up, all missed scheduled runtimes will be evaluated against the configured recovery mode by job.
- All recovery means that all missed instances will be run. This feature logically requires the job to have access to the scheduled run time. This can be retrieved from the Context. See Implementing Jobs for details.
- None recovery means no recovery takes place. Missed jobs will be marked as such.
- Last recovery means only the last missed job will be run.
- Conflicted recovery means that a conflicted job will be run regardless of the elapsed time since it was originally scheduled once higher priority conflicts clear. See Conflicts for more details.
Resubmission of Abnormally Terminating Jobs
Obsidian allows you attempt re-execution of any job that did not complete normally for any failure reason whatsoever. You can correct any conditions that caused the failure and resubmit the job for execution. A user with admin rights can even configure which abnormal termination states allow resubmission.
Auto Retry Abnormally Terminating Jobs
Obsidian supports auto retries for abnormally terminating jobs up to a configured maximum of retries or until the job gets manually resubmitted, arrives at a normal schedule or is chained to. You may also specify the minimum interval in minutes between retries and if you wish to exponentially increase the interval between subsequent retries.
Obsidian also supports standby nodes. Standby node(s) are an available strategy to ensure ongoing load balancing and distribution even after instance failures. When an instance fails and once the license lease expires, your standby nodes will join the serving pool and share in the load of executing your jobs.
If you choose to limit your licensing to a single node, you still can achieve limited failover by having a standby node running. The standby node will not be able to successfully obtain a license and will therefore not run any jobs. If the primary node fails, the standby node can claim the license once the lease on the license has expired and start running jobs. Carfey Software encourages use of multiple licensed hosts in your production environment if at all possible as you can suffer an outage while the standby node waits for the lease to expire.