Implementing Jobs

From Obsidian Scheduler
Jump to: navigation, search

This information covers implementing jobs in Java. This includes how to write your own jobs, use parameterization and job result features, and how to set up your classpath to include your own job implementations. If you want to schedule execution of scripts, please see our Scripting Jobs topic.

We recommend you review this page fully before implementing your own jobs. Obsidian provides you features that are not available in other schedulers which greatly improve re-usability and help ensure reliable execution. Reviewing this page and considering all available features will help you make the best choices for your needs.

You can also look at examples in our convenience Built-in Jobs that have been open-sourced under the [MIT Licence] as of Obsidian 2.7.0. In the root of the installation folder, you can find the source in obsidian-builtin-job-src.jar.


Contents

SchedulableJob Interface

Note: If you need to set up a development environment to create Obsidian jobs, see the Classpath section.

Implementing jobs in Obsidian is very straightforward for most cases. At its most basic, implementing a job simply requires implementing the SchedulableJob interface which has a single method, as shown below.

public interface SchedulableJob {
      public void execute(Context context) throws Exception;
}

In your implementation, the execute() method does any work required in the job and it can throw any type of Exception, which is handled automatically by Obsidian.

If you aren't using parameterization or saving job results, that's all you need to do. It's likely you'll just be calling some existing code through your job implementation. Here's an example:

import com.carfey.ops.job.Context;
import com.carfey.ops.job.SchedulableJob;
import com.carfey.ops.job.param.Description;

@Description("This helpful description will show in the job configuration screen.")
public class MyScheduledJob implements SchedulableJob {
	public void execute(Context context) throws Exception {
		CatalogExporter exporter = new CatalogExporter ();
		exporter.export();
	}
}

All executed jobs are supplied a Context object is used to expose configuration parameters and job results.

You can also access the scheduled runtime of the job using com.carfey.jdk.lang.DateTime Context.getScheduledTime(). If you wish to convert this to another Date type, such as java.util.Date, you can use the getMillis() method which provides UTC time in milliseconds from the epoch:

Date runTime = new java.util.Date(context.getScheduledTime().getMillis());

Note: You can annotate your job with the com.carfey.ops.job.param.Description annotation to provide a helpful job description which is shown in the job configuration screen. This can be useful for indicating how a job should be configured.

Dependency Injection via Spring

Obsidian supports executing jobs wired as components via Spring. See our dedicated page on Spring Integration for full details.

Parameterization

Obsidian offers flexibility and reuse in your jobs by supplying configurable parameters for each job schedule.

If you would like to parameterize jobs, you can define parameters on the job class itself, or use custom parameters which are only set when configuring a job. Defined parameters are automatically displayed in the Jobs screen to help guide configuration, but also to provide defaults and enforce data types and required values. Custom parameters can be set for any job, and lack additional validation.

Defined parameters are specified on the job class using the @Configuration annotation.

The following example shows a job using various parameters. Int includes a required url parameter, an optional set of names for saving the results and a Boolean value to determine whether compression should be used. It shows a fairly comprehensive usage of various data types and other parameter settings.

import com.carfey.ops.job.param.Configuration;
import com.carfey.ops.job.param.Parameter;
import com.carfey.ops.job.param.Type;

@Configuration(knownParameters={
		@Parameter(name="url", required=true, type=Type.STRING),
		@Parameter(name="saveResultsParam", required=false, allowMultiple=true, type=Type.STRING),
		@Parameter(name="compressResults", required=false, defaultValue="false", type=Type.BOOLEAN)
	})
public class MyScheduledJob implements SchedulableJob {

If you are running parameterized jobs, these parameters are very easy to access. Both defined and custom parameters are accessed in the same way. Example:

public void execute(Context context) throws Exception {
	JobConfig config = context.getConfig();

	MyExistingFunction function = new MyExistingFunction();

	String url = config.getString("url");
	function.setUrl(url);

        boolean compress = config.getBoolean("compressResults"); // defaults to false
        function.setCompress(compress);
	
        String result = function.go();
        
        for (String resultsName : config.getStringList("saveResultsParam")) {
             context.saveJobResult(resultsName, result);
        }
}

Here are all the available methods on JobConfig for retrieving your named parameters.

  • java.lang.Boolean getBoolean(java.lang.String name)
  • java.util.List<java.lang.Boolean> getBooleanList(java.lang.String name)
  • java.math.BigDecimal getDecimal(java.lang.String name)
  • java.util.List<java.math.BigDecimal> getDecimalList(java.lang.String name)
  • java.lang.Integer getInt(java.lang.String name)
  • java.util.List<java.lang.Integer> getIntList(java.lang.String name)
  • java.lang.Long getLong(java.lang.String name)
  • java.util.List<java.lang.Long> getLongList(java.lang.String name)
  • java.lang.String getString(java.lang.String name)
  • java.util.List<java.lang.String> getStringList(java.lang.String name)


The following is the @Parameter source code, which helps illustrate attributes that can be configured:

public @interface Parameter {
	public String name();
	public boolean required();
	public Type type() default Type.STRING;
	public boolean allowMultiple() default false;
	public String defaultValue() default "";
}

Global Parameters

Obsidian 2.5 introduced Global Parameters. These let you configure job parameters globally, and then simply import them into jobs as needed. Global parameters help avoid repeating the same configuration steps over and over, and can even be used to hide sensitive values from users, since they have separate access control in the admin web application.

By default, if a job parameter is configured with a value that is surrounded by double curly braces (e.g. {{param}}), then it is treated as a global parameter reference. When Obsidian sees a global parameter reference in this format during job execution, it imports all configured global parameters under the name (e.g. param) in place of the reference.

Obsidian will perform automatic type conversion for all values - a global parameter's type definition doesn't have to match the type of the defined parameter that references it. Once Obsidian has resolved all global parameter values, it will validate them to ensure all defined parameter restrictions are respected. Note that Obsidian strictly enforces that a global parameter must exist when referenced.

Note that you can configure a job parameter with multiple global parameter references along with normal values, and Obsidian will combine them all into the configuration passed into your job.

The Global Parameters page explains how to configure global parameters.

Note: If you wish to change the tokens used to surround global parameters, you may override them using properties outlined in Advanced Configuration.

Ad Hoc & One-Time Run Parameters

In addition to defining parameters for at the job level, Obsidian supports accepting parameters for a specific run time (i.e. job history) through the Jobs screen, or via the REST or Embedded APIs.

These parameters are treated the same as those at the job level, and are exposed to the job in the same manner as parameters at the job level. Note that parameters must have the same data type as any already configured for the job, and must conform to restrictions defined by the @Configuration annotation if applicable.

Config Validating Job

In addition to providing simple validation mechanisms through the @Parameter annotation, Obsidian gives you a way to add custom parameter validation to a job.

The interface com.carfey.ops.job.ConfigValidatingJob extends SchedulableJob and allows you to provide additional parameter validation that goes beyond type validity and mandatory values. Below is its definition:

public interface ConfigValidatingJob extends SchedulableJob {	

	public void validateConfig(JobConfig config) throws ValidationException, ParameterException;

}

When a job implementing this interface is configured or executed, the validateConfig() method is called. All configured parameters are available in the same JobConfig object that is provided to the execute() method. You can perform any validation you require within this method. If validation fails, the job will not be created, modified or executed (depending on when validation fails), and the messages you added to the ValidationException are displayed to the user. Consider this example:

public void validateConfig(JobConfig config) throws ValidationException, ParameterException {
	List<String> hosts = config.getStringList("hosts");
	ValidationException ve = new ValidationException();
	if (hosts.size() < 2) {
		ve.add("Host syncronization job requires at least two hosts to synchronize.");
	}
	int timeout = config.getInt("timeout");
	if (timeout < 0) {
		ve.add(String.format("Timeout must be 0 indicating no timeout or greater than 0 to indicate timeout duration.  Timeout provided was %s.", timeout));
	}
	if (!ve.getMessages().isEmpty()) {
		throw ve;
	}
}

Validation on Non-Scheduler Instances

If you configure a ConfigValidatingJob on a non-scheduler web application which does not have the job classpath available, Obsidian is forced to skip calling the corresponding validation method when the job is saved, but it will still do so during execution.

Job Results

Obsidian also allows for storing information about your job execution. This information is then available in chained and resubmitted jobs. In addition, as of release 1.4, jobs can be conditionally chained based on the saved results of a completed trigger job.

Job Results can be viewed after a job completes in the Job History screen. They are also exposed in the Obsidian REST API.

Note this example that both evaluates source job information (i.e. job results saved by the job that chained to this one) and saves state from its own execution which could be used by a subsequently chained job:

public void execute(Context context) throws Exception {
	Map<String, List<Object>> sourceJobResults = context.getSourceJobResults();
	
        // Grab results from the source job that was chained to this one
        List<Object> oldResultsList = sourceJobResults.get("inputFile");
	String oldResults = (String) oldResultsList.get(0);

	... job execution ...

        // This saved value is then available to chained jobs and can be viewed in the UI
	context.saveJobResult(resultsParamName, oldResults + " Updated");

        // As of 2.2, you can save multiple results at a time as a convenience.
	context.saveMultipleJobResults("file", Arrays.asList("first", "second"));
}

The signatures for the three methods used for retrieving and storing results:

  • java.util.Map<java.lang.String,java.util.List<java.lang.Object>> getSourceJobResults()
  • void saveJobResult(java.lang.String name, java.lang.Object value)
  • void saveMultipleJobResults(java.lang.String name, Collection<?> values) (from 2.2 onward)


Annotation-Based Jobs

While Obsidian offers a simple Java interface for creating new jobs, Obsidian also provides a way to use annotations to make an arbitrary Java class executable.

com.carfey.ops.job.SchedulableJob.Schedulable is a class-level marker annotation indicating that methods are annotated for scheduled execution. Adding this annotation allows you to configure a job in the Obsidian web app or REST API despite the class not implementing SchedulableJob.

com.carfey.ops.job.SchedulableJob.ScheduledRun is a method-level annotation to indicate one or more methods to execute at runtime. It has an int executionOrder() method that defaults to 0. This value indicates the order in which to execute methods. Duplication of execution order is not permitted. Annotated methods must have no arguments and must be public.

Note: Using these annotations precludes you from storing job results or parameterizing your job.

Interruptable Jobs

As of Obsidian 1.5.1, it is possible to terminate a running job on a best effort basis.

In some exceptional cases, it may be necessary or desirable to force termination of a job. Since exposing this functionality for all jobs could result in unexpected and even dangerous results, Obsidian provides an additional job interface that is used specifically for this function.

com.carfey.ops.job.InterruptableJob extends SchedulableJob and flags a job as interruptable. Technically speaking, this means that the main job thread will be interrupted by Thread.interrupt(), when an interrupt request is received via the UI or REST API.

The InterruptableJob interface mandates implementation of a void beforeInterrupt() method. This method allows for you to perform house-cleaning before Obsidian interrupts the job thread. For example, you may have additional threads to shut down, or other resources to release. You may also want to set a flag on the job instance to indicate to the executing thread that it should shut down, rather than rely on checking Thread.isInterrupted(). You should attempt to have your beforeInterrupt() execute in a timely fashion, though it will not interrupt other job scheduling/execution functionality if it takes some time.

It is possible that the job completes either successfully or with failure before the interrupt can proceed. If the interrupt proceeds, the job will be marked as Error and the interruption details will be made available for review in both the Job History and Log views.

Note: After invoking void beforeInterrupt(), Obsidian will invoke Thread.interrupt() to try to get the job to abort. Note that Thread.interrupt() does not forcibly terminate a thread in most cases, and it is up to the job itself to support aborting at an appropriate time when an interrupt is received. This tutorial explains the details of thread interrupts.

Classpath for Building and Deploying

To implement jobs in Java, you will need to reference Obsidian base classes in your Java project. In addition, to run jobs, your built code and any 3rd party libraries it requires must be included in the Obsidian servlet container (e.g. Tomcat) under /WEB-INF/lib. This could be a plain installation of Obsidian, or a web app which contains both your application and Obsidian.

If you are running a standalone web application that does not have a scheduler running, you do not have to update its classpath with your compiled jobs, unless you are running a version older than 2.6. To be able to configure jobs in your standalone web application, your Obsidian instances which run jobs will need to have been started at least once after your latest classpath changes, and after configuring classpath scanning. This is because scheduler instances store job metadata in the Obsidian database so the admin web application can properly validate jobs.

The base libraries to build Java jobs are found in the zip file you downloaded under the /standalone directory:

  • obsidian.jar

Prior to Obsidian 2.1.1, the following libraries were also included.

  • jdk.jar
  • jdk-gen.jar
  • suite.jar
  • suite-gen.jar
  • obsidian-gen.jar
  • carfey-date-1.2.jar or carfey-date-1.1.jar

These libraries should not conflict with your existing build classpath since they are internal Obsidian libraries.

To build a custom WAR, you can use the provided WAR artifacts in the Obsidian zip package you downloaded, and customize it in your desired build technology (e.g. Ant, Maven, Gradle, etc.).

Maven users: Note that we do not publish Maven artifacts for Obsidian, so you will not be able to include them by referencing a public repository.

Classpath Scanning

As of Obsidian 2.0, Obsidian supports classpath scanning to find your jobs for display in Job Administration, whether they are implementations of com.carfey.ops.job.SchedulableJob or use the com.carfey.ops.job.SchedulableJob.Schedulable and com.carfey.ops.job.SchedulableJob.ScheduledRun annotations.

You must specify one or more package prefixes using the Admin System. Select the "Job" category, "packageScannerPrefix" parameter. Specify your comma delimited list of package prefixes.

PackageScannerPrefix.PNG

If you are using Spring and wish to integrate Obsidian and Spring, you will likely not need to use this distinct classpath scanning functionality as jobs found in the Spring context will be available in the Job Administration screen.

Initializing Job Schedules

As of Obsidian 2.7.0

At times, you may wish to automate initialization of your jobs into Obsidian. Obsidian 2.7.0 introduced file-based initialization support as a supplement to our APIs. The file-based initialization uses a JSON collection of our JobCreationRequest (see sample below). You may find it helpful to review a complete sample of a single JobCreationRequest element in our REST API Documentation.

By default, Obsidian will look for a file on the classpath named /obsidianInitialization.json. You may override the classpath resource name using the system property obsidianInitClasspath. For example, you could add the java system property -DobsidianInitClasspath=/com/mycompany/obsidianInit.json. You can also use a file-based resource by using the system property obsidianInitFile. For example, you could add the java system property -DobsidianInitFile=/var/obsidian/obsidianScheduleInitialization.json.

Job initialization uses nicknames to uniquely identify jobs. If the nickname doesn't exist, it will attempt to create the initial schedule and configuration. If it does exist, it will not change its configuration and it will be noted in the logs. Any initialization error for a given nickname will also be logged. The initialization is best effort and will complete as many configurations as possible, meaning any failures will not impact other nicknamed jobs.

Note: Job schedules are only initialized on scheduler instances. This means that a standalone Obsidian web application with no scheduler running will not create job configuration based on the presence of the appropriate JSON file.

{
	"jobs" : [
		{
		  "jobClass": "com.carfey.ops.job.maint.LogCleanupJob",
		  "nickname": "Daily Debug Log Cleanup",
		  "recoveryType": "NONE",
		  "state": "ENABLED",
		  "schedule": "@daily",
		  "parameters": [
		      {
		        "name": "level",
		        "type": "STRING",
		        "value": "DEBUG"
		      },
		      {
		        "name": "maxAgeDays",
		        "type": "INTEGER",
		        "value": "7"
		      }
		  ]
		},
		{
		  "jobClass": "com.carfey.ops.job.maint.LogCleanupJob",
		  "nickname": "Weekly Info Log Cleanup",
		  "recoveryType": "NONE",
		  "state": "ENABLED",
		  "schedule": "@weekly",
		  "parameters": [
		      {
		        "name": "level",
		        "type": "STRING",
		        "value": "INFO"
		      },
		      {
		        "name": "maxAgeDays",
		        "type": "INTEGER",
		        "value": "30"
		      }
		  ]
		}
	]
}