Overview

The standard Elastic Path Quartz jobs are tied to the JVM and are not distributed.  If configured on multiple servers they will be executed on multiple servers simultaneously.  The challenge is that jobs which update entities should not be executed on multiple servers simultaneously due to risks of concurrent access issues.

This can be alleviated by tying them to a specific server when deploying to production.  However, this introduces a single point of failure and is a risk to the reliability and scalability of the system.

To resolve this issue use the persisted scheduler implementation available in the Quartz framework, which uses database tables to persist the job trigger schedules and controls which servers execute which jobs.

It is rather simple to implement this.

1. Place the attached RowLockSemaphore and JobStoreTX classes into the core library of your development environment
   • These are necessary to resolve a known issue with the locking logic in Quartz 5.1.
2. Place the attached AbstractProcessorJob class into the core library of your development environment
3. Extend AbstractProcessorJob for each Quartz job to be distributed
4. Place the Quartz SQL script appropriate for your RDBMS into your development environment
   • This can be found in the Quartz distribution
5. Add a Spring module property that specifies the JDBC data source name (usually jdbc/epjndi)
6. Configure the scheduler factory as well as trigger and job beans in your quartz.xml file

Warning:  You should rename the packages in the Java files to suit your project and to avoid conflicts if these files are ever included in the EP code base.

The following is an example of a persisted scheduler factory in a quartz.xml file:

<bean id="myPersistedSchedulerFactory"  class="org.springframework.scheduling.quartz.SchedulerFactoryBean">

  <property name="applicationContextSchedulerContextKey"><value>applicationContext</value></property>

  <property name="quartzProperties">

  <props>

    <prop key="org.quartz.scheduler.instanceName">ClusteredScheduler-cmserver</prop>

    <prop key="org.quartz.scheduler.instanceId">AUTO</prop>

    <!-- ThreadPool -->

    <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>

    <prop key="org.quartz.threadPool.threadCount">10</prop>

    <prop key="org.quartz.threadPool.threadPriority">5</prop>

    <!-- Job store -->

    <prop key="org.quartz.jobStore.misfireThreshold">30000</prop>

    <prop key="org.quartz.jobStore.class">com.customer.ep.quartz.JobStoreTX</prop>

    <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate</prop>

    <prop key="org.quartz.jobStore.useProperties">true</prop>

    <prop key="org.quartz.jobStore.dataSource">myDataSourceName</prop>

    <prop key="org.quartz.jobStore.isClustered">true</prop>

    <prop key="org.quartz.jobStore.clusterCheckinInterval">20000</prop>

    <prop key="org.quartz.jobStore.selectWithLockSQL">UPDATE {0}LOCKS SET LOCK_NAME = ? WHERE LOCK_NAME = ?</prop>

    <!-- Configure Plugin -->

    <prop key="org.quartz.plugin.shutdownhook.class">org.quartz.plugins.management.ShutdownHookPlugin</prop>

    <prop key="org.quartz.plugin.shutdownhook.cleanShutdown">true</prop>

    <!--  Datasource -->

    <prop key="org.quartz.dataSource.mfldirectDS.jndiURL">${ep.quartz.datasource}</prop>

    <prop key="org.quartz.dataSource.mfldirectDS.validationQuery">select 0 from dual</prop>

  </props>

  </property>

  <property name="triggers">

  <list>

    <ref bean="customJobTrigger"/>

  </list>

  </property>

</bean>

Place this scheduler configuration in each application that will act as the container for the clustered jobs.

Warning:  Refer to the section below on multiple clustered schedulers when you need separate sets of clustered jobs.

Distributing a Job Bean

Distributed Quartz job beans are serialized and persisted in the Quartz database tables.  As a result, the Spring injection mechanism will not work for these beans.  Therefore, job beans must retrieve any beans they use from the bean factory directly.  The typical MethodInvokingJobDetailFactoryBean configuration will not work.

The attached AbstractProcessorJob Java class, which extends QuartzJobBean, provides a simple framework that allows extensions to easily get to the Elastic Path bean factory.

Extend this class to implement a custom Quartz job class and override the executeProcess method to implement the job logic.

Note:  Preferably the job logic itself should be implemented in a separate service bean and the custom job bean just calls the necessary method on that service.

The following code snippet shows an extension class that calls importJobProcessor.launchImportJob().

public class ImportProcessorJob extends AbstractProcessorJob {

    @Override

    protected void executeProcess(final ApplicationContext context) {

      try {

        ImportJobProcessor importJobProcessor;

        importJobProcessor = (ImportJobProcessor) context.getBean("importJobProcessor");

        importJobProcessor.launchImportJob();

      } catch (Exception e) {

        // Log the error and handle as appropriate.

      } finally {

        // Any appropriate finally logic.

      }

    }

}

Once the job class has been implemented configure the Quartz trigger for the job to use the new job class.

<bean id="processImportJobTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerBean">

  <property name="jobDetail">

        <ref bean="processImportJob" />

  </property>

  <property name="startDelay" value="10000" />

  <property name="repeatInterval" value="5000" />

  <property name="group" value="MflClusteredScheduler-cmserver"/>

</bean>

<bean name="processImportJob" lazy-init="default" autowire="default" dependency-check="default">

  <property name="jobClass" value="com.myproject.ep.cmserver.quartz.ImportProcessorJob" />

</bean>

This is different than using the Quartz MethodInvokingJobDetailFactoryBean used for the OOTB configurations, which allows you to specify a service class and method.  As AbstractProcessorJob extends QuartzJobBean it is already a job bean class, and since you would retrieve beans explicitly from the bean factory, the job bean can then be serialized.

Distributing OOTB Quartz Jobs

The OOTB Quartz jobs in the CM Server are, by default, not distributed.  They should be clustered in a production environment to provide reliability.

To get around this, simply use the  AbstractProcessorJob described above to implement custom extensions for each CM Server job bean and configure them appropriately.

These include the following jobs in the CM Server quartz.xml:

• topSeller
• productRecommendation
• demoProductRecommendation
• releaseShipment
• cleanupOrderLocks
• processImportJob
• importJobCleanup
• staleImportJob
• cleanupSessions

Multiple Clustered Schedulers

It may sometimes be necessary to setup different Quartz job containers for different sets of clustered jobs.  For example, the EP Connect application may contain a completely different set of jobs than the CM Server.  In these cases it is necessary to configure different persisted schedulers for the different sets.

However, there is a catch:  Quartz requires separate database tables for each persisted scheduler even if the scheduler names and triggers are different.  So, even if you specify different scheduler names and triggers, all triggers from all schedulers will be placed in the database tables and each scheduler instance will attempt to execute all of them.  For example, a job in the EP Connect scheduler whose class resides in the EP Connect application will result in a ClassNotFoundException when the CM Server scheduler is executed.

To get around this limitation you will need to create separate sets of Quartz tables in your database, one for each persisted scheduler.

Copy the Quartz SQL scripts to create separate script files for each scheduler.  Modify them to change the prefix on each table:  the default prefix is QRTZ_.  The following excerpt is from a scheduler for the CM Server application and specifies "qrtz_cm_" as the prefix:

CREATE TABLE qrtz_cm_job_details ...

A separate scheduler for the EP Connect application might then use "qrtz_connect_".

Add these scripts to your development and deployment processes.

Then, specify the table prefix in the corresponding scheduler factory configuration in quartz.xml:

<bean id="schedulerFactoryMfl"

  class="org.springframework.scheduling.quartz.SchedulerFactoryBean">

...

<prop key="org.quartz.jobStore.tablePrefix">QRTZ_CM_</prop>

...

</bean>

The prefix is case-insensitive.

Operations Implications

Changes to the quartz schedules require the scheduler tables to be cleared.  Quartz may not always update the trigger schedules from the XML configuration when the scheduler starts up.

This typically impacts new deployments.

Include the attached reset SQL script to your operations manual and add steps to deployment manuals to execute it.

Note:  Be sure to modify the script to include the prefix if you have specified one. Also, it may be advisable to create separate scripts for each persisted scheduler when configuring multiple clustered schedulers.