Technical Blog

1 Post tagged with the import tag

Automated Import Jobs

Posted by Tony McAffee Mar 3, 2009

Elastic Path Commerce includes out-of-the-box import tools, which can add or update products, inventories, prices, etc. Unfortunately, import job execution is manual and requires user effort every time the import is to be run. Fortunately, automating the execution process is a fairly straightforward customization.

 

For example, assume that you want to run an import job to import inventory from an ERP system every day at three in the morning. The following high level steps will be necessary to meet this requirement.

 

  1. Create an import job.
  2. Expose the import job execution facility for ease of configuration.
  3. Define and configure a quartz job for automation.
  4. Schedule the data file transfer via FTP.

 

Creating an import job

 

Before you can run an import job, either manually or automatically, you need to define the import job in Commerce Manager.You need to map the columns of the CSV data files to data fields in Elastic Path. This is documented in the Commerce Manager User Guide, which you can get from the Elastic Path documentation site.

 

Exposing the import job execution facility

 

Let's define a class called AutomatedImportServiceImpl, which will serve as our interface to the import job facility. This class will have a few attributes and a trigger method. The following snippet illustrates the heart of the customization.

 

    private ImportService importService;
    private CmUserService cmUserService;
    private String importJobName;
    private String importJobOwnerUsername;

       public void triggerImport() {
        CmUser cmUser = cmUserService.findByUserName(importJobOwnerUsername);
        final ImportJob importJob = importService.findImportJob(importJobName);
        importJob.setCmUser(cmUser);    
          importService.runImportJob(importJob);
       }

 

The importService and cmUserService attributes will be injected by Spring. The importJobName and importJobOwnerName properties can be configured to allow different import jobs. The actual code required to invoke the import job is quite minimal as can be seen in the triggerImport method. (Note that the cmUser lookup is necessary because part of the job post processing is to email a status report to the user running the job.)

 

The next step is to define the Spring service. The best place to do this is in the cmserver web project's serviceCM.xml Spring configuration file.

 

    <bean id="updateInventoryService" class="com.example.service.dataimport.impl.AutomatedImportServiceImpl">
        <property name="importService">
            <ref bean="importService" />
        </property>
        <property name="cmUserService">
            <ref bean="cmUserService" />
        </property>
        <property name="importJobName">
            <value>UpdateInventory</value>
        </property>
        <property name="importJobOwnerUsername">
            <value>admin</value>
        </property>
    </bean>

 

Additional services could be defined with different import job names to automate other import jobs.

 

Defining the quartz job

 

The quartz job configuration is straightforward. Define a quartz job to invoke the service previously created along with a trigger to activate the quartz job.

 

    <bean id="updateInventoryJob" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
        <property name="targetObject">
            <ref bean="update
InventoryService" />
        </property>
        <property name="targetMethod">
            <value>triggerImport</value>
        </property>
        <property name="concurrent">
            <value>false</value>
        </property>
    </bean>
 
          <bean id="update
InventoryTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
        <property name="jobDetail">
            <ref bean="
updateInventoryJob" />
        </property>
        <property name="cronExpression">
            <value>0 0 3 * * ?</value>
        </property>
    </bean>

 

Scheduling the data file transfer via FTP

 

When an import job is executed, the data file needs to be transfered to the assets/import directory on the server. When an import job is executed manually, the Commerce Manager takes the user specified data file and sends it to server, usually via FTP. Your automated import process needs to do this as well. One way to do this by creating a cron job on the ERP system to FTP the data file before the import job is scheduled to run.

 

Other considerations

 

Before implementing an automated import process, the following should be taken into consideration:

 

  • Automated jobs should usually be scheduled during low-activity periods to avoid affecting the performance of the cmserver web application.
  • Failure conditions are not well represented in the example. What should happen if the FTP fails?  Should the import job be deleted after processing to prevent reprocessing?  If the import fails should an email be sent to an administrator?  Beefing up AutomatedImportServiceImpl to be smarter about failure conditions would be a wise thing to do.
  • Logging is always a good thing. Think about logging start/end processing times and perhaps even number of records processed.
  • Other automation techniques can be applied. For instance, instead of quartz, a polling method could be implemented such that files are processed as soon as they are available.
0 Comments 0 References Permalink