1 2 3 ... 7 Previous Next

Technical Blog

101 Posts

Fun with Camel

Posted by Darren Pendery Dec 23, 2011

When customizing Elastic Path the most common form of Java-code level customization is integration with an external fulfillment service.  This integration can take many forms, depending on your architecture:  SOAP or Restful Web services, JMS messages, flat files, RMI calls, etc.

Such integration is always tricky, and usually painful.  It doesn’t have to be.

The Elastic Path checkout architecture has become much more modularized, making it very easy to implement a new CheckoutAction where your fulfillment integration can start, but you still have to implement the code to make the call.

This is where Apache Camel can make your life much easier, by abstracting the plumbing of such a call from your Java code.

This post will show how you can use Apache Camel and Apache CXF to call an external SOAP-based fulfillment Web service using a few simple lines of code in your CheckoutAction.

First, a couple of assumptions:

  • You have a WSDL for the Web service
  • You have created an extension core project to extend the Elastic Path Core project

Now, create a new Maven project that will generate the Web service code and expose it to your application via a JAR file.  The following pom.xml file can be used as a starting point:

 

 

WS Client Project pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>com.mycompany.fulfillment.client</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>Test extension fulfillment client</name>
<properties>
<cxf-version>2.5.0</cxf-version>
</properties>
<build>
<finalName>orderfulfillment-client</finalName>
<plugins>

<!-- to compile with 1.5 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
<!-- CXF wsdl2java generator, will plugin to the compile goal -->
<plugin>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-codegen-plugin</artifactId>
<version>${cxf-version}</version>
<executions>
<execution>
<id>generate-sources</id>
<phase>generate-sources</phase>
<configuration>
<sourceRoot>${basedir}/target/generated/src/main/java</sourceRoot>
<wsdlOptions>
<wsdlOption>
<wsdl>${basedir}/src/main/resources/wsdl/orderfulfillment.wsdl</wsdl>
</wsdlOption>
</wsdlOptions>
</configuration>
<goals>
<goal>wsdl2java</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

 

 

This pom assumes you have placed the Web service’s WSDL file in the src/main/resources/wsdl folder under your project folder.  It uses the Apache CXF plugin to generate the sources from the WSDL.

Executing “mvn install” will then create the JAR file, containing the classes and the WSDL file, and install it into your local Maven repository.

Add a dependency on this library in your core extension project.

Next, in your extension core project, add a couple other new dependencies:

 

Core Extension Dependencies
<!—Properties for the new dependencies. -->
<properties>
<cxf-version>2.5.0</cxf-version>
<camel-version>2.8.3</camel-version>
</properties>
<dependencies>

<!-- camel & cxf -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>${camel-version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
<version>${camel-version}</version>
</dependency>
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-frontend-jaxws</artifactId>
<version>${cxf-version}</version>
</dependency>
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-transports-http</artifactId>
<version>${cxf-version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-cxf</artifactId>
<version>${camel-version}</version>
</dependency>

<dependencies>

 

 

 

Now, implement a type converter class that will be used by Camel to convert the Elastic Path Order object to the object used by the Web service:

 

Type Converter

import org.apache.camel.Converter;
import org.apache.camel.Exchange;

 

@Converter
public class EpOrderToFulfillmentOrderConverter {

 

@Converter
public com.pendery.testextension.orderfulfillment.Order toEpOrder(final Order epOrder, final Exchange exchange) {

 

com.fulfillmentcompany.fulfillmentservice.orderfulfillment.Order fulfillmentOrder = new com.fulfillmentcompany.fulfillmentservice.orderfulfillment.Order();

 

// Copy the EP Order data into the fulfillmentOrder here.

 

return fulfillmentOrder;

}

 

}

 

 

 

Then add a file called TypeConverter (with no extension) to the src\main\resources\META-INF\services\org\apache\camel in your core extension project.  It should contain a single line, the fully qualified class name of your converter.  For example:

 

 

TypeConverter

com.mycompany.coreextension.camel.converter.EpOrderToFulfillmentOrderConverter

 

 

 

Next implement a Camel Route builder class that will route the EP Order to the Web service:

 

Route Builder

import org.apache.camel.builder.RouteBuilder;

 

public class OrderFulfillmentRouteImpl extends RouteBuilder {

 

@Override
public void configure() throws Exception {

 

final String toEndpoint = "cxf://http://externalservicehost/webservice/fulfillorder"
+ "?wsdlURL=wsdl/orderfulfillment.wsdl"
+ "&serviceName={http://orderfulfillment.fulfillmentservice.fulfillmentcompany.com/}FulfillOrderEndpointService"
+ "&portName={http://orderfulfillment.fulfillmentservice.fulfillmentcompany.com/}FulfillOrderSOAP"
+ "&serviceClass="com.fulfillmentcompany.fulfillmentservice.orderfulfillment.FulfillOrderEndpoint";

 

from ("direct:startorderfulfillment")
.convertBodyTo (com.fulfillmentcompany.fulfillmentservice.orderfulfillment.Order.class)
.to (toEndpoint);

}

 

}

 

 

This class defines a Camel endpoint called “direct:startorderfulfillment”.  It first converts the data sent to the endpoint, which should be an EP Order object, to the Order object expected by the fulfillment Web service.  Camel is smart enough to find the type converter you implemented earlier and uses that to do the conversion.  Finally it sends that converted object to the Web service.  It refers to the WSDL file from the class path, which is provided by the client project created earlier.

The Web service’s path, service name, port name and class name will be dependent on your situation.

 

Now implement your CheckoutAction to send the EP order to Camel:

 

Checkout Action

import org.apache.camel.CamelContext;

 

public class SendToFulfillmentCheckoutActionImpl implements FinalizeCheckoutAction {

 

public void execute(final FinalizeCheckoutActionContext context) throws EpSystemException {


Order epOrder = context.getOrder();

 

// Get Camel context bean.

CamelContext camel = (CamelContext) beanFactory.getBean("camel");

 

Object out = null;
OrderResponse response;

 

try {
// Send the fulfillment order to the Camel route.
response = (OrderResponse) camel.createProducerTemplate().requestBody("direct:startorderfulfillment", epOrder);
} catch (Exception e) {
throw new EpSystemException("Error sending order to fulfillment.", e);
}

 

// Check order response and any codes here.
}

}

 

 


This class sends the EP Order object to the Camel endpoint called “direct:startorderfulfillment”, which we defined in the route builder class we implemented earlier.

Finally, add the following Spring bean configuration in your core extension project’s plugin.xml file:

 

Spring Configuration (plugin.xml)

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

...

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<package>com.mycompany.coreextension.camel.route</package>
</camelContext>

...

</beans>

 

 

Make sure to add http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd to the xsi:schemaLocation attribute.

The <package> element must specify the name of the package where your Camel route builder class is implemented.  Camel will automatically pick it up and configure it.

That’s it.  Compile and run your storefront and watch the orders pile up.

The really great thing about this design is that you can implement any endpoint you wish in the Camel route builder and never need to modify your checkout service.  Camel includes components for sending to JMS queues, RMI services, flat files, etc.
In your route, you can also include any number of endpoints, whether for logging/auditing purposes, messaging multiple services, etc.

You can also configure the routes in your Spring configuration file without having to implement a custom route builder class, but that’s a personal preference.

Enjoy the Camel ride.

0 Comments Permalink

git-svn, meet git-p4

Many developers have started to use Git, a distributed VCS that offers a lot of advantages over centralized repositories when it comes to team development.  This is further fueled by the success of the integration tool between Git and Subversion, git-svn.  But did you know Git can also interact with Perforce?  While not as mature as its Subversion counterpart, git-p4 has the basic feature set to achieve the same objective: the ability to leverage Git for your daily work and only connect to Perforce when necessary.

The scenario

You're working on a project trunk for a customer, whose organization uses Perforce.   The customer has given you the proper Perforce access and expects your team to be committing regularly against it.  On the other hand, you need to maintain this project in your organization's Subversion as well.

 

So, you start off by doing an initial code drop to Perforce.  But what's the best way to bring your changes over?  The most obvious approach is doing a manual merge at regular intervals.  If done frequently and carefully, this may be the way to go.  However, besides the obvious disadvantages  (time consuming, human error), we also lose our original commit history.  What we would really like is a way to forward-port our commits from Subversion to Perforce.  Let's see how we can leverage Git to do this for us.

A thousand words...

svn-p4-git.png

There's a lot happening in this diagram, but the basic flow is easy to understand.

 

First, we set things up by cloning the project from Subversion and Perforce.  This is represented by Git branches master and master_perforce, respectively.

 

Meanwhile, the team hacks away and eventually, commit E is committed to Subversion.  The person performing the "merge" (we're not really merging here, it's a rebase as we will see later) ensures his/her repository is fully synced with Subversion (git svn rebase or git svn dcommit).

 

Next, what follows is the repeatable workflow to be done at regular intervals:

  1. Identify commit(s) to be ported to master_perforce.  In this case that would be commit E.
  2. Forward-port commit E from master to master_perforce, which then creates a new commit E' on master_perforce.
  3. Submit commit E' to the remote Perforce repository.

 

The rest of this article will now explain in detail the setup step as well as the repeatable workflow.

Cloning the project from Subversion and Perforce

This is one-time setup.  In fact, if you are already using git svn for your project, you can even skip the step on cloning the Subversion project.

 

Also note that the commands are based on Mac OS X with Bash.

Download and install the tools

  1. Download git at http://git-scm.com/Make sure to download 1.7.1 or later.
  2. Download git-p4 at https://github.com/ermshiperete/git-p4As Perforce recommends, do not use any other version other than the one provided by this link.
    • So we can invoke it from command line like git p4, we need to create a soft link.
  3. sudo ln -s ~/apps/git-p4 /usr/local/git/libexec/git-core
    
    1. git-p4 is a Python script, so download Python if necessary.
    2. Download p4 at http://www.perforce.com/product/components/perforce_commandline_client and add it to your path.
      sudo ln ~/apps/p4 /usr/bin/p4

          Configure Perforce client settings

          Create a properties file, say settings.config.

          P4PORT=<Perforce server URL>:<Perforce server port>
          P4USER=<username>
          P4PASSWD=<password>
          P4EDITOR=<path to text editor, e.g. "C:\Windows\notepad.exe" or vim>
          P4CLIENT=projectx_trunk_ws

          The last variable, P4CLIENT denotes the workspace you will be creating in the next step, so feel free to improvise.

           

          Create environment variable P4CONFIG and point to the properties file.

          export P4CONFIG=~/settings.config

          Create Perforce client workspace

          We need to create the Perforce client workspace.  Perforce requires a client workspace that defines a "client view".  This view determines the the path that the workspace is mapped to in Perforce.  For this article, we'll make the assumption that the path of the initial code drop to Perforce is //depot/projectx/trunk.  You'll also need to create a directory for this client workspace, I chose the same name but it doesn't have to be.

          mkdir projectx_trunk_ws
          cd projectx_trunk_ws
          p4 client projectx_trunk_ws

          The last line should launch the editor specified in P4EDITOR.  Look for the View: field which has the format [path in Perforce] maps to [path on your machine].  So, change //depot/... to //depot/projectx/trunk.

          View:
                  //depot/projectx/trunk/... //projectx_trunk_ws/...

          Save and exit.

          Test Perforce connectivity

          p4 info should give Perforce server connectivity as opposed to an error. So try it out now!

          Clone the Subversion project

          This should be famiar to those already using git-svn to connect to Subversion.

          mkdir projectx.git
          cd projectx.git
          git svn clone http://subversion/projectx/trunk .
          

          Add your standard Git configuration, etc...

          git config user.email "youremail@elasticpath.com"
          git config user.name "username"
          ...

          git branch should now show that we have master.  Not surprisingly, this represents the trunk in Subversion.

          rlim@project.git> git branch
          * master
          

          Clone the Perforce project

          cd projectx.git
          git p4 sync //depot/projectx/trunk@all

          @all will bring in all history. If you just want the head revision, leave that out.

           

          Unlike git svn clone, the command git p4 sync does not automatically create a development branch for us.  This is where we create perforce_master, which, not surprisingly, represents the trunk in Perforce.

          git branch master_perforce remotes/p4/trunk

          Where did remotes/p4/trunk come from?  Similar to git svn clone, the command git p4 sync creates a remote (tracking) branch named remotes/p4/trunk to track changes made in the Perforce trunk.  You don't do work on this branch, you need to check out this branch to a development (topic) branch, master_perforce.  Although not shown, git svn clone also created a remote branch behind the scenes.  The difference is it automatically created a development branch master off this remote branch for us.

          Where are we now?

          That's a whole lot of set up, but thankfully it is just one-time.  git branch should now show two branches.

           ______________________
          rlim@projectx.git> git branch
            master
          * master_perforce

          Identify the commits

          This signals the start of the repeatable workflow as described previously.  In this step we will need to identify the "candidate" commits - those that we want to port over to master_perforce.  This boils down to determining the start and end point for porting.  Any commits between these "points" will be ported over.

           

          Remember, you've done initial code to Perforce, which means at some point in Subversion history, the code was copied to Perforce.  Since we've cloned the history from Subversion, we have the commit history available in Git.  This needs to be figured out in relation to the most current commit, HEAD.  For our example, let's assume this was done at HEAD^.  Couple of ways to look at it just to drill the point home:

          • It was one revision ago (HEAD^) that we dropped the code to Perforce, or ...
          • ... since we dropped the code to Perforce, we've made one commit in Subversion - that's commit E.

          In Git, a symbolic reference is a name that points to an object.  HEAD is such a symbolic reference and the object that it points to is the most recent commit of the branch you are currently in.  Because a parent commit is always reachable from any  commit, we can refer to previous commits using the notation HEAD^ (previous commit from most recent) or HEAD~1 (previous commit again) or HEAD~2 (two commits back from most recent).  Much easier than specifying the commit object's SHA-1 hash!

           

          So, we now know when we did the initial code drop, and it also means commit after this we'll need to port over to master_perforce.  The initial code drop becomes the starting point for porting.  Let's tag this initial starting point.

          git checkout master
          git tag initial_code_drop HEAD~1
          

          What about the ending point?  Assuming we want all commits ported over, the end point is already defined for us - that's HEADPutting it together, we want all the commits between HEAD~1 (excluding) and HEAD (including).  In our simple, hypothetical scenario, that's just one commit, E

          All Your Rebase Belong to ... Git?!

          We have identified the starting and end point in which to port our commits over from master to master_perforce.  Here's the actual commands to do the actual forward-porting.  The lines are numbered for further analysis.

          1   git checkout master
          2   git branch -f start initial_code_drop
          3   git branch -f end HEAD
          4   git rebase --onto master_perforce start end
          5   git branch -f master_perforce end
          

          1 - Checkout master branch.

          2 - Rebase works off branches.  So, we must create branch start (denoting our starting point) off the tag initial_code_drop.

          3 - Same reason as previous, except for branch end (denoting our ending point) off HEAD.

          4 - This command reads, "forward port all commits onto master_perforce after start (remember, it's excluding) and ending at end".

          5 - After the previous command is executed, master_perforce has not changed but end has been rebased onto it.  We need to re-point master_perforce to end (also known as a fast-forward merge of branches master_perforce and end).  For those uninitiated to rebasing and branching with Git, search for "rebase" on the Wiki and hopefully my rambling there will shed more light on this.

           

          Line 4 is the one that does the heavy lifting.  Git takes each commit and applies it (as a patch) sequentially to master_perforce.  We also have two branches, start and end that are no longer needed.  It's safe to delete these branches (e.g. git branch -d start), but you don't really have to, since they will be used again the next time around.

          Tag new starting point for next round

          As the title states, we should tag where we last left off.

          git tag 09Dec HEAD
          

          This will become the new starting point (instead of initial_code_drop) for the next round of rebasing!  By using the current date, I know when was the last time I performed this workflow (to see the actual commit that was tagged, i.e. HEAD at that time, git show 09Dec).

           

          If you look at the 5 lines above again, it means everytime we do this, the only line that changes is line 2!  Better yet, dump these lines to a script that take a single argument - the starting point.

          Committing to Perforce

          Hopefully this should be the easiest step of the workflow.  Review your changes, as this is the point of no return!  If you want to revert before the rebase, the commands git checkout master_perforce and then git reset --hard HEAD~1 will undo commit E'.

          git checkout master_perforce
          git-p4 submit
          
          

          Each commit will bring up the editor specified in P4EDITOR.  Notice the commit message is already populated with the exact message that was committed in Subversion for commit E.

          Final thoughts

          Porting a commit that was previously ported

          If you lose track of the starting point, and accidentally port over a commit that was previously ported, don't fret!  The rebase operation works on patches and Git is smart enough to know it was already applied previously.  For this commit you will get a message like No changes – Patch already applied.  Git will happily continue along applying the other commits sequentially.

          Cherry picking a range of commits

          As of Git 1.7.2, there is another approach to port commits over instead of using rebase.  You can specify a range of commits with git cherry-pick start..end, which would essentially be a one liner operation instead of five lines above.  However, I have yet to try this and secondly, I don't believe cherry picking offers the chance to continue after a merge conflict as rebase does.

          0 Comments Permalink

          Introduction

           

          Although at times the hashCode and equals methods seem to be a “no-glory” implementation, they are extremely important in maintaining correct data manipulation. Joshua Bloch's Effective Java has an excellent overview of the concerns involved when overriding these two methods. For convenience, a copy of this chapter can be found at http://java.sun.com/developer/Books/effectivejava/Chapter3.pdf.

          Also, for a quick summary of what is involved in implementing these methods, please refer to the javadoc on these methods http://download.oracle.com/javase/6/docs/api/java/lang/Object.html#hashCode(). A good approach to coding these methods, and maintaining the general contracts, as described in the previous references, is to use the same objects to determine the hash code as the objects tested for equality in the equals method.

           

           

          Issues with Object Persistence

           

          I have encountered a few times where these methods have not been properly implemented and with testing discovered that weird things were happening, like the wrong object being deleted or updated, giving spurious results. Tracking down the cause was usually conclusive during an operation when some oblique modification occurred when it wasn't expected. In any case, an area that I've seen this occur was during database object mapping operations.

           

          In one particular case we had this code in place:

          ...

           

          public boolean equals(final Object obj) {

               if (!(obj instanceof AttributeGroupAttribute)) {

                    return false;

               }

               return this.getAttribute().getUidPk() == ((AttributeGroupAttribute) obj).getAttribute().getUidPk();

          }

           

          public int hashCode() {

               return ((Long) this.getAttribute().getUidPk()).hashCode();

          }

           

          ...

           

          There were debates on a simple check to determine equality between objects without getting wrapped up in which elements should be included in the equality equation that gave business value. Originally the uidPk, which represents the database unique identifier, for the object when persisted was being used as in the above case. But using this method broke down when we attempted to use the equality method across items in attempting to synchronize databases, as well as comparing items that were not yet persisted. To solve this we came up with using GUIDS as identifiers. This ensured that we could test equality between non-persisted objects and were not getting spurious results when synchronizing across databases. There is still debate on what we need to compare in object equivalence. Sometimes using GUIDs make sense, but then sometimes business logic comparisons makes sense as well, where GUIDs do not, as within import export of data.

           

          As another example, previously in AttributeImpl we didn't have customized hash code / equals methods in place, so in looking at the parent class AbstractEntityImpl for something reasonable there was also no customized hash code and equals methods defined. This continued along the chain of extended classes from AbstractEntityImpl. This meant the default hash code / equals method was used, which only compares object references, and was not what we needed. This caused a database deadlock when trying to view attributes on the catalog view pages.

           

          We also had issues with Customer Addresses, which used the AbstractAddressImpl implementation of hash code / equals. This caused different customer addresses to be updated than what was expected. In run time, the first customer address in the list was always the one that was updated, it wasn't able to differentiate the one that was modified on persisting.

          ...

           

          public boolean equals(final Object obj) {

                if (obj instanceof AbstractAddressImpl) {

                     AbstractAddressImpl addressEntity = (AbstractAddressImpl) obj;

                     if (checkIdentityStrings(this.lastName, addressEntity.lastName) && checkIdentityStrings(this.firstName, addressEntity.firstName)

                          && checkIdentityStrings(this.street1, addressEntity.street1) && checkIdentityStrings(this.street2, addressEntity.street2)

                          && checkIdentityStrings(this.city, addressEntity.city) && checkIdentityStrings(this.country, addressEntity.country)

                          && checkIdentityStrings(this.faxNumber, addressEntity.faxNumber)

                          && checkIdentityStrings(this.phoneNumber, addressEntity.phoneNumber)

                          && checkIdentityStrings(this.subCountry, addressEntity.subCountry)

                          && checkIdentityStrings(this.zipOrPostalCode, addressEntity.zipOrPostalCode)) {

                          return true;

                     }

                }

                return false;

          }

           

          private boolean checkIdentityStrings(final String string1, final String string2) {

               boolean identity = false;

                if (string1 == null && string2 == null) {

                     return true;

                } else if ((string1 != null && string2 != null) && string1.equals(string2)) {

                     identity = true;

                }

                return identity;

          }

           

          public int hashCode() {

                return 0;

          }

          ...

          On inspection, the equals method looks okay, but the underlying issue was that it could not determine differences in instances when the above fields in the equals method were the same. We introduced a GUID to help resolve that. The hashCode method is legal since it returns the same value for any equals comparison, but it is very inefficient since all values are placed in the same “bucket” location.

           

          With OpenJPA the hash code equals methods are equally important. When persistence operations are not working as expected, a potential culprit could be an erroneous hash code or equals method. These cases are harder to track down since we enhance our classes which add methods to the class, among other things, for persistence management. These lines do not appear when stepping through the code in debug mode. When you expect your instance to be populated for persisted properties, you would expect the classes' setXXX method would be called. But in the case of extended OpenJPA classes, it uses a combination of the getXXX method name and the enhanced classes' getPCXXX / setPCXXX methods to populate the instance variable which can be easily missed when stepping through the stack. Another issue previously experienced with OpenJPA was that when hash code and equals methods used getters to compare items in the hash code equals methods, it would cause stack overflow issues as the enhanced methods would also call the getters in figuring out the state of the object and put itself in a loop to to determine the state of the object. That is why you will see particular instances in the code where we avoid using getters over calling the instance variables directly.

           

          When approaching implementation of your own classes, Apache has a few helper utilities that have been helpful in constructing the hash code and equals contracts. The first is found within ObjectUtils, which makes things much more readable than the eclipse code generation alternative and has convenient methods to get the job done. The second is found within EqualsBuilder and HashCodeBuilder, which makes the operation even more readable and simpler than the first, since you do not need to specify a prime or result in the hash code case and the equals case uses the same equals signature in each case. Also a nicety of the second utility case is that you can chain appends on the builders. Both are used within our code base. Some examples of the two cases are as follows:

           

          In the AbstractAttributeValueImpl Class,

           

          import org.apache.commons.lang.ObjectUtils:

           

          ...

          public boolean equals(final Object obj) {

                if (this == obj) {

                     return true;

                }

           

                if (!(obj instanceof AbstractAttributeValueImpl)) {

                     return false;

                }

                AbstractAttributeValueImpl other = (AbstractAttributeValueImpl) obj;

                return (ObjectUtils.equals(this.integerValue, other.integerValue)

                     && ObjectUtils.equals(this.decimalValue, other.decimalValue)

                     && ObjectUtils.equals(this.booleanValue, other.booleanValue)

                     && ObjectUtils.equals(this.dateValue, other.dateValue)

                     && ObjectUtils.equals(this.attribute, other.attribute)

                     && StringUtils.equals(this.shortTextValue, other.shortTextValue)

                     && StringUtils.equals(this.longTextValue, other.longTextValue)

                     && StringUtils.equals(this.localizedAttributeKey, other.localizedAttributeKey)

                     && this.attributeTypeId == other.attributeTypeId);

          }

           

          public int hashCode() {

                final int prime = 31;

                int result = 1;

           

                result = prime * result + ObjectUtils.hashCode(this.integerValue);

                result = prime * result + ObjectUtils.hashCode(this.decimalValue);

                result = prime * result + ObjectUtils.hashCode(this.booleanValue);

                result = prime * result + ObjectUtils.hashCode(this.dateValue);

                result = prime * result + ObjectUtils.hashCode(this.attribute);

                result = prime * result + ObjectUtils.hashCode(this.shortTextValue);

                result = prime * result + ObjectUtils.hashCode(this.longTextValue);

                result = prime * result + ObjectUtils.hashCode(this.localizedAttributeKey);

                result = prime * result + this.attributeTypeId;

           

                return result;

          }

           

          And in the CampaignImpl Class:

           

          import org.apache.commons.lang.builder.EqualsBuilder;

          import org.apache.commons.lang.builder.HashCodeBuilder;

           

          ...

          public int hashCode() {

                return new HashCodeBuilder().append(thirdPartyId).toHashCode();

          }

           

          public boolean equals(final Object obj) {

                if (this == obj) {

                     return true;

                }

                if (!(obj instanceof CampaignImpl)) {

                     return false;

                }

                CampaignImpl other = (CampaignImpl) obj;

                return new EqualsBuilder().append(thirdPartyId, other.thirdPartyId).isEquals();

          }

           

           

          Recursive Approach in Finding Potential Missing Hash Code and Equals Methods

           

          In one project, I had the opportunity to implement change set object auditing within the data synchronization tool and was asked to reconcile the existing audit list with the one that we were initially given. I started going through each class and tried to follow each extension as well as each OpenJPA annotation to try to figure out if we indeed had covered each class in order to recreate the data that was modified. I started getting overwhelmed by the second or third class thinking this is crazy doing this by hand I won't be able to keep things straight going through this. So I created an application that would go through a number of target classes and create a list, with comments similarly to the list given for the expected auditable classes in the spring configuration. It was extremely helpful and thorough in it's output and we were able to isolate which items were missing from the list. We were able to feed back the classes we didn't want in the list to generate a new list with more exact results. Another side benefit of the application was to figure out which classes were not implementing hash code and equals methods because we were getting strange results when syncing to the remote database (adding of domain objects when they already existed and were not correctly updated). So again given a set of target classes, which was the same as the auditable list, we were able to isolate candidates that needed review from the unmanageable list of all classes to only a few. This proved quite useful and was used a few times during the project to verify these methods were being implemented.

           

          In summary of the project, it uses recursion to iterate through the target classes, checking the parent classes as well as any classes defined in any OpenJPA annotations.

          ...

          for (final Class< ? > clazz : targetClasses) {

                recordMessage("", createComment("*** Class under change set policy " + clazz.getName() + " ***"));

                recurse(clazz, null, DEFAULT_TAB_SPACE);

                recordMessage("", LINE_SEPARATOR);

          }

          ...

           

          Each class traversed is checked for a hash code equals method. Also a log of classes recursed through is kept (for the audit table).

           

          public static void recurse(final Class< ? > clazz, final Class< ? > subclass, final String space) {

                if (clazz == Object.class) {

                     return;

                }

           

                if (processedClasses.contains(clazz)) {

                     recordMessage(space, createAlreadyCoveredComment(clazz, subclass));

                     return;

                }

           

                // log against the class itself

                if (subclass != null) {

                     recordMessage(space, createComment("Parent class of " + subclass.getName()));

                }

           

                if (ignoredClasses.contains(clazz)) {

                     recordMessage(space, createComment("Ignoring class element " + createValueElement(clazz.getName())

                          + " but still traversing it's candidates"));

                } else {

                     recordMessage(space, createValueElement(clazz.getName()));

                }

           

                // track this as a processed class to trim the recursive tree

                processedClasses.add(clazz);

           

                // recurse on parent

                recurse(clazz.getSuperclass(), clazz, space + DEFAULT_TAB_SPACE);

           

                // get object classes used in annotations

                final List<Class< ? >> annotatedClasses = getAnnotatedClasses(clazz);

           

                // track object classes with no customised hashCode/equals methods

                trackNoCustomHashCodeEqualsMethods(clazz);

           

                // recurse on annotated classes

                if (annotatedClasses.isEmpty()) {

                     recordMessage(space, createComment("No annotated classes under " + clazz.getName()));

                } else {

                     recordMessage(space, createComment("Start annotated classes under " + clazz.getName()));

                     for (final Class< ? > annotatedClass : annotatedClasses) {

                          recurse(annotatedClass, null, space + DEFAULT_TAB_SPACE);

                     }

                     recordMessage(space, createComment("End annotated classes under " + clazz.getName()));

                }

          }

           

          All recursed classes are added to a processed classes list to trim the recursion path and potentially weed out any circular dependencies. A full listing of the project is included as an attachment with this article.

           

          As a post-commentary, perhaps the initial target classes could be populated from the list of persistent classes in the persistence-renamed.xml file instead of being hard-coded, but it did the trick in identifying areas of concern. We were able to modify the comments for the audit table so that it would generate a new list as we liked on demand, rather than going through things by hand again. You can try it out and tailor it as you wish to suit your own purpose. I placed it within the com.elasticpath.tools/com.elasticpath.tools.sync project under com.elasticpath.tools.sync.utils.impl for convenience. You can set it up within eclipse by right clicking on it and choosing either run or debug. Then in the debug configurations you can configure it in the arguments section to a different location or file name for the reports.

          0 Comments Permalink

          Clustering Quartz Jobs

          Posted by Darren Pendery Sep 19, 2011

          Overview

          The standard Elastic Path Quartz jobs are tied to the JVM and are not distributed.  If configured on multiple servers they will be executed on multiple servers simultaneously.  The challenge is that jobs which update entities should not be executed on multiple servers simultaneously due to risks of concurrent access issues.

          This can be alleviated by tying them to a specific server when deploying to production.  However, this introduces a single point of failure and is a risk to the reliability and scalability of the system.

          To resolve this issue use the persisted scheduler implementation available in the Quartz framework, which uses database tables to persist the job trigger schedules and controls which servers execute which jobs.

          It is rather simple to implement this.

          1. Place the attached RowLockSemaphore and JobStoreTX classes into the core library of your development environment
            • These are necessary to resolve a known issue with the locking logic in Quartz 5.1.
          2. Place the attached AbstractProcessorJob class into the core library of your development environment
          3. Extend AbstractProcessorJob for each Quartz job to be distributed
          4. Place the Quartz SQL script appropriate for your RDBMS into your development environment
            • This can be found in the Quartz distribution
          5. Add a Spring module property that specifies the JDBC data source name (usually jdbc/epjndi)
          6. Configure the scheduler factory as well as trigger and job beans in your quartz.xml file

           

           

          Warning:  You should rename the packages in the Java files to suit your project and to avoid conflicts if these files are ever included in the EP code base.

           

           

           

           

           

           

          The following is an example of a persisted scheduler factory in a quartz.xml file:

           

           

           

          <bean id="myPersistedSchedulerFactory"  class="org.springframework.scheduling.quartz.SchedulerFactoryBean">

            <property name="applicationContextSchedulerContextKey"><value>applicationContext</value></property>

            <property name="quartzProperties">

            <props>

              <prop key="org.quartz.scheduler.instanceName">ClusteredScheduler-cmserver</prop>

              <prop key="org.quartz.scheduler.instanceId">AUTO</prop>

              <!-- ThreadPool -->

              <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>

              <prop key="org.quartz.threadPool.threadCount">10</prop>

              <prop key="org.quartz.threadPool.threadPriority">5</prop>

              <!-- Job store -->

              <prop key="org.quartz.jobStore.misfireThreshold">30000</prop>

              <prop key="org.quartz.jobStore.class">com.customer.ep.quartz.JobStoreTX</prop>

              <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate</prop>

              <prop key="org.quartz.jobStore.useProperties">true</prop>

              <prop key="org.quartz.jobStore.dataSource">myDataSourceName</prop>

              <prop key="org.quartz.jobStore.isClustered">true</prop>

              <prop key="org.quartz.jobStore.clusterCheckinInterval">20000</prop>

              <prop key="org.quartz.jobStore.selectWithLockSQL">UPDATE {0}LOCKS SET LOCK_NAME = ? WHERE LOCK_NAME = ?</prop>

              <!-- Configure Plugin -->

              <prop key="org.quartz.plugin.shutdownhook.class">org.quartz.plugins.management.ShutdownHookPlugin</prop>

              <prop key="org.quartz.plugin.shutdownhook.cleanShutdown">true</prop>

              <!--  Datasource -->

              <prop key="org.quartz.dataSource.mfldirectDS.jndiURL">${ep.quartz.datasource}</prop>

              <prop key="org.quartz.dataSource.mfldirectDS.validationQuery">select 0 from dual</prop>

            </props>

            </property>

            <property name="triggers">

            <list>

              <ref bean="customJobTrigger"/>

            </list>

            </property>

          </bean>

           

           

          Place this scheduler configuration in each application that will act as the container for the clustered jobs.

           

          Warning:  Refer to the section below on multiple clustered schedulers when you need separate sets of clustered jobs.

           

           

           

          Distributing a Job Bean

          Distributed Quartz job beans are serialized and persisted in the Quartz database tables.  As a result, the Spring injection mechanism will not work for these beans.  Therefore, job beans must retrieve any beans they use from the bean factory directly.  The typical MethodInvokingJobDetailFactoryBean configuration will not work.

          The attached AbstractProcessorJob Java class, which extends QuartzJobBean, provides a simple framework that allows extensions to easily get to the Elastic Path bean factory.

          Extend this class to implement a custom Quartz job class and override the executeProcess method to implement the job logic.

           

           

          Note:  Preferably the job logic itself should be implemented in a separate service bean and the custom job bean just calls the necessary method on that service.

           

          The following code snippet shows an extension class that calls importJobProcessor.launchImportJob().

           

           

          public class ImportProcessorJob extends AbstractProcessorJob {

              @Override

              protected void executeProcess(final ApplicationContext context) {

                try {

                  ImportJobProcessor importJobProcessor;

                  importJobProcessor = (ImportJobProcessor) context.getBean("importJobProcessor");

                  importJobProcessor.launchImportJob();

                } catch (Exception e) {

                  // Log the error and handle as appropriate.

                } finally {

                  // Any appropriate finally logic.

                }

              }

          }

           

          Once the job class has been implemented configure the Quartz trigger for the job to use the new job class.

           

           

          <bean id="processImportJobTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerBean">

            <property name="jobDetail">

                  <ref bean="processImportJob" />

            </property>

            <property name="startDelay" value="10000" />

            <property name="repeatInterval" value="5000" />

            <property name="group" value="MflClusteredScheduler-cmserver"/>

          </bean>

          <bean name="processImportJob" lazy-init="default" autowire="default" dependency-check="default">

            <property name="jobClass" value="com.myproject.ep.cmserver.quartz.ImportProcessorJob" />

          </bean>

           

          This is different than using the Quartz MethodInvokingJobDetailFactoryBean used for the OOTB configurations, which allows you to specify a service class and method.  As AbstractProcessorJob extends QuartzJobBean it is already a job bean class, and since you would retrieve beans explicitly from the bean factory, the job bean can then be serialized.

          Distributing OOTB Quartz Jobs

          The OOTB Quartz jobs in the CM Server are, by default, not distributed.  They should be clustered in a production environment to provide reliability.

          To get around this, simply use the  AbstractProcessorJob described above to implement custom extensions for each CM Server job bean and configure them appropriately.

          These include the following jobs in the CM Server quartz.xml:

          • topSeller
          • productRecommendation
          • demoProductRecommendation
          • releaseShipment
          • cleanupOrderLocks
          • processImportJob
          • importJobCleanup
          • staleImportJob
          • cleanupSessions

          Multiple Clustered Schedulers

          It may sometimes be necessary to setup different Quartz job containers for different sets of clustered jobs.  For example, the EP Connect application may contain a completely different set of jobs than the CM Server.  In these cases it is necessary to configure different persisted schedulers for the different sets.

          However, there is a catch:  Quartz requires separate database tables for each persisted scheduler even if the scheduler names and triggers are different.  So, even if you specify different scheduler names and triggers, all triggers from all schedulers will be placed in the database tables and each scheduler instance will attempt to execute all of them.  For example, a job in the EP Connect scheduler whose class resides in the EP Connect application will result in a ClassNotFoundException when the CM Server scheduler is executed.

          To get around this limitation you will need to create separate sets of Quartz tables in your database, one for each persisted scheduler.

          Copy the Quartz SQL scripts to create separate script files for each scheduler.  Modify them to change the prefix on each table:  the default prefix is QRTZ_.  The following excerpt is from a scheduler for the CM Server application and specifies "qrtz_cm_" as the prefix:

           

           

          CREATE TABLE qrtz_cm_job_details ...

           

          A separate scheduler for the EP Connect application might then use "qrtz_connect_".

          Add these scripts to your development and deployment processes.

          Then, specify the table prefix in the corresponding scheduler factory configuration in quartz.xml:

          <bean id="schedulerFactoryMfl"

            class="org.springframework.scheduling.quartz.SchedulerFactoryBean">

          ...

          <prop key="org.quartz.jobStore.tablePrefix">QRTZ_CM_</prop>

          ...

          </bean>

           

           

           

          The prefix is case-insensitive.

           

           

          Operations Implications

           

          Changes to the quartz schedules require the scheduler tables to be cleared.  Quartz may not always update the trigger schedules from the XML configuration when the scheduler starts up.

           

          This typically impacts new deployments.

          Include the attached reset SQL script to your operations manual and add steps to deployment manuals to execute it.

           

          Note:  Be sure to modify the script to include the prefix if you have specified one. Also, it may be advisable to create separate scripts for each persisted scheduler when configuring multiple clustered schedulers.

           

           

           

           

          0 Comments Permalink

          Code review is a good way to catch mistakes, review designs, and help developers learn from each other.  Over-the-shoulder reviews are an easy habit to enforce for small teams, where developers are only a chair slide away.  With more projects spanning into multiple teams that has developers working with partners an ocean apart, code reviews become more of an inconvenience.  There are many code review tools out there to help with this problem.  For one of our projects, our customer introduced us to the Code Collaborator tool, which makes code reviews over the wire a lot easier.

           

          HOW IT WAS DONE BEFORE…

           

          When developers are not working in the same office, it makes sharing code a little harder.  A common practice is to commit the code and have the reviewer rely on the revision logs to find all the changes.  This can quickly become a headache if the changes span over multiple commits.  It also means that you are committing potentially buggy or smelly code to the build, which should make any good developer squeal.

           

          Another way is to send code directly through emails.  The downside, being, you will have to download each file to your machine, and manually diff each one of them to find what the changes are.  This is also a great way to fill up your inbox, with code conversations and attachments going back and forth.

           

          THE TOOL!

           

          Smart Bear's Code Collaborator is a peer review tool that automates many parts of the code review process and also cuts down some of the little annoyances.  You can easily connect it to your favourite version control system, and have it collect all the code changes you have made, without the need to commit actual code.  This can be done by either using the tool’s GUI client, or through the various plug-ins available for existing source control clients.  Perforce 4 was the VCS of our project.  We simply installed the plug-in available for P4V client, and we were able to create reviews with just a few clicks.

           

          perforce-p4v-changelistmenu.zoom50.png

           

          As a developer uploads the code to be reviewed, he can specify who should participate in the review.  Once done, he can simply create the review and continue making changes to the code and upload the changes to the same review later, or begin the code review process.

           

          CC3.png

           

          Once submitted for review, the participants are notified through email.  They can then login to the server and begin looking through the code changes.  They can also see any outstanding reviews that they need to perform.

           

          CC4.png

           

          A reviewer can comment specific lines of code, or even mark defects found.  One can even track a defect to an external issue tracker.  Conversations between the reviewer and author are also tracked per highlighted line.

           

          CC5.png

           

          Admittedly, the notification emails can easily spam up your inbox if you are reviewing for multiple people.  We found email filters essential for taming the tool.  Otherwise, all the conversations, code changes and issues found can be managed inside the tool.  Being able to review code without any commits increases collabortaion without any risk of having the CI build break.  This tool also has a search function that let’s you search for particular comments within your reviews, making it easier to back trace reasons for changes.  There are still many features that we haven’t fully explored yet.

           

          Overall, we found this tool to be very helpful for maintaining our code quality, as well as for transfering knowledge and keeping team members aware of all the project changes.  There are also many alternatives to Code Collaborator, some of which are free.  Atlassian’s Crucible and Google’s code review app both have similar functionality.  Code review tools like these are worth considering for larger teams.

          0 Comments Permalink

          Having fun with SWT

          Posted by Arman Sharif Sep 9, 2011
          Background

          Being new to the SWT I wanted to learn a little about the API by making a simple change to the Commerce Manager UI. I put this post together with the hope that it might have some useful tidbits for other SWT newbies out there.

           

          Goal

          The CM client displays a product icon in the catalog's Product Listing view. The icon may vary depending on whether the product is a bundle or has multiple SKUs, but otherwise it is the same icon for every product. To make things more interesting I decided to customise the view to generate an icon specific to each individual product. In the real world, the image location would probably come from a product attribute. To keep things simple however we will simply generate a small square image with a random background colour.

           

           

          Solution

           

          To achieve our goal we will create a new class ProductIconImageRegistry. This class will be responsible for creating SWT product image icons that can be obtained via the following get(Product) method.

           

          /**
          * Returns an image for the given product.
          */
          public Image get(final Product product) {
               final Long key = product.getUidPk();
               Image image = imageCache.get(key);
           
               if (image == null) {
                    if (isCacheFull()) {
                         freeCache();
                    }
                         
                    image = getProductIcon(product);
                    addToCache(key, image);
               }
                    
               return image;
          }
          

           

           

          Because every new SWT image instance requires an allocation of OS resources, we will cache the icons in a hash map. To keep the cache from growing indefinitely we will define an upper limit and free up some cache when it reaches the maximum size.

           

          Two important rules to bear in mind (see References) when working with SWT components are:

           

              "If you created it, you dispose it."

              "Disposing the parent disposes the children"

           

          The rules apply to a number of SWT classes including Image, Color, Font, Widget, GC, and so forth. If you invoke a constructor to instantiate one of these classes, you must free them using the dispose() method.

           

          Color color = new Color(...); // allocates platform resources
          color.dispose();
          

           

          However if you acquire an instance without calling the constructor there is no need to dispose().

           

          Color color = display.getSystemColor(SWT.COLOR_BLUE);
          

           

          In the case of the ProductIconImageRegistry class, we are creating a new icon image as follows:

           

          /**
            * Returns an icon for the given product.
            */
          private Image getProductIcon(final Product product) {
               final Display display = Display.getCurrent();
               final Color color = createRandomColor(display);
               final Image iconImage = createIconImage(display, color);
               return iconImage;
          }
           
          /**
            * Creates a random Color.
            */
          private Color createRandomColor(final Display display) {
               final int red = random.nextInt(256);
               final int green = random.nextInt(256);
               final int blue = random.nextInt(256);
               return new Color(display, red, green, blue);
          }
           
          /**
            * Creates an Image with the specified background Color.
            */
          private Image createIconImage(final Display display, final Color color) {
               final Image image = new Image(display, ICON_LENGTH, ICON_LENGTH);
               final GC gc = new GC(image);
               gc.setBackground(color);
               gc.fillRectangle(0, 0, ICON_LENGTH, ICON_LENGTH);
               
               drawIconBorder(gc, display.getSystemColor(SWT.COLOR_GRAY));
               gc.dispose();
               return image;
          }
           
          /**
            * Draws a border around the icon.
            */
          private void drawIconBorder(final GC gc, final Color borderColor) {
               final int border = ICON_LENGTH - 1;
               gc.setForeground(borderColor);
               gc.drawLine(0, 0, 0, border);
               gc.drawLine(0, 0, border, 0);
               gc.drawLine(0, border, border, border);
               gc.drawLine(border, 0, border, border);
          }
          

           

           

          Our freeCache() method will simply purge a tenth of its contents and call dispose() on every removed image.

           

          private void freeCache() {
               int removeQty = CACHE_SIZE / 10;
               for (int i = 0; i < removeQty; i++) {
                    final Long removedKey = cacheKeys.removeFirst();
                    final Image removedImage = imageCache.remove(removedKey);
                    removedImage.dispose();
               }
          }
          

           

          We will also provide a disposeAllImages() method on the ProductIconImageRegistry class to allow the client code purge everything on shutdown.

           

          public void disposeAllImages() {
               for (Long key : imageCache.keySet()) {
                    final Image image = imageCache.get(key);
                    image.dispose();
               }
               imageCache.clear();
               cacheKeys.clear();
          }
           
          

           

          Finally, we need to plug in our new class into the existing CatalogImageRegistry and CoreImageRegistry classes. This can be done by updating the getImageForProduct(Product) method:

           

          public static Image getImageForProduct(final Product product) {
               if (product instanceof ProductBundle) {
                    return getImage(PRODUCT_BUNDLE);
               }
                    
               if (product.hasMultipleSkus()) {
                    return getImage(PRODUCT_MULTI_SKU);
               }
                    
               return getImage(PRODUCT);
          }
          

           

          and replacing the last line with

           

          return productIconImageRegistry.get(product);
          

           

          We also need to hook in the disposeAllImages() method:

           

          static void disposeAllImages() {
               for (final ImageDescriptor desc : IMAGES_MAP.keySet()) {
                    final Image image = IMAGES_MAP.get(desc);
                    if (!image.isDisposed()) {
                         image.dispose();
                    }
               }
           
               productIconImageRegistry.disposeAllImages();
          }
          

           

          With the all pieces together, the final result looks as shown in the screen shot. The icons in the Product Listing view are provided by the CatalogImageRegistry class.

           

          grep-cmclient.png

           

          The CoreImageRegistry class provides icons for the Select a Product dialog, which looks as follows with the changes in place:

           

          grep-cmclient-search.png

           

           

           

          References

           

          http://www.eclipse.org/articles/Article-SWT-images/graphics-resources.html

          http://www.eclipse.org/articles/swt-design-2/swt-design-2.html

          http://eclipse.org/articles/Article-SWT-graphics/SWT_graphics.html

          0 Comments Permalink

          NOTE:  All testing done using EP 6.3.1(solr 1.4).  Any numbers regarding performance should be treated as results of beta testing.

           

          Previous to Solr 1.4 (included in EP 6.3), index replication was handled by a cron job.  The job would execute scripts which would take a snapshot of the master index, rsync the snapshot to the slave servers, and install the indexes.  This was a fairly clunky process – complicated to configure and maintain, and also limited to only *nix type systems.  A new feature in solr 1.4 is index replication over HTTP.  No external scripts are necessary.  All of the configuration is done in the config.xml files for each solr core you want to replicate across.

           

          Below outlines some of  the changes we made to set this up and what we saw. You may want to read  over the previous replication method and even  search server clustering before continuing.

          Master Config

          The config files are found under searchserver/WEB-INF/solrHome/conf.  To replicate product indexes, the following would need to be added to product.config.xml on the master server:

          <requestHandler name="/replication" class="solr.ReplicationHandler">
               <lst name="master">
          
                    <!--Replicate on 'startup' and 'commit'. 'optimize' is also a valid value for replicateAfter. -->
                    <str name="replicateAfter">startup</str>
                    <str name="replicateAfter">commit</str>
                    <!-- <str name="replicateAfter">optimize</str> -->
          
                    <!--Create a backup after 'optimize'. Other values can be 'commit', 'startup'. It is possible to have multiple entries of this config string.  Note that this is just for backup, replication does not require this. -->
                    <!-- <str name="backupAfter">optimize</str> -->
          
                   <!--If configuration files need to be replicated give the names here, separated by comma -->
                    <str name="confFiles">schema.xml,stopwords.txt,elevate.xml</str>
          
                    <!--The default value of reservation is 10 secs.See the documentation below . Normally , you should not need to specify this -->
                    <str name="commitReserveDuration">00:00:10</str>
               </lst>
          </requestHandler> 
          
          

           

          Slave Config

          The slave configuration is straight forward as well.  The most important part about slave configuration is the master url tag: <str name="masterUrl">http://{master IP}:{port}/searchserver/{core name}/replication</str>

          Make sure to put the correct core name in the url.  Since we are configuring replication for product indexes, the url is: http://{master IP}:{port}/searchserver/product/replication

           

          Add the following to the corresponding core config.xml on the slave servers:

           

          <requestHandler name="/replication" class="solr.ReplicationHandler" >
               <lst name="slave">
                    <str name="enable">${enable.slave}</str>
          
                    <!--fully qualified url for the replication handler of master . It is possible to pass on this as a request param for the fetchindex command-->
                    <str name="masterUrl">http://{master IP}:{port}/searchserver/product/replication</str>
          
                    <!--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically.
                     But a fetchindex can be triggered from the admin or the http API -->
                    <str name="pollInterval">00:00:20</str>
          
                    <!-- THE FOLLOWING PARAMETERS ARE USUALLY NOT REQUIRED-->
                    <!--to use compression while transferring the index files. The possible values are internal|external
                     if the value is 'external' make sure that your master Solr has the settings to honour the accept-encoding header.
                     see here for details http://wiki.apache.org/solr/SolrHttpCompression
                     If it is 'internal' everything will be taken care of automatically.
                     USE THIS ONLY IF YOUR BANDWIDTH IS LOW . THIS CAN ACTUALLY SLOWDOWN REPLICATION IN A LAN-->
                    <!--<str name="compression">internal</str>-->
          
                    <!--The following values are used when the slave connects to the master to download the index files.
                     Default values implicitly set as 5000ms and 10000ms respectively. The user DOES NOT need to specify
                     these unless the bandwidth is extremely low or if there is an extremely high latency-->
                    <str name="httpConnTimeout">5000</str>
                    <str name="httpReadTimeout">10000</str>
          
                    <!-- If HTTP Basic authentication is enabled on the master, then the slave can be configured with the following
                    <str name="httpBasicAuthUser">username</str>
                    <str name="httpBasicAuthPassword">password</str>
                    -->
          
               </lst>
          </requestHandler>
          
           
          

           

          Solrcore.properties Config

          You may have noticed the line: <str name="enable">${enable.slave}</str>  This property is grabbed from solrcore.properties.  You will have to create this file in the same directory as your config.xml file.  Open the solrcore.properties file and write: enable.slave=true.  Replication can be disabled on the slave server by setting this property to false.

           

          Summary

          That's all there is to it.   With this configuration, the slave server will poll the master every 20 seconds to check if their index versions are the same.  If the master has a new version of the indexes, the slave will start replication of any old files.  The files are stored into a temp folder until the download is complete.  This way if anything happens to the master during replication, there is no corruption of the indexes.  The master is unaware of the slaves.

           

          Performance Aside

          As for performance, I tested replication of indexes from a database with over three million products and skus.  The indexes were 5.2Gb in size, took 165 seconds to transfer, and the maximum bandwidth during the transfer was 8.7 mb/s.  Running the same replication test with two slave servers resulted in a maximum bandwidth of 15.5 mb/s and again 169 seconds to transfer the indexes.  Taking the average of the tests, the transfer rate is roughly 18,000 product indexes per second.  To see if the replication had any impact on performance, we ran our standard benchmarking test with 20-50 concurrent virtual users making requests to the storefront.  With the server under load, I continuously made changes to products, committing product indexes and forcing them to replicate to the slave servers.  There was no noticeable drop in throughput on the server, nor a spike in bandwidth usage.  Of course your mileage may change depending on a large number of factors, mostly including outside network noise.

          0 Comments Permalink

          In Star Trek there is a test named the Kobayashi Maru given to all Starfleet Academy cadets. In this test the civilian vessel Kobayashi Maru is disabled and trapped in the Klingon Neutral Zone, and the cadet must choose whether to leave the ship to certain destruction or to attempt a rescue but risk the lives of their own ship and crew and possibly provoke the Klingons into an all-out war. If the cadet chooses to attempt a rescue the simulation inevitably ends in battle and complete destruction of the cadet's ship. The test is designed as a no-win situation to test the cadet's character.

           

          How is this relevant to OpenJPA and EP you ask? In our case the Kobayashi Maru is your e-commerce store - stuck in the neutral zone between development and go live, disabled due to performance issues caused by a large object graph. The cadet is you, the developers. The Klingons represent the project stakeholders who will fight you to meet deadlines whilst requiring quality maintainable and functional code. What makes it a no-win situation is there is no simple way to tell OpenJPA to load just specific parts of the object graph. You have the following options:

           

          • Try to use the EP load tuners to flag which parts of the object graph you are not interested in.
            • For example, the ProductLoadTuner lets you specify setLoadingCategories(false) to avoid loading all of the categories that a product belongs to. Seems sound, right? The only problem is it doesn't work! For legacy reasons there are many service methods and queries that load a product (either directly or as part of another object graph) and expect the categories - so a long time ago the JPA annotations on ProductImpl.getProductCategories() was set to FetchType.EAGER. This means JPA will always load the categories. If you change the annotation to use FetchType.LAZY then you would need to find everywhere in the system that loads a product expecting to see categories and change it to use the appropriate load tuner. You don't have that much time, and so you are defeated by the Klingons.
          • Create a new OpenJPA FetchGroup which is a way of defining exactly which fields to load.
            • This is the strategy that has already been used in parts of the code - for example when building the product index. The drawbacks of this approach is that a fetch group will only load the specified fields, and you can only define a fetch group through annotations in the same classes that the fields are defined in. If you are trying to use Binary Based Development then this option is already disqualified. If you are happy to make core changes, then you need to edit every class in the object graph to define the fields that should be loaded as part of your fetch group. A quick search shows the PRODUCT_INDEX fetch group is defined across 20 classes! Miss one field in one class and you get an unexpected NullPointerException. You don't have time to do all the testing required to ensure you have all the necessary fields, and so you are either defeated by the Klingons or your ship implodes later due to a null pointer in the warp core.
          • Change the OpenJPAFetchPlanHelperImpl to remove the default fetch group.
            • This is a special fetch group created by OpenJPA (FetchGroup.NAME_DEFAULT) which includes all basic fields, *ToOne fields, and other fields marked with FetchType.EAGER. Remove this and you then need to add back all the fields that you want to load. So you are most likely in the same situation as above. Even if you try to do something clever like having a spring injected list of fields to include, this will need to be maintained in the future when new fields are added to any of the classes involved, and you still need to do exhaustive testing to ensure no essential field was missed. So the Klingons win again.

           

           

          I am now going to play the role of Captain James Kirk. Kirk stated "I don't believe in the no-win scenario", and his response to the Kobayashi Maru test was to subtely reprogram the simulator to make it possible to rescue the vessel. I'm going to reprogram our simulator using some of the OpenJPA API.

           

          We can use the OpenJPA metadata API to recreate the "default" fetch plan in such a way as to allow ad-hoc addition and removal from this plan. Here's a code snippet that does this:

           

          protected Set<String> getDefaultFetchFields(final Class< ? > persistenceClass, final EntityManager entityManager, final Set<Class<?>> configuredObjects) {
               Set<String> fetchFields = new HashSet<String>();
           
               if (configuredObjects == null) {
                    configuredObjects = new HashSet<Class<?>>();
               } else if (configuredObjects.contains(persistenceClass)) {
                    return fetchFields;
               }
               configuredObjects.add(persistenceClass);
           
               ClassMetaData classMetaData = JPAFacadeHelper.getMetaData(entityManager, persistenceClass);
               FieldMetaData[] fields = classMetaData.getFields();
               for (FieldMetaData field : fields) {
                    if (field.isInDefaultFetchGroup()) {
                         fetchFields.add(field.getFullName(false));
                         if (field.isTypePC()) {
                              fetchFields.addAll(getDefaultFetchFields(field.getType(), entityManager, configuredObjects));
                         } else if (field.getElement() != null && field.getElement().isTypePC()) {
                              getDefaultFetchFields(field.getElement().getType(), entityManager, configuredObjects);
                         }
                    }
               }
           
               return fetchFields;
          }
          

           

          The first block keeps track of any classes we have already processed since there can be self-references or bi-directional relationships in an object graph. We then use OpenJPA's JPAFacadeHelper class to get the metadata for the given class. This provides a collection of persistence fields - both in the current class and any of its superclasses. We loop through these fields and examine each one in turn. The field metadata will tell us if the field is in the default fetch group, and if so we add it to the list of fields to include. Next, we check if the field is itself a PersistenceCapable class, in which case we recurse down into the fields of that class, and so on. For a *ToMany relationship the field will have an element which may be PersistenceCapable so we recurse into that similarly.

           

          The results of all this is a list of (fully qualified) field names that are in the default fetch group for the entire object graph of the given class. We can then tweak this list, removing any fields we don't want loaded and adding other (non-default) fields we specifically want to load. We can then tell OpenJPA to remove the default fetch group, and give it our list as the definitive list of fields to load. For example:

           

          Set<String> productFetchFields = getDefaultFetchFields(beanFactory.getBeanImplClass(ContextIdNames.PRODUCT), entityManager, null);
          fetchPlan.removeFetchGroup(FetchGroup.NAME_DEFAULT);
          fetchPlan.addFields(productFetchFields);
          fetchPlan.removeField(ProductImpl.class, "productCategories"); 
          

           

          We now have a fetch plan that tells OpenJPA to load a product without the categories!

           

          Using this technique you can easily define to a fine degree exactly what gets loaded in your object graph and thus rescue the civilian vessel without arousing the wrath of the Klingons.

           

          In my next mission as captain of this starship I will refactor the OpenJPAFetchPlanHelperImpl to use this technique which will enable the load tuners flags to actually work as expected!  I'll also add methods to that class (and interface) to allow easy removal of fields and also a method for removal of all fields (except the uidPk) of a class. Sometimes having that object "reference" on the object graph is sufficient without requiring any of the other fields, and it would be tiresome to remove the fields from the plan individually. For an encore I will replace all the hard-coded @FetchGroup annotations with a spring-injectable plan which is more extension-friendly (whilst still preserving current behaviour).

           

          Feeback welcome, in the meantime Live Long and Prosper!

          10 Comments Permalink

          The performance of the OOTB CachingStoreResolver begins to degrade when more than a handful of stores are configured in EP. This post describes a simple replacement implementation that scales much better.

           

          The problem

           

          The StoreResolver is used by the storefront to resolve stores by either store code or domain. The OOTB CachingStoreResolver uses a SimpleTimeoutCache to save store codes and domain-to-store mappings. The default cache timeout is 1 minute. When an entry is not found in the cache, the request is delegated to the StoreResolver to resolve the request from the database and the result is cached if valid. All this looks pretty reasonable on the surface.

           

          The scalability problems are caused by how the cache is populated. If a store code or domain are not found in cache, then the StoreResolver reads ALL stores from the database and then returns only the matching store code. If there are 100 stores and the default cache timeout is used then we can expect cache entries to expire nearly twice every second. While this is not ideal, there wouldn't be much of an issue if reading a store were a lightweight query. However OpenJPA brings along a large number of related objects and the operation ends up being quite expensive.

           

          It is also worth noting that a request with an invalid store code or domain will always have a cache miss and will result in fetching all stores from the database.

           

          Two solutions

           

          The simplest approach is to increase the cache timeout, but this does not fix the issue of handling invalid store codes or domains.

           

          A more complete solution is to replace the CachingStoreResolver with the following class which caches all store codes or domain-to-store mappings with a single query.

           

          import java.util.Collection;
          import java.util.Map;

          import com.elasticpath.sfweb.util.impl.StoreResolverImpl;

           

          /**
          * This re-implements the OOTB CachingStoreResolver to cache all stores in a single database query.
          */
          public class ReplacementCachingStoreResolverImpl extends StoreResolverImpl {

           

              /**
               * Get the set of valid store codes.
               *
               * @return a collection of all codes for complete stores.
               */
              @Override
              protected Collection<String> getStoreCodes() {
                  final String cacheKey = "storeCodes";
                  Collection<String> storeCodes = (Collection<String>) getFromCache(cacheKey);
                  if (storeCodes == null) {
                      storeCodes = super.getStoreCodes();
                      addToCache(cacheKey, storeCodes);
                  }
                  return storeCodes;
              }

           

              /**
               * Creates a map with all the store domains mapped to their store codes.
               *
               * @return a map populated with all store domains mapped to
               *            their corresponding store codes.
               */
              @Override
              protected Map<String, String> getDomainToStoreMapping() {
                  final String cacheKey = "domainToStoreMap";
                  Map<String, String> storeCodeMap = (Map<String, String>) getFromCache(cacheKey);
                  if (storeCodeMap == null) {
                      storeCodeMap = super.getDomainToStoreMapping();
                      addToCache(cacheKey, storeCodeMap);
                  }
                  return storeCodeMap;
              }

           

              private Object getFromCache(final String cacheKey) {
                  // TODO - Hook in your cache of choice, could be a SimpleTimeoutCache
                  return null;
              }
             
              private void addToCache(final String cacheKey, final Object obj) {
                  // TODO - Hook in your cache of choice, could be a SimpleTimeoutCache
              }

           

          }

          3 Comments Permalink

          Why is memory never enough?

           

          Our customer project has a few dozens of integration JUnit tests.  Those are tests coded as JUnit tests loading the core application Spring context to verify the correct behavior of certain parts of our application. They have been pretty useful proving us right and a lot of  times wrong after we have committed something into our source code repository.

           

          The only problem that we were facing was that the tests were really slow. They would take 6, 8, even 12 hours sometimes in a Hudson build. That was a little suspicious and by taking a closer look at one of the integration test runs using Visual VM we  managed to find out that we were having a memory leak. The max heap size  would reach 1.5GB and because of the huge overload of garbage collection our tests were taking a lot more time than they should have. The JVM was trying hard to collect whatever it could to fit the tests into this pretty big heap.

           

          One interesting thing was that after moving the project from JDK 5 to JDK 6 the tests started failing much quicker with an OutOfMemoryError and they were taking 2 hours less in total. The error would print the mysterious

          java.lang.OutOfMemoryError: GC overhead limit exceeded.

           

          It was the first time I had seen such an error and after some googling it turned out that JDK 6 has this new feature to fail the JVM with an out of memory error sooner rather than later.
          They also added an option to suppress/enable that feature by adding -XX:-UseGCOverheadLimit to the JVM arguments.

           

          After  adding the option -XX:+HeapDumpOnOutOfMemoryError the JVM produced  valuable heap dumps on an out of memory errors. Using the Memory  Analyzer Tool (MAT) we were able to see where the memory went.

          Surprisingly,  the JVM was always crashing on two particular test suites. Both of them having 40+ test cases. The biggest heap consumer was an array of objects pointing to instances of either of the test suite classes.

           

          Here's what was uncovered after some investigation:

          • JUnit creates a new instance of the test suite class every time a new test case is launched
          • JUnit will not invoke TestCase.tearDown() if an exception occurs in TestCase.setUp()
          • JUnit keeps reference to all test cases in a test suite until the test suite completes
          • Having fields in your test suite class can hold reference to objects that otherwise would be due for garbage collection

           

          tearDown() or rather tearDownHappyCase()

           

          Because our tests had a few fields holding reference to various services and the test application context (based on Spring  application context) this was making JUnit overload the memory with  ~40MB per test case. Until a test suite would complete we were ending up with 40 test cases x 40MB = ~1.6GB of RAM.

           

          The most  obvious way of fixing that was to make sure all the fields in the test class were nullified so that the objects referenced would be garbage collected.

          In JUnit tearDown() is the best location for that. The problem was that in case while setting up a test case an error occurs  JUnit will continue with the next test case without even bothering invoke tearDown(). This was causing even more troubles with the test cases memory footprint.

          Our setUp() method was modified to look as follows:

           

          public void setUp() throws Exception {
              try {
                  super.setUp();
              } catch (Throwable exc) {
                    tearDown();
                    throw exc;
             }
          }
          

           

          And the tearDown() method looked like:

          public void tearDown() {
             this.productService = null;
             this.productSkuService = null;
          ...
           
          }
          

           

          Even though not so beautiful this allowed us to get the overall memory usage to reasonable levels and sped up our test cases quite a bit.

           

          JUnit 4

           

          Interestingly enough the test workflow in JUnit 4 is different. There are the @Before and @After annotations to be used on methods in your test case. Even if the method(s) annotated with @Before throw any error JUnit 4 will still execute all the @After annotated methods.

          This means the the above mentioned problem will not occur in JUnit 4 test cases.

          It's good to know things have been improved...

          2 Comments Permalink

          Introduction

           

          EclEmma is a free code coverage tool; it works by determining which  parts of Java code are executed during particular  program launches.  For our purposes, these will likely be JUnit test programs.

           

          EclEmma:

          • installs as a plugin into Eclipse
          • both individual programs or entire suites can be checked with a single mouse click
          • generates code coverage report
          • highlights which areas of the source code are executed by the tests

           

          Installation

          1. In Eclipse, select Help > Install New Software...

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-install0.png?version=1&modificationDate=1307650332000

          2. In the Install dialog enter http://update.eclemma.org/ at the Work with field.

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-install1.png?version=1&modificationDate=1307650332000

           

          3. Click the checkbox for EclEmma and click Next.

           

          4. Follow the remaining steps in the installation wizard.

           

          Usage

           

          Once EclEmma is installed, the Coverage As option should become available in the right-click menu.

           

          To run EclEmma on a test or suite of tests, right-click one of the following:

           

          • the Package or source folder in the Package Explorer
          • the Project Name or source folder in the Project Explorer
          • on the test source code itself in the code view

           

          Then select Coverage As > JUnit Test.

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-usage1.png?version=1&modificationDate=1307650332000

           

          The test or test suite will then execute as normal in the JUnit View.

           

          Coverage View

           

          Directly afterward, the Coverage View will open:

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-usage3.png?version=1&modificationDate=1307650332000

           

           

          The columns show Coverage percentage, Items covered, Items not covered, and Total items

           

          Note that it's possible to expand each Element and drill down to code at the method level.

           

           

          Source Code Highlighting

           

          In the Source Editor panel, the code will be highlighted with green, yellow, or red to indicate code coverage.

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-usage4.png?version=1&modificationDate=1307651283521

           

          • green for fully covered lines
          • yellow for partly covered lines
          • red for lines that have not been executed at all

           

          Note that all executed code is highlighed, which includes the JUnit tests.

           

          All code will stay highlighted until the Coverage session is cleared or the file is edited.

           

           

          Removing Coverage Sessions and Source Code Highlighting

           

          To remove a Coverage session and its associated results and source  code highlighting, click the grey 'X' icon on the toolbar on the  coverage tab.

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-remove-coverage-session.png?version=1&modificationDate=1307652807037

           

          If the source code highlighting does not disappear, you must delete additional Coverage sessions.


          Filtering Results

           

          To focus on a particular unit, you can filter out all classes which   have not been loaded during the test run.

           

          1. Select the Coverage View drop-down menu by clicking the down arrow.

           

          https://wiki.elasticpath.com/download/attachments/57639629/eclemma-drop-down.png?version=2&modificationDate=1307653705758

           

          2. Select Hide Unused Types in the Coverage View drop-down menu.

           

          http://www.eclemma.org/userdoc/images/coverageviewmenu.png

           

          Tip: Selecting Show Types in the drop-down menu along with Hide Unused Types shows a simple list of all classes loaded for the specific test case.

           

          Additional Information

           

          A detailed user guide may be found at http://www.eclemma.org/userdoc/index.html

          1 Comments Permalink

          I have a love/hate relationship with Eclipse.  It's a great editor with sane default key mappings, but it tries to be too much.  One thing that never worked all that well is the WTP plugin for managing application servers.


          How many times have you refreshed/published/republished Tomcat in Eclipse?  Sometimes the configuration files get out of sync and you have to clean it.  And since all the wars end up in the same Tomcat instance the start up time is horrendous.


          An alternative to WTP is the Tomcat Maven plugin.  It allows you to manage Tomcat servers, but what I want to show you is how use it to run a war project in an embedded Tomcat instance.


          Unfortunately our war projects are not fully Mavenized (yet), but we can get around that by using the fact it is just a war in the end.  This allows us to use the Maven War plugin to create an overlay over our war projects and then run that in Tomcat.


          So enough pre-amble, show me the code!  Let’s get com.elasticpath.sf running using the Tomcat Maven plugin and MySQL.


          First we need to get some environment specific settings out of the way.  Since we are doing a war overlay we won’t have access to any of the settings in env.config.  Instead we’ll put our environment specific settings in ~/.m2/settings.xml:

          <?xml version="1.0"?>

          <settings>

            <activeProfiles>

              <acitveProfile>mysql-dev-db</activeProfile>

              <activeProfile>storefront-tomcat-properties</activeProfile>

              <activeProfile>keystore-properties</activeProfile>

            </activeProfiles>

                                 

            <profiles>

              <profile>

                <id>storefront-tomcat-properties</id>

                <properties>

                  <storefront.context>/storefront</storefront.context>

                  <storefront.http.port>8080</storefront.http.port>

                  <storefront.https.port>8443</storefront.https.port>

                </properties>

              </profile>

              <profile>

                <id>keystore-properties</id>

                <properties>

                  <keystore.file>/your/very/own/.keystore</keystore.file>

                  <keystore.pass>changeit</keystore.pass>

                </properties>

              </profile>

            </profiles>


                 …


          </settings>


          Other war projects will need their own profile.  The profile mysql-dev-db will come from the grandparent pom.  It is available since 6.3 and the source is included under elasticpath-grandparent/pom.xml.  If the default properties defined in the mysql-dev-db profile are unsuitable for your needs, they can be overriden using another profile in ~/.m2/settings.xml and activate it after the mysql-dev-db profile.


          We then create a new Maven project called storefront which will overlay com.elasticpath.sf.  Here we will use the released version of the grandparent pom and define the parent of the storefront project as the grandparent pom:

          <?xml version="1.0" encoding="UTF-8"?>

          <project

          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"

          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

            <modelVersion>4.0.0</modelVersion>


            <parent>

              <groupId>com.elasticpath</groupId>

              <artifactId>grandparent</artifactId>

              <version>16</version>

            </parent>

            <groupId>com.elasticpath.extensions</groupId>

            <artifactId>storefront</artifactId>

            <version>1.0-SNAPSHOT</version>

            <packaging>war</packaging>

            <name>Storefront Extension</name>


            <dependencies>

              <dependency>

                <groupId>com.elasticpath</groupId>

                <artifactId>com.elasticpath.sf</artifactId>

                <version>...your Elastic Path version...</version>

                <type>war</type>

              </dependency>

            </dependencies>


          Now define the Tomcat Maven plugin configuration.  The path and port configuration is defined in the settings profile.  And finally we choose the JDBC driver.  Yes this means you never again have to figure out where is Tomcat’s shared library folder.  All the epdb.* properties are defined in the mysql-dev-db profile.

            <build>

              <plugins>

                <plugin>

                  <groupId>org.codehaus.mojo</groupId>

                  <artifactId>tomcat-maven-plugin</artifactId>

                  <version>1.1</version>

                  <configuration>

                    <path>${storefront.context}</path>

                    <port>${storefront.http.port}</port>

                    <httpsPort>${storefront.https.port}</httpsPort>

                    <keystoreFile>${keystore.file}</keystoreFile>

                    <keystorePass>${keystore.pass}</keystorePass>

                  </configuration>

                  <dependencies>

                    <dependency>

                      <groupId>${epdb.maven.groupId}</groupId>

                      <artifactId>${epdb.maven.artifactId}</artifactId>

                      <version>${epdb.maven.version}</version>

                    </dependency>

                  </dependencies>

                </plugin>


          We need to define the jdbc/epjndi resource as a MySQL datasource.  This is done with META-INF/context.xml.  To get our settings into context.xml we’ll filter it using the Maven War plugin:

                <plugin>

                  <groupId>org.apache.maven.plugins</groupId>

                  <artifactId>maven-war-plugin</artifactId>

                  <version>2.1.1</version>

                  <configuration>

                    <webResources>

                      <resource>

                        <directory>src/main/webapp-filtered</directory>

                        <filtering>true</filtering>

                      </resource>

                    </webResources>

                  </configuration>

                </plugin>

              </plugins>

            </build>

          </project>


          That will filter any file in src/main/webapp-filtered and put the result in the root of the built war.  This allows us to create src/main/webapp-filtered/META-INF/context.xml with the contents:

          <Context>

            <Resource auth="Container" name="mail/Session" type="javax.mail.Session" />

            <Resource

              name="jdbc/epjndi"

              auth="Container"

              scope="Shareable"

              type="${epdb.datasource.classname}"

              maxActive="100"

              maxIdle="30"

              maxWait="10000"

              removeAbandoned="true"

              username="${epdb.username}"

              password="${epdb.password}"

              driverClassName="${epdb.jdbc.driver}"

              url="${epdb.url}" />

          </Context>


          ...and have it filled out by the mysql-dev-db profile.


          Our storefront project depends on com.elasticpath.sf so make sure it's in your local repo by running ant install under com.elasticpath.sf.


          Since we are running Tomcat with Maven in order to pass JVM arguments to Tomcat you’ll need to configure the environment variable MAVEN_OPTS instead of CATALINA_OPTS.  Note for Java 6 users to copy over their special JVM args to MAVEN_OPTS.  This also means remote debugging parameters should be set in MAVEN_OPTS.  For more information on remote debugging see: http://java.dzone.com/articles/how-debug-remote-java-applicat


          Finally we run our storefront project with

              mvn tomcat:run-war


          Storefront is now running under http://localhost:8080/storefront and can be killed with ^C.


          Repeat for the remaining wars and you can have each war project running in a separate embedded Tomcat instance.  Just remember to give each instance its own port and update settings like COMMERCE/SYSTEM/SEARCH/searchHost.  See http://mojo.codehaus.org/tomcat-maven-plugin/index.html for more details on how to configure the run-war goal.  Special care is needed to ensure each project has its own remote debugging ports.  This is accomplished by setting different MAVEN_OPTS for each project.  Simply use

              MAVEN_OPTS=”$MAVEN_OPTS <project specific opts>” mvn tomcat:run-war

          This will temporary append project specific options to MAVEN_OPTS.  There is no semicolon before mvn because we only want to set this variable for this process and not back to the shell.


          This gives you great control to start and stop individual wars.  Only changed com.elasticpath.sf?  Just ant install and restart that Tomcat instance.


          While it took a little bit of work to get the environment specific settings in the right place this now gives us a platform to use more Maven without having to wait for the full Mavenization of our projects.  Let’s get cracking!

          0 Comments Permalink

          Overview

           

          'A picture is worth a thousand words' is a really good saying and it proves to be true in many situations. At Elastic Path, we use a tool called OpenJPA Grapher that helps us build a complex graph of entities that are part of the Elastic Path Commerce platform. Having a tool like this can help you visualize the relationships between your entities and keep you up to date with the multiple changes that occur in a fast-paced development environment.

           

          How does it work?

           

          OpenJPA enhances all the unenhanced entity classes at build time. When the entities are being enhanced, OpenJPA collects all the metadata it needs. The metadata consists of the type of relationship between two entities, the cascade type supported, and various other properties.

          In the attached openjpa-grapher-0.0.1-SNAPSHOT-sources.jar, you will find a class (Grapher.java) that hooks to the OpenJPA enhancer. For every entity that goes through the OpenJPA enhancer, Grapher.java reads the metadata and builds a link between that entity and any other referenced entities. The class uses the notion of an AuxiliaryEnhancer, provided by the OpenJPA library.

          The resulting graph file is in text format and includes the entities and their type of relationship. You will see something like 'orderInternal M  R' where the first part of the string is the field name in the class and the letters are denoting the cascade type that could be M - Merge, R - Refresh, D - Delete, P - Persist.

           

          Let's look into how to set up the OpenJPA Grapher and generate a visual graph.

           

          Building GraphViz graph

           

          GraphViz, as you can tell by the name, is a visualization tool that helps you build different graphs and diagrams to help you represent certain information in a more user-friendly format.

          The tool supports different formats. We have been using the 'dot' format for directed graphs.

           

           

             1.  Installing the grapher component

                Place the attached com.elasticpath.openjpa.grapher.jar and pom files into your local maven repository.

          mvn install:install-file -Dfile=com.elasticpath.grapher-0.0.1-SNAPSHOT.jar -DpomFile=com.elasticpath.grapher.pom
              2. Adding a dependency to the grapher in com.elasticpath.core/pom.xml

                This allows the grapher to hook to the OpenJPA Enhancer. The dependency jar file contains a class name in META-INF/services, which is the way it gets plugged in.

                Add the following to the dependencies section in com.elasticpath.core/pom.xml

              <dependency>
                <groupId>com.elasticpath</groupId>
                <artifactId>openjpa-grapher</artifactId>
                <version>0.0.1-SNAPSHOT</version>
                <scope>compile</scope>
              </dependency>

           

               3. Generating the openjpagrapher.dot file

                  Run the following commands to generate the dot file:

               > cd com.elasticpath.core
               > ant clean compile
          

           

               The openjpagraph.dot file is saved in the com.elasticpath.core folder.

              Note that you must call clean when building the graph, otherwise only classes modified after the last enhance are added to the graph.

           

               4. Installing the GraphViz library and generating the SVG file

                 GraphViz converts the dot format into a scalable vector graphics (SVG) format. You can find a GraphViz installation information for your OS on their download page.

                 After you have installed it, run the following command:

               > dot -Tsvg openjpagraph.dot -odomain-relations.svg

           

          How to view the graph

           

          In a Browser

          The easiest way is to open the file domain-relations.svg in a browser like Safari. Here's how it looks like:

          SVGinBrowser.png

          As you can see the graph is quite big and cannot be easily browsed.

           

          Standalone ZGRVIewer

          ZGRViewer is an open source project that allows you to easily view and browse graphs generated with GraphViz. You can download ZGRVIewer from this page.

          I used SVN to download the latest trunk version of the tool as described on the download page. ZGRVIewer is a maven project so it can be easily built using mvn package.

          Now that you have the tool built, you can go to the target folder and run the viewer with:

          > cd zgrviewer/trunk/target
          > java -jar zgrviewer-0.9.0-SNAPSHOT.jar
          

          Here's what you should see.

          ZGRViewerStandalone.pngZGRViewerStandaloneOverview.png

          The main advantage of ZGRViewer is that the graph is zoomable and you can choose from a few tools to actually see the graph relationships in a better way.

           

          ZGRViewer as an Applet

          A good feature of the ZGRViewer is that you can run it as a Java Applet in a browser thus giving you the best of both worlds - zoomable graph browsing and having the graph viewable without having to download the standalone tool.

          In order to do that, we can create a simple web page with the applet embedded.

           

          <html>
            <body>
              <div id="main">
                <applet code="net.claribole.zgrviewer.ZGRApplet.class"
                        archive="zvtm-core-0.11-SNAPSHOT.jar,zvtm-svg-0.2.0-SNAPSHOT.jar,zgrviewer-0.9.0-SNAPSHOT.jar,timingframework-1.0.jar"
                        width="720" height="480">
                  <param name="type" value="application/x-java-applet;version=1.5" />
                  <param name="scriptable" value="false" />
                  <param name="width" value="720" />
                  <param name="height" value="480" />
          
                  <param name="svgURL" value="/~yzhivkov/openjpa/domain-relations.svg" />
                  <param name="showNavControls" value="true" />
                  <param name="showFCPalette" value="true" />
                  <param name="title" value="Domain Object Graph" />
                  <param name="appletBackgroundColor" value="#DDD" />
                  <param name="graphBackgroundColor" value="#DDD" />
                  <param name="highlightColor" value="red" />
                  <param name="displayOverview" value="true" />
                  <param name="focusNodeMagFactor" value="2.0" />
                </applet>
              </div>
            </body>
          </html>
          

          The most important parameter here is svgURL that must point to the SVG file.

          All you need to do is copy the jars into an http server folder along with the above code as an html page (e.g. openjpa.html).

          The listing of your folder should be similar to the following:

          antlr-2.7.7.jar
          domain-relations.svg
          jcip-annotations-1.0.jar
          junit-3.8.1.jar
          openjpa.html
          timingframework-1.0.jar
          xercesImpl-2.8.1.jar
          xml-apis-1.3.03.jar
          xmlParserAPIs-2.6.2.jar
          zgrviewer-0.9.0-SNAPSHOT-sources.jar
          zgrviewer-0.9.0-SNAPSHOT.jar
          zvtm-core-0.11.0-SNAPSHOT.jar
          zvtm-svg-0.2.0-SNAPSHOT.jar
          

          You can use Apache Web Server (httpd) to run the example like I did, but in general any web server should do.

          The result would be a web page with the applet running on it.

          ZGRViewerApplet.png

          Conclusion

          The OpenJPA grapher can give you a good overview of your domain model in terms of what entities exist in your application.

          A good thing about the ZGRViewer Applet is that you can even integrate the whole process with your favorite continuous integration server (Hudson, Jenkins, Bamboo, etc.) and have a consistent, up-to-date visual representation of your entity relationships.

           

          References

           

          OpenJPA - http://openjpa.apache.org

          GraphViz - http://www.graphviz.org

          ZGRViewer - http://zvtm.sourceforge.net/zgrviewer.html

          Auxiliary Enhancer - http://ivansthunks.com/blog/tag/auxiliaryenhancer/

          1 Comments Permalink

          One of the quirks that I used to have while developing using binary based development is the fact that we have multiple maven projects, and changes to one of the projects (ie. core) will need to be rebuilt in other projects that depend on it (ie. storefront).

           

          This was typical what I had to do:

           

          cd project-root
          cd core-project
          mvn clean install
          

          <take a nap for a few minutes>

          <check to make sure the project built successfully>

          cd ../storefront-project
          mvn clean install
          

           

          The other option is to build all the projects in the workspace.

           

          cd project-root
          mvn clean install
          

          <take a longer nap for 10 minutes>

           

          That is actually ineffective and a waste of time. One could argue you can write a script that will invoke those commands for you. However, you'd have to check for cases whent the build fails. Luckily, I found that there is a maven plugin that helps with automating that for us.

           

          This is where the Maven Reactor plugin comes to the rescue.

           

          The solve the previous problem where you changed your core project and only want to build the storefront, all you have to do is type this in the project-root directory:

           

          mvn reactor:make -Dmake.folders=storefront-project

           

          The Maven Reactor plugin will build core-project and then the storefront-project for you. If there is a problem with the core project (ie. compilation failures), it will stop the build and tell you.

          4 Comments Permalink

          Overview:

          Elastic Path 6.2.2 comes with a little known project called "com.elasticpath.test.application". This project is used as a connector to enable integration testing by exposing the spring application context to hook up Elastic Path core services for retrieving, updating, and removing EP domain objects. This article will go through the steps on how to use this project for integration testing.

           

          Why?

           

          Our client is using EP's web services to perform shopping related operations, and while we can test the web service calls with a framework like Groovy, what we also needed was the ability to check the backend to ensure the result of the web service call has been persisted in the database. We also needed to manipulate the data objects so we could test the edge cases through the web service. We could have written raw SQL statements to insert these rows, but then again you would have to insert the object graph of these domain objects. The downside to this is if one were to add a new column or remove an existing column, these SQL statements will have to be changed.

           

          The Solution:

           

          We rely on one of the projects that come packaged seperately in 6.2.2: com.elasticpath.test.application. This project will give us hooks into Spring and the service layer which we can then use to manipulate the domain objects. In addition, it provides us with a few Factory methods which we can use to create and persist certain domain objects for testing.This project was originally designed for use with FIT tests, but with a little bit of tweaking we can use it to our advantage.

           

          The Setup:

           

          - Elastic Path 6.2.2, taking advantage of the binary-based development approach

          - Java 6

          - Maven 2.2.1

           

          The Details:

           

          Step 1: Create your Integration test project.

           

          In the pom file reference these jars. Here I am assuming you have a core extension project already, such that it is available for reference in this test project.

           

               <dependencies>
                  ...
                  <dependency>
                      <groupId>com.elasticpath</groupId>
                      <artifactId>com.elasticpath.test.application</artifactId>
                      <version>6.2.2</version>
                      <scope>compile</scope>
                  </dependency>
                  <dependency>
                      <groupId><your group id></groupId>
                      <artifactId><your core artifact id></artifactId>
                      <version><your version #></version>
                      <scope>compile</scope>
                  </dependency>
                  <dependency>
                      <groupId>com.elasticpath</groupId>
                      <artifactId>com.elasticpath.core.resources</artifactId>
                      <version>6.2.2</version>
                      <scope>compile</scope>
                  </dependency>
                    ...
                </dependencies>
          

           

          NOTE: We need the com.elasticpath.core.resources package because it contains the out-of-the-box Spring bean definitions that we will use later.

          Step 2: Copy the files from the com.elasticpath.core.itest project into our new project.

           

          The reason we need this is because we need to set up the database configuration and spring definitions specific for integration tests. Copy the two files integrationtests/test_application.config and integration-core-context.xml into your Maven test project's src/main/resources folder.

           

          Step 3: Modify your test_application.config file.

           

          Change the setting in the file to correspond to the line below. We need to perform this change because all of our required spring definitions come from this file.

          context.definitions_file=integration-core-context.xml

           

          Step 4: Modify your integration-core-context.xml file.

           

          Here we will need to do a couple of things. First, append this line to the end of the file:

              <import resource="classpath*:META-INF/conf/plugin.xml" />

           

          This will import your definition from your core extension project.

           

          The next step involves an examination of the integration-core-context.xml file. Due to the fact that there are still some FIT test references in this file, we will have to look for them and remove them as neccessary. Remove the beans with the following IDs:

           

          importJobRunnerBaseAmount
          importJobRunnerBaseAmountForFit
          baseAmountDtoInsertUpdateImporterForFit
          contentWrapperLoader
          contentWrapperRepository
          velocityRenderer
          checkoutEventHandler
          

           

          Step 5: Modify the com.elasticpath.test.application project.

           

          A couple of modifications to the com.elasticpath.test.application project are required in order for us to use it. First off, we will need to add the following to TestApplicationContext.java:

           

          A.) Change the variable TEST_APPLICATION_CONFIG_FILE to read as shown on the next line. This is because the location of the file has changed.

              private static final String TEST_APPLICATION_CONFIG_FILE = "test_application.config";
          

           

           

          B.) Change the method overrideAllDefinitionsToLazy() to correspond to the code block below. We need to do this because we want to load the additional prototype beans defined by the BeanRegistrar.

              private void overrideAllDefinitionsToLazy() {
                  for (int i = 0; i < beanFactory.getBeanDefinitionNames().length; i++) {
                      String name = beanFactory.getBeanDefinitionNames()[i];
                      if (name != null && !name.startsWith("beanRegistrar")) {
                          AbstractBeanDefinition beanDefinition = (AbstractBeanDefinition) beanFactory.getBeanDefinition(name);
                          beanDefinition.setLazyInit(true);
                      }
                  }
              }
          

           

          C.) Inside the initializeBeanFactory() method, make the change described below. We need to do this because we are now loading the file from the Classpath.

                  beanFactory = new XmlBeanFactory(new FileSystemResource(definitionsFile));  
          

                    to

                  beanFactory = new XmlBeanFactory(new ClassPathResource(definitionsFile));
          

           

          D.) Replace the method getApplicationProperties() with the following code block below. We need to do this because we are now loading the file from the Classpath.

              private Properties getApplicationProperties() {
                  Properties properties = new Properties();
                  try {
                      InputStream inStream = this.getClass().getClassLoader().getResourceAsStream(TEST_APPLICATION_CONFIG_FILE);
                      properties.load(inStream);
                      
                  } catch (IOException e) {
                      throw new TestApplicationException("Failed to load properties from file:" + TEST_APPLICATION_CONFIG_FILE, e);
                  }
                  return properties;
              }
          

           

          Step 6: Rebuild the com.elasticpath.test.application project by navigating to the project root directory and issuing the following command:

          ant clean jar

           

          Step 7: Now test that your integration project works by creating a test called MyTest.java under src/test/java in the new integration test project and copy the following into that file:

           

          public class CreateWarehouseTest {
           
            private StoreTestPersister storeTestPersister;
           
            @Before
            public void setUp() {
              TestApplicationContext tac = TestApplicationContext.getInstance();
              tac.useDb();
           
              storeTestPersister = tac.getPersistersFactory()
                  .getStoreTestPersister();
            }
           
            @Test
            public void testCreateWarehouse() {
              storeTestPersister.persistDefaultWarehouse();
            }
          

           

          Step 8: Now configure your test_application.config file in your integration test project to point to your database and the initial database script location, and then run the test you just created.

           

          db.script.dir=<your db script dir>

           

          You should see the warehouse being persisted to the database.

          0 Comments Permalink
          1 2 3 ... 7 Previous Next