Background
Our client's e-commerce store has been powered by Elastic Path for well over a year now. Our team started development of the store on a beta version of Elastic Path 6.0 in January 2008. After 3 months of development, including a merge with the public release of Elastic Path 6.0, we launched the store in March 2008. We continued to add new features and bug fixes. We even took advantage of EP 6.0's multi-store support to add a few more stores in August 2008. At the beginning of 2009, after a year's worth of customizations atop the client's EP 6.0 deployment, we decided that it was time to upgrade to EP 6.1.
Why Upgrade?
Upgrading a customized version of Elastic Path Commerce is not a trivial process. So you need to ask yourself (and the client) if an upgrade is even necessary. The first step of upgrading Elastic Path is determining whether the benefits of upgrading exceed the costs of upgrading. The cost is time and money.
What are the benefits? Here are some of ours:
- EP 6.1 provides full multistore. In EP 6.0, multiple stores required multiple storefront WAR files: one for each store. EP 6.1 runs multiple stores from a single WAR file. Less WAR files translates to quicker deployments and a smaller resource footprint. This is especially relevant to us since our client is running six stores.
- EP 6.1 comes with 300+ bug fixes.
- Upgrading to EP 6.1 paves the way for upgrades to future versions of Elastic Path. This is crucial, because our client is anticipating features that are planned in future versions of EP (e.g. import/export of promotion data, available in EP 6.1.1).
- EP 6.1 comes with FIT integration tests.
- EP 6.1 introduces the import/export utility, used to migrate catalog data between databases.
- EP 6.1 introduces the settings framework, an improved way to manage configuration settings.
- EP 6.1 allows search index rebuilding to be triggered from the CM Client.
- The Store configuration UI has been overhauled in EP 6.1; the Store wizard has been replaced by a Store editor.
- In EP 6.1, assets (e.g. velocity templates, images, etc.) are stored outside the web application, making the view layer much easier to modify.
We presented a list of EP 6.1 benefits to our client, provided an estimate on the amount of time it would take to complete the upgrade, and got the green light to go ahead with it. These were the benefits relevant to us, but EP 6.1 has many other improvements. For example, here are 7 Big Code Changes in EP 6.1.
The Upgrade
After we convinced ourselves and our client that upgrading to EP 6.1 was worthwhile, we moved on to the fun part: the upgrade itself. There are several steps that need to be completed during an upgrade and it's easy to become overwhelmed at first, so we broke the upgrade down into more manageable sub-tasks:
- Database updates
- Merging the code
- Moving customized named database queries
- Using the new settings framework
- Relocating the assets directory
- Setting up Maven
1. Database Updates
The database schema did not change significantly between EP 6.0 and EP 6.1. Elastic Path has a script that upgrades the database from EP 6.0.x to EP 6.0.3 and another that upgrades it from EP 6.0.3 to EP 6.1. We ran both of these scripts against our database with no problems. Most of the schema changes consists of new tables and new columns and there are few changes to existing columns. Click here for detailed information about the EP 6.1 database updates.
After upgrading the schema with the scripts, a few data updates needed to be made:
- By default, full credit card numbers are stored in the database for each order; we turned this off (the ability to turn it off is a new EP 6.1 feature). This is accomplished by setting STORE_FULL_CREDIT_CARDS to 0 for each row in TSTORE.
- Populate the new TSTORESUPPORTEDLOCALE table with store-locale mappings; these mappings mirrored the catalog-locale mappings already in TCATALOGSUPPORTEDLOCALE.
- Populate the new TSTORESUPPORTEDCURRENCY table with the store-currency mappings; these mappings are similar to the catalog-currency mappings already in TCATALOGSUPPORTEDCURRENCY.
The majority of the time spent on the upgrade was in this step: merging code. Generally, the more changes you make to a codebase, the harder it is to upgrade in the future. I say "generally" because there are ways to customize code that minimize future code merge conflicts. For example, this GREP technical blog post shows how to customize code with the decorator design pattern; a strategy that significantly reduces code conflicts.
The easiest way to merge large amounts of code is by using your version control software's merge tools combined with vendor branches. We have a great GREP article explaining what vendor branches are and how they can be used for merging code here. Since we use subversion for this particular project, we ended up using svn merge to merge the code.
This is the svn merge command we used:
svn merge https://svn.elasticpath.com/perfect_code/pd/ep5/tags/publicrelease6.0/ https://svn.elasticpath.com/perfect_code/pd/ep5/tags/publicrelease6.1/ . --accept postponeThis command takes the differences between OOTB EP 6.0 and OOTB EP 6.1 (whose vendor branches are specified in green and blue respectively), and applies those differences onto our customized codebase (specified in red, this command was run from our project's root level directory). The --accept postpone part tells the utility to postpone resolving any merge conflicts that occur.
The following diagram shows the branches referred to in the svn merge command and our desired "Customized EP 6.1" end state:
When svn merge hits a conflicted file, it produces 3 output files. For example, consider a file called MyClass.java that is conflicted; svn merge will produce the files MyClass.java.left, MyClass.java.right, and MyClass.java.working. The highlighted colors correspond to the branches in the svn merge command above. The .left file is the EP6.0 version, the .right file is the EP 6.1 version, and the .working file is our customized codebase version.
When svn merge hits a conflicted file, it produces 3 output files. For example, consider a file called MyClass.java that is conflicted; svn merge will produce the files MyClass.java.left, MyClass.java.right, and MyClass.java.working. The highlighted colors correspond to the branches in the svn merge command above. The .left file is the EP6.0 version, the .right file is the EP 6.1 version, and the .working file is our customized codebase version.
The utility does a pretty good job at merging the source files. Some stats we gathered after running this command:
- 3682 files were added
- 118 files were deleted
- 953 files were automatically merged successfully
- 132 files were conflicted and not automatically merged
There are many approaches to manually merging a conflicted file. The .left, .right, and .working files produced by svn merge, described above, are invaluable to this process.
This is my strategy:
- diff (i.e. compare) the .left file (EP 6.0 version) with the .right file (EP 6.1 version); this shows Elastic Path's changes to the file from version 6.0 to 6.1.
- diff the .left file (EP 6.0 version) with the .working file (your customized version); this shows your customizations to the file from 6.0 to its current state.
- Based on the comparisons above, who changed the file more? Elastic Path or you?
- If Elastic Path made more changes to the file, start with the .right file and incorporate your changes (indicated by the diff in step 2) to the file.
- If you made more changes to the file, start with the .working file and incorporate Elastic Path's changes (indicated by the diff in step 1) to the file.
- If they have the same amount of changes, it doesn't really matter which version you pick as your starting point; the amount of work will roughly be the same. I lean towards picking the .right file (EP 6.1 version) because aligning your codebase to Elastic Path's will make future upgrades smoother.
- After merging the file in step 3, you are done. Move onto the next file!
- 88 were relatively easy/trivial merges
- 23 were moderately complicated merges
- And 21 were very complicated merges
All in all, the code merge is likely not as painful as most anticipate it to be. Even with 40+ files requiring non-trivial manual merges, our code merge went pretty smoothly.
A couple tips:
3. Moving Customized Named Database Queries
- Take advantage of dry-run modes of your merging tool to preview how much work needs to be done without actually performing the merge. For example, adding the --dry-run option to our svn merge command enabled us to see how many merge conflicts we'd have to resolve in advance of the merge itself; this information is very useful during the estimation phase.
svn merge https://svn.example.com/project/tags/publicrelease6.0/ https://svn.example.com/project/tags/publicrelease6.1/ . --accept postpone --dry-run
- As we manually merged the conflicted files, we took note of the merge details in a list. For each file that required non-trivial merging, we jotted down a couple sentences describing the merge (e.g. which methods were merged, etc.). Taking notes during the code merge requires a little overhead, but the list proved to be very useful when bugs caused by the merge were found later on.
In EP 6.0, named JPA database queries were embedded in the class files. For example, this bit of code is in CatalogImpl.java:
@NamedQueries({
@NamedQuery(name = "CATALOG_IN_USE_BY_PRODUCT_TYPE",
query = "SELECT pt.uidPk FROM ProductTypeImpl pt"
+ " LEFT JOIN pt.catalog c WHERE c.uidPk = ?1")
...
})
In EP 6.1, the named JPA database queries have been moved from the class files to separate object-relational mapping XML files. The query above has been moved to the catalog-orm.xml file.
The code merge should take care of the OOTB (out-of-the-box) named queries but any customized name queries that we added had to be manually moved from the class file to the appropriate XML file.
The code merge should take care of the OOTB (out-of-the-box) named queries but any customized name queries that we added had to be manually moved from the class file to the appropriate XML file.
4. Using the New Settings Framework
In EP 6.0, configuration settings were stored in various XML files (e.g. commerce-config.xml, default.xml, etc.). EP 6.1 introduces a new settings framework that stores settings in the database and allows them to be managed from the CM Client; the setting values are retrieved wherever they are needed in the code via the SettingsService class. The new settings framework is a big improvement for managing configuration settings, but some work needs to be done to take advantage of it.
First, some of the OOTB setting values need to be overridden with your project's values. For instance, our client's stores use Elastic Path's one-page checkout. There is a boolean setting that turns this feature on. By default, this setting is off. In EP 6.0, we turned this setting on by modifying a line in commerce-config.xml: onepage.enable=true. In EP 6.1, this setting value is no longer retrieved from commerce-config.xml (although the value may still be there depending on how it was handled during the code merge), but it is not being used. The setting value is now retrieved from the database via the new settings framework. We located the setting (COMMERCE/STORE/enableOnePageCheckout) and changed its value from false to true with a database update. After we were done modifying the OOTB setting values, we removed all unused setting definitions from the XML files.
Second, we had to migrate all of the custom settings we created to the new settings framework. We have approximately 60 custom settings -- much more than typical projects. Migrating custom settings is not absolutely required since they are still parsed from XML files and retrievable using the ElasticPath class, but there are many advantages of using the settings framework.
To migrate a custom setting, you first need to create the setting definition with a database insert statement. For instance, the following insert statement creates our "tradeshow timeout" setting:
INSERT INTO TSETTINGDEFINITION(UIDPK, PATH, DEFAULT_VALUE, VALUE_TYPE, DESCRIPTION, MAX_OVERRIDE_VALUES)
VALUES(81, "CUSTOM/tradeshowTimeout", "120000", "Integer", "Indicates the tradeshow timeout interval, in seconds.", -1);
Next, find all the places in the code that use this setting. Modify the code to use the SettingsService to retrieve the setting value from the database instead of from the XML files (via ElasticPath). Going back to the example above, we make the following javascript change:
timeoutId = window.setTimeout("resetTradeshowPage()", $ctxStoreConfig.getSetting("CUSTOM/STORE/tradeshowTimeout").getValue());
...and the following java code change:
final String settingValue = getSettingsService().getSettingValue("CUSTOM/STORE/tradeshowTimeout").getValue();
In EP 6.1, the assets directory was moved outside of the war file. For example, storefront templates were previously located in com.elasticpath.sf/WEB-INF/templates/; they are now located in assets/themes/storecode/default/templates. Note that the storefront templates were not the only ones that were relocated; CM templates, like those used for order confirmation emails, were also moved from the com.elasticpath.sf project to the new assets directory. The relocation decouples the view (i.e. templates, images, etc.) from the model and controller (the application). There are many advantages to this, including the ability to easily hot-swap template changes. However, the relocated assets directory made the code merge more complex.
Depending on your merging tool, the automated code merge may handle the relocated assets directory differently. We used svn merge. Unfortunately, svn merge was not smart enough to realize that the assets were relocated; it assumed that the assets directory was simply removed from its old location and that a new assets directory was added to its new location. As a result, the files were not merged at all. The new assets directory was the OOTB version.
Thus, we needed to manually merge all of our template changes. This process is not as bad as it sounds, since Elastic Path did not make any major changes to the templates from EP 6.0 to EP 6.1. We merged most of the templates by first diffing the EP 6.0 and EP 6.1 versions and then adding those differences to our customized versions.
Some of the bigger template changes made between EP 6.0 and EP 6.1 include:
Note: I mentioned above that template files can now be easily hot-swapped. One thing to keep in mind is that by default, templates are cached when the application starts up. If you hot-swap or modify a template file, you need to force a cache refresh by invalidating the cache. A quick and easy way to do this is to open the invalidate-cache.ep URL in a web browser.
Some of the bigger template changes made between EP 6.0 and EP 6.1 include:
- Retrieving the store and catalog not from the ElasticPath object anymore but from the new StoreConfig object
- Using the #emailMessage macro (instead of the #springMessage macro) in the email templates to display localized messages
Note: I mentioned above that template files can now be easily hot-swapped. One thing to keep in mind is that by default, templates are cached when the application starts up. If you hot-swap or modify a template file, you need to force a cache refresh by invalidating the cache. A quick and easy way to do this is to open the invalidate-cache.ep URL in a web browser.
EP 6.1 uses Maven to manage project dependencies. In the past, if a project required a component (e.g. a 3rd party library) that was not used OOTB, we would put that component in the libs directory. With Maven, there's no longer a need for a libs directory, so we were able to remove it altogether during our merge. There was one caveat however: we were using a 3rd party library for GeoIP lookups (Maxmind). We added the dependency to our GeoIP library by adding the following in the storefront's pom.xml file:
<dependency>
<groupId>maxmind</groupId>
<artifactId>geoip</artifactId>
<version>1.2.1</version>
<scope>compile</scope>
</dependency>
Elastic Path uses Archiva repositories to store the dependent libraries. Rather than adding the Maxmind files to the internal Elastic Path repositories, we decided to add it to our team's Archiva repository. Archiva has a nice web interface to upload jar files into the repository. We added the following to elasticpath.pom to tell Maven that it needed to look in our team's archiva repository (maven-ps.elasticpath.net) for some of its dependencies:
<repository>
<id>ps-extras</id>
<name>Elastic Path PS Extras Repository</name>
<url>http://maven-ps.elasticpath.net:8080/archiva/repository/extras/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
To sum up, setting up Maven should require no work if your project does not have any custom dependencies. If your project does have dependencies that are not included with OOTB Elastic Path, they will need to be defined in the project object model (POM) files.
Testing & Bug Fixing
Our QA team's first task was to verify that EP 6.1 worked on our test servers. We grabbed the OOTB public release version of EP 6.1, deployed it on our test machines, and ran through a smoke test. Once we were able to confirm that the OOTB EP 6.1 worked on our servers, we focused on our merged codebase.
The code merge modified numerous files across the entire project. As a result, a full regression test on all of our customizations was required to verify that the code merge was done correctly. It goes without saying that the full regression test took a considerable amount of time to complete. The faster the code is merged, compiled cleanly, and deployed to test servers, the sooner testing can begin. After merging the code, we pushed a testable build out as soon as we could so that our QA team could begin testing. The first half-dozen builds contained many blocking bugs, but they were still testable. The key is to fix the blocking bugs quickly so that the QA team can be kept busy with the full regression test.
The JUnit tests should also be run and fixed as soon as possible, preferably even before the first testable build is deployed; they are a great way to catch bugs at the source code level.
The new FIT tests in EP 6.1, however, were a little more difficult to run. They require some setup and many may be broken after the code merge depending on the customizations. We skipped the FIT tests during the code merge and instead chose to run them after the upgrade.
Time Estimates
There's no definitive answer for how long an upgrade will take. It depends on the amount and type of customizations. For our client's upgrade, the entire process -- including scoping, upgrading, testing, and bug fixing -- took three months. One developer completed the upgrade (i.e. database updates, code merge, etc.), two QA analysts performed the testing, and three developers tackled the bugs found by the testers. 8% of our budget was spend on scoping and estimating, 60% was spend on the actual upgrade tasks and bug fixing, and 32% was spent on testing.
These numbers will vary from project to project. For example, the client's custom settings required additional custom work, which took approximately two weeks to design and implement. This task is something we developed specifically for this project only.
Conclusion
Upgrading Elastic Path is neither trivial or impossible. Elastic Path provides documentation on the various aspects of upgrading to Elastic Path 6.1 (database updates, settings migration, assets relocation, etc.), but the complete story is different for each project. The amount of work depends on the number and type of customizations that have been made. If you have any questions or comments, feel free to post them here.
