Technical Blog

3 Posts tagged with the index tag

Search Server Failover

Posted by Alan Schroder Jan 21, 2009

Every Elastic Path administrator knows: if your search server goes down, your storefront is down and so is Commerce Manager. You can make your Elastic Path deployment more resilient by setting up search server failover.

 

In a nutshell, you set up two machines to run the search server web application. One is the main search server (master). The other is a backup (slave). Both machines are running behind a load balancer. The master search server uses rsync to synchronize multiple index files to the slave machine and then commits these new indexes into the slave's indexes. This essentially creates a duplicate of the master on the slave and, if the master server goes down, the slave server will be able to handle all requests.

 

The failover configuration consists of three components:

  • The master server
  • The slave server
  • The load balancer.

The load balancer is a PC running Apache web server with the mod_proxy module. The master server contains the search server web application, the index builders, and the indexes. The slave server also contains the search server web application, but it does not build its own indexes. Instead, the master uses rsync to replicate the search index files to the slave. Then, the data in the updated index files is committed into the slave search server's indexes. During normal operation, this machine receives search requests and forwards them to the master.

SSFailover_Figure1.png

If the master becomes unavailable, due to a failure or planned downtime, the load balancer redirects search requests to the slave.

SSFailover_Figure2.png

To make it all work, there are some scripts that need to get run by cron jobs on the master and slave. Note these scripts use rsync and Unix hard-links, so they only work on Linux/Unix environments.

 

You also need to set the searchHost setting in Elastic Path to point to the proxy server.

 

You can download the scripts and the setup documentation from the downloads page.

0 Comments 0 References Permalink

Many classes were refactored in Elastic Path 6.1 to improve testability of existing code and to accommodate new features (most notably the ability for a single Storefront WAR file to host multiple Stores). Here are the changes with the highest impact for developers (in no particular order).

1. Refactoring of ElasticPathImpl.java and Creation of the Settings Framework

ElasticPathImpl covers several distinct areas of responsibility: BeanFactory, Configuration, Filtered Navigation, asset lookup... In 6.1, new interfaces were created to move those areas of functionality out of ElasticPathImpl with the intention of rendering the object obsolete in future versions.

  • Most configuration settings are now accessed via the SettingsService, which will be covered in a separate article. Commerce-config.xml files are no longer used to populate a map of settings in ElasticPathImpl, with the exception of the one in the CM web application, which is only used to keep track of the EP version and build number
  • Asset locations are now retrieved from the AssetRepository interface (see heading elsewhere in this article)
  • Search Configuration is now retrieved entirely from the Settings framework rather than from the ElasticPath object
  • Filtered Navigation information is still retrieved from ElasticPathImpl, but only because it implements the new FilteredNavigationConfiguration interface. The actual retrieval implementation now lives in a separate class and references the new Settings framework
  • Bean instances can still be retrieved from ElasticPath because ElasticPathImpl implements the BeanFactory interface, but from this point onward the ElasticPath object is not required to be injected into classes that only need a BeanFactory. Instead, you should inject an implementation of BeanFactory

If any customizations were performed on some Elastic Path 6.0 code base that must be transferred to an Elastic Path 6.1 code base, most careful attention should be paid to any changes in the ElasticPath singleton, especially if any new settings were introduced that are designed to be read in from a commerce-config.xml configuration file. Since most settings were moved to the Settings framework, the code to parse the configuration file has mostly been removed.

New customizations should pay special attention to the Settings framework and use it where appropriate. Also, the BeanFactory should be injected into new classes where required instead of the ElasticPath object, as its use as a bean factory is being phased out.

New persistent classes should have a DAO that is separate from any new services, to allow for more flexible persistence implementations if functionality changes.

2. Testing Framework Enhancements

In testing, the big news is deprecation of ElasticPathTestCase.java and the introduction of JUnit4 and JMock2 testing frameworks.

Prior to 6.1, many test classes extended ElasticPathTestCase.java, which is a JUnit 3.8 extended test class that mocks out much of the application's domain using mock objects created with the JMock 1.x framework. This class is being phased out, and use of it is no longer recommended because it:

  • is large and unwieldy
  • encourages poor code factoring and integration tests using mock objects
  • is complex; modification may affect many test cases in an unforeseen manner

 

Instead, unit test classes should focus on unit-testing rather than integration-testing methods. Use of the JUnit4 and JMock2 framework features should help in that regard. There are many examples of their use in the new code; but for a start you may want to look at GiftCertificateThemeImplTest (for a relatively simple test class) and StoreResourceManagerImplTest (which takes advantage of some advanced features in the new JMock framework).

3. New Store UI Themes and Asset Retrieval

Store-specific Velocity template resolution has been introduced. Most static and storefront theme content (storefront velocity templates, email velocity templates, javascript files, css files, Power Reviews, etc) was moved outside the separate application WAR files into the pre-existing Assets folder on the file system.

 

In 6.0 only some assets (store images and digital goods) lived on the file system under the assets folder, while others (email templates, storefront UI files, and PowerReviews files) lived inside the WAR. All of these files were moved to appropriate locations under the file system's assets folder and the paths to such folders were made available through an AssetRepository object rather than through methods on the ElasticPath object. This allows the assets to be modified at runtime.

 

The Storefront application's Themes feature (see Themes documentation) required the introduction of a StoreResourceManager to handle Velocity's lookups and caching for Storefront velocity templates. It does not use the AssetRepository, but instead uses very similar code combined with hardcoding the velocity template directory within the assets folders (see StoreVelocityConfigHelper.java). A separate Velocity Engine is created for each and every Storefront operating within a WAR.

The Storefront's asset requests that are not for Velocity templates (e.g. css, javascript, images, Power Reviews) go through the AssetResourceController, which uses the AssetRepository indirectly through an AssetRetrievalStrategy. In order for controllers to know which store they are working with, a ThreadLocal reference to Store has been introduced, set by the StoreSelectionFilter. Any storefront controller requiring parameters specific to the store that is accessing the controller can inject the StoreConfig object, or they can retrieve it from the RequestHelper, which also has it injected. The current store is no longer retrieved from ElasticPath.java.

4. New DAO classes

New DAO classes were introduced in some areas to separate service logic from persistence operations. In addition to some new features using separate Service and DAO classes, existing Services which have had DAO functionality factored out include:


  • ProductService / ProductDao
  • ProductTypeService / ProductTypeDao

 

DAO classes should generally not be accessed directly from outside of the Service layer. Such separation of business logic and persistence logic allows for far more flexibility and decoupling in the application. Future development should endeavor to follow this pattern (See the GiftCertificateThemeService / GiftCertificateThemeStoreAssignmentDao for an example of how such decoupling can be of benefit).

5. Index UID / GUID fields

We are phasing out UID fields in the indexes, in favor of GUID fields, for greater decoupling. Secondary object UID fields have, for the most part, been replaced with GUID fields in object indexes so that searches can be performed based on GUIDs. For instance, the CategoryIndex contains the CategoryUid so that the UID can be retrieved from the index when a search match is found, but the index is built using the GUIDs from associated objects (e.g. you can search the CategoryIndex to return categories that are associated with a particular CatalogCode, but not with a particular CatalogUid). The corresponding SearchCriteria objects have been modified to reflect the index schema changes.

Index fields that have been modified as a result of this change are as follows:


  • CategoryIndex:

 

  • parentCategoryUids replaced with parentCategoryCodes
  • catalogUid replaced with catalogCode

 

  • PromotionIndex

 

  • promotionRuleSetUid continues to live in the index, but the promotionRuleSetName is now also in the index. This is for legacy reasons (the PromotionSearchCriteria object still contains both fields as well), and the UID field is expected to be removed in a future version of Elastic Path.
  • catalogUid continues to live in the index, but catalogCode is now also in the index. This is for legacy reasons (the PromotionSearchCriteria object still contains both fields as well), and the UID field is expected to be removed in a future version of Elastic Path.

 

  • ProductIndex

 

  • CategoryUid replace with CategoryCode
  • parentCategoryUids replaced with parentCategoryCodes
  • catalogUid replaced with catalogCode
  • Addition of a new storeCode field

 

In addition, several fields were added to the OrderReturn index schema.

6. Improved Index Build Status Tracking

The status of indexes, and the method of causing them to rebuild, has changed drastically:


  • The index build status (including the last time the index was built) is stored in the new TINDEXBUILDSTATUS database table, and can be accessed using the IndexBuildStatusService.
  • On the Search Server, the AbstractIndexServiceImpl class manipulates the status of index builds directly through the IndexBuildStatusDao.
  • The Commerce Manager Client has a new View that enables index rebuilds to be triggered manually.

 

In addition, if the server is brought down (either manually or catastrophically) while in the process of building an index, it will now remember its status upon restart and will start a new rebuild rather than assuming that the index is complete.

7. New Store Editor GUI

The Store Editor has been enhanced to better handle store themes. In addition, it has abstracted most of the business logic from where it may previously have been embedded in the UI code.

0 Comments 0 References Permalink

       

The search index rebuild process has undergone some substantial changes in Elastic Path 6.1, making it more robust and more usable.

 

In the past, to trigger a search index rebuild, you needed to manually modify the contents of some properties files on the search server. This is no longer supported. Now, Commerce Manager users with the proper permissions can initiate search index rebuilds.

 

In the Activity menu, choose Configuration. In the Configuration page on the left, under System Administration, click Search Indexes.

 

Select an index in the list, and click Rebuild Index. The rebuild should now be scheduled and the status column will show Rebuild is Scheduled.

When the index rebuild begins, the status column for the index being rebuilt will contain Rebuild in Progress. Note that you can rebuild multiple indexes at the same time. When the rebuild is done, the status is changed to Complete.


When the user runs an index rebuild from the Commerce Manager client, this is what happens behind the scenes:

 

  1. The TINDEXNOTIFY table in the database contains a row for each search index. When the user clicks Rebuild Index, the UPDATE_TYPE column for that search index's row is set to REBUILD.
  2. The quartz.xml in com.elasticpath.search/WEB-INF/conf/spring/scheduling defines several Quartz jobs that check to see if the search indexes need to be rebuilt. Each time one of these scheduled jobs is triggered, it calls the method buildIndexJobRunner method in the AbstractIndexService and passes it a search index identifier. This method checks the TINDEXNOTIFY database table to see if the corresponding index needs to be rebuilt (if the value is REBUILD), has never been built, or if a previous rebuild was interrupted and did not complete successfully. If any of those conditions are true, a rebuild of that index is initiated.
  3. When an index rebuild is initiated, the INDEX_STATUS column in the TINDEXBUILDSTATUS table is set to REBUILD_IN_PROGRESS.
  4. After the index has finished rebuilding, the INDEX_STATUS column is set to COMPLETE.

 

You can monitor the status of rebuild process by checking the Status column in the search index list.

 

Finally, search index rebuilds are now more resilient to failures. If an index build is interrupted or fails to complete normally, the system will try to build it again on server startup or at the start of the next quartz job. If there is a rebuild in progress for an index and another rebuild for that index is requested by the user, the system will wait until the first rebuild is complete before processing the second request.

0 Comments 0 References Permalink