(Or: How to Copy and Paste XML Your Way to Greatness)
We have a truckload of Selenium tests that poke and probe away at the user interface of our project here at EP. We realized it would be awesome if this testing could just automagically happen and let us know the results. So we took a look at our Maven build and realized the steps we wanted to automate were:
- Check out our source code, build the latest set of artifacts and deploy them to our internal Maven repository.
- Stop the web application running on our QA server environment.
- Recreate our test MySQL database with our default schema, apply any change sets needed to bring it to the latest version and load our QA data.
- Update the test data to reflect our QA server environment.
- Deploy our demo assets to a known location.
- Start our web application and wait for it to fully start.
- Launch Selenium and run our tests against our application on Firefox on our Linux Hudson slave.
- Record and announce the results of the tests.
In this blog post, I'm going to cover how we automated the last three steps.
The first thing we needed to do was find a tool that could automate configuring a web app container and deploying our WAR files. This tool would need to support multiple web containers without having to write custom scripts for each one. After doing some research, we decided to go with the fairly capable Cargo project. It has a fairly mature Maven plugin and good support for our preferred container, Tomcat, and it's come a long way since its inception. On our side, all we'd need to do is write a simple Groovy script to wait until our web application is in a ready-to-test state.
Our particular application really benefits from the new WebDriver features in Selenium 2.0, and while we'd love to take advantage of the Selenium Plugin for Maven, it's tied to Selenium 1.0, so we can't fully use it yet. We can however use it to bootstrap an X server on our Hudson build slave.
Our Selenium tests are written using TestNG and stored in their own Maven artifact. We can easily bind the Maven Failsafe Plugin to run during our integration-test phase, but we'd like to publish the output to our internal reporting site, so we can take a look at the last results. Luckily, we can use the Maven Surefire Report plugin to generate our site. Lastly, we'll get Hudson to announce the results of our test. (The page on the Maven build lifecycle is invaluable when figuring out when to run different plugins.)
So, let's get started!
Properties
To make Cargo and Selenium more useful, we pushed a handful of settings into a parent POM. This lets us override values and use the same artifacts for our own local development testing.
<properties>
<cargo.jvmargs>-XX:MaxPermSize=512M -Xmx1024m</cargo.jvmargs>
<cargo.servlet.port>8080</cargo.servlet.port>
<cargo.tomcat.ajp.port>8009</cargo.tomcat.ajp.port>
<cargo.rmi.port>1099</cargo.rmi.port>
<cargo.tomcat.shutdown.port>8205</cargo.tomcat.shutdown.port>
<cargo.wait>true</cargo.wait>
<demo.hostname>10.9.8.7</demo.hostname>
<searchserver.context>searchserver</searchserver.context>
<ourapp.context>ourapp</ourapp.context>
<cmserver.context>cmserver</cmserver.context>
<storefront.context>storefront</storefront.context>
<assets.directory>${user.home}/ep-assets</assets.directory>
<jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
<jdbc.host>10.9.8.7:3306</jdbc.host>
<jdbc.db>OURAPP_DB</jdbc.db>
<jdbc.username>atwill</jdbc.username>
<jdbc.password>grep</jdbc.password>
</properties>
Okay, that's more than a handful. but let's go over them really quickly. The cargo.* properties are there to let us easily have multiple Tomcats running on the same IP, the *.context properties tell Cargo where to deploy our various WAR files. The cargo.wait property tells Cargo to pause after it starts the application server. If set to false, Cargo will proceed along the Maven build lifecycle. The assets.directory is used by steps 4 and 5. The demo.hostname is the public hostname (or IP in our case) for our deployment (you'll see why we do this as we move along). The jdbc.* properties describe our JDBC DataSource. We've broken the information up so that we can use it in steps 3, 4 and 6. As you can imagine, it's pretty easy to override these properties in your local settings.xml and use Cargo to start a local app server for you - lucky you!
Containing the Excitement
Our first goal is to start a container, deploy our artifacts and wait for them to all start. For our situation, we felt it was best to have a separate container Maven artifact (with pom packaging) dedicated to steps 5 and 6. So, first we'll declare the WAR and JAR files that we depend on for the container. We included the JDBC driver and the JAI JARs for our Storefront too. We've already declared the versions for all these artifacts in a parent POM.
....
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.media</groupId>
<artifactId>jai-core</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.sun.media</groupId>
<artifactId>jai-codec</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
Next up, we'll start Cargo in the pre-integration-test phase and stop it when we hit the post-integration-test phase. If you're copying and pasting this XML, you'll want to paste the next three sections in sequence into your POM, but we'll cover them individually.
<build>
<plugins>
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<version>1.0.1-beta-2</version>
<executions>
<execution>
<id>start-container</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-container</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
<configuration>
<container>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>javax.media</groupId>
<artifactId>jai-core</artifactId>
</dependency>
<dependency>
<groupId>com.sun.media</groupId>
<artifactId>jai-codec</artifactId>
</dependency>
</dependencies>
<containerId>tomcat5x</containerId>
<zipUrlInstaller>
<url>file:${project.basedir}/apache-tomcat-5.5.29.zip</url>
<installDir>${installDir}</installDir>
</zipUrlInstaller>
</container>
In the previous example, we bind the plugin to the lifecycle and tell Cargo to include the JDBC driver and JAI JARs into the classpath. (We won't get native acceleration for JAI, but that's fine for our purposes.) You'll probably notice the tricky thing we did for the zipUrlInstaller. To remove a dependency from our startup, we've checked in a copy of Tomcat into our project. Let's take a look at the Cargo configuration:
<configuration>
<home>${project.build.directory}/tomcat5x/container</home>
<properties>
<cargo.jvmargs>${cargo.jvmargs}</cargo.jvmargs>
<cargo.logging>high</cargo.logging>
<cargo.servlet.port>${cargo.servlet.port}</cargo.servlet.port>
<cargo.tomcat.ajp.port>${cargo.tomcat.ajp.port}</cargo.tomcat.ajp.port>
<cargo.rmi.port>${cargo.rmi.port}</cargo.rmi.port>
<cargo.tomcat.shutdown.port>${cargo.tomcat.shutdown.port}</cargo.tomcat.shutdown.port>
<cargo.datasource.datasource>
cargo.datasource.url=jdbc:mysql://${jdbc.host}/${jdbc.db}?AutoReconnect=true&amp;useUnicode=true&amp;characterEncoding=utf-8|
cargo.datasource.driver=${jdbc.driver}|
cargo.datasource.username=${jdbc.username}|
cargo.datasource.password=${jdbc.password}|
cargo.datasource.type=javax.sql.DataSource|
cargo.datasource.jndi=jdbc/epjndi
</cargo.datasource.datasource>
</properties>
Here we're passing in the cargo.* properties from our parent POM and we're asking Cargo to create a DataSource for us with the jdbc.* properties and add it to the JNDI tree. For our situation, we're always using MySQL, so we can make a handful of assumptions in the URL. Now that we've picked a container and configured it, we'll tell Cargo which artifacts we'd like to have deployed:
<deployables>
<deployable>
<groupId>com.elasticpath.mdm</groupId>
<artifactId>ourapp-war</artifactId>
<type>war</type>
<properties>
<context>/${ourapp.context}</context>
</properties>
<pingURL>http://${demo.hostname}:${cargo.servlet.port}/${ourapp.context}
</pingURL>
<pingTimeout>60000</pingTimeout>
</deployable>
<deployable>
<groupId>com.elasticpath.mdm</groupId>
<artifactId>searchserver</artifactId>
<type>war</type>
<properties>
<context>/${searchserver.context}</context>
</properties>
<pingURL>http://${demo.hostname}:${cargo.servlet.port}/${searchserver.context}/product/select?q=*:*
</pingURL>
<pingTimeout>60000</pingTimeout>
</deployable>
<deployable>
<groupId>com.elasticpath.mdm</groupId>
<artifactId>cmserver</artifactId>
<type>war</type>
<properties>
<context>/${cmserver.context}</context>
</properties>
<pingURL>http://${demo.hostname}:${cargo.servlet.port}/${cmserver.context}/
</pingURL>
<pingTimeout>60000</pingTimeout>
</deployable>
<deployable>
<groupId>com.elasticpath.mdm</groupId>
<artifactId>storefront</artifactId>
<type>war</type>
<properties>
<context>/${storefront.context}</context>
</properties>
<pingURL>http://${demo.hostname}:${cargo.servlet.port}/${storefront.context}/
</pingURL>
<pingTimeout>60000</pingTimeout>
</deployable>
</deployables>
</configuration>
</configuration>
</plugin>
This is pretty much textbook Cargo; you'll see our use of demo.hostname above in PingURL. Cargo will fetch the artifacts and deploy them into Tomcat waiting for the specified URL to not return an error, waiting up to pingTimeout milliseconds. Since we're just a demo application, the PingURL for the searchserver does a query for products. The problem here is that, while the searchserver may be available, the search indexes might not be built yet. So, we'll need something to check that the searchserver is not only started, but that its indexes are populated.
For this, we put together a simple Groovy script that waits until the searchserver returns products when queried. We put this in src/main/script/waitforstartup.groovy and it looks a little something like this:
import java.net.URL
import groovy.xml.StreamingMarkupBuilder
def url = project.properties['searchserverUrl'];
def timeout = project.properties['timeout'].toLong();
/* For standalone testing, use something like:
*
* def url = "http://localhost:8080/searchserver/product/select?q=*:*"
* def timeout = 120*1000;
*/
def ready = false;
def text = "";
def first = true;
def begin = System.currentTimeMillis()
while (!ready) {
if (System.currentTimeMillis() > (begin+timeout)) {
// provided by gmaven
fail("Timeout of "+timeout+"ms reached waiting for "+url+" to return at least one search result.")
}
if (!first) {
Thread.sleep(10*1000);
} else {
first = false;
}
URL searchServer;
try {
searchServer = new URL(url);
text = searchServer.getText();
} catch (e) {
print "Server not yet ready, sleeping for 10 seconds. ("+e+")\n"
continue;
}
if (text.size() == 0) {
print "Not yet ready, sleeping for 10 seconds. (Empty HTTP response received)\n"
continue;
}
try {
def root = new XmlSlurper().parseText(text)
if (numFound(root)!=0) {
ready = true;
} else {
print "No elements returned in search result, sleeping for 10 seconds.\n"
continue;
}
} catch (e) {
print "Trouble parsing XML result, sleeping for 10 seconds. ("+e+")\n"
continue;
}
}
def numFound(root) {
return root.result.@numFound;
}
XmlSlurper was pretty neat (as used in numFound). Running it is pretty trivial with the GMaven plugin. You'll notice how we can easily pass in properties to the Groovy script via project.properties.
<plugin>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-plugin</artifactId>
<configuration>
<properties>
<timeout>300000</timeout>
<searchserverUrl>http://${demo.hostname}:${cargo.servlet.port}/${searchserver.context}/product/select?q=*:*</searchserverUrl>
</properties>
<source>${basedir}/src/main/script/waitforstartup.groovy</source>
</configuration>
</plugin>
The source element is relative to the current working directory, not your project's base directory, so it's a good idea to put in ${basedir} to prevent surprises later on.
Running mvn integration-test should scroll for a while as Maven downloads all required dependencies and eventually the container will start. If we run gmaven:execute, our Groovy script will run and happily poll our searchserver until it's ready!
This alone is a great set up for running builds of our project both on my development box and for our QA server!
In Hudson, I have one Job which spawns both targets in parallel. The integration-test job never returns (until it's killed in step 2), but once the gmaven:execute target returns, it runs the Selenium tests. Let's talk about those now...
Prepare for Take Off
Our second goal is to launch Selenium, run our tests and shut Selenium down. It made sense for us to have a separate selenium-tests artifact for that purpose, let's start by taking a look at the properties we decided to set:
<properties>
<selenium.version>2.0a4</selenium.version>
<selenium.host>127.0.0.1</selenium.host>
<selenium.port>14444</selenium.port>
<selenium.DISPLAY>:17</selenium.DISPLAY>
<selenium.background>true</selenium.background>
<xvfb.option>-ac</xvfb.option>
<seleniumUrl>http://${selenium.host}:${selenium.port}/wd/hub</seleniumUrl>
<appUrl>http://${demo.hostname}:${cargo.servlet.port}/${ourapp.context}</appUrl>
</properties>
We're making use of the Maven Selenium Plugin to start up Xvfb on display :17, and we disable access control to Xvfb to work around any xauth issues we may run into. Here you see another use of demo.hostname, passing it to Selenium as the application under test. The selenium.background property allows us to run Selenium on the command line and keep the server running with -Dselenium.background=false, this running these tests during development makes testing a lot quicker!
Next we'll specify the dependencies we need to make this all happen:
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium</artifactId>
<version>${selenium.version}</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>5.11</version>
<scope>test</scope>
<classifier>jdk15</classifier>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<version>${selenium.version}</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>selenium-repository</id>
<url>http://selenium.googlecode.com/svn/repository/</url>
</repository>
</repositories>
No real surprises here, although I'm adding an extra repository just to pull down Selenium 2.0a4. You could easily mirror it locally in your own company-wide repository.
Now we need to start and stop Selenium. Normally we'd use the Maven Selenium Plugin, but it's pinned to the older generation of Selenium. Instead of trying to be overly clever, we just ask Ant to start and stop Selenium during the pre-integration-test and post-integration-test phases, we pass in a DISPLAY variable which is ignored on non-Unix operating systems.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>start-selenium</id>
<phase>pre-integration-test</phase>
<configuration>
<tasks>
<echo taskname="start-selenium"
message="Starting Selenium Server v${selenium.version} on ${selenium.port} offering display ${selenium.DISPLAY}" />
<java taskname="start-selenium"
jar="lib/selenium-server-standalone-${selenium.version}.jar"
fork="true" spawn="${selenium.background}">
<env key="DISPLAY" value="${selenium.DISPLAY}" />
<arg line="-timeout 30 -debug -browserSideLog -port ${selenium.port}" />
</java>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>stop-selenium</id>
<phase>post-integration-test</phase>
<configuration>
<tasks>
<echo message="Stopping Selenium Server" />
<get taskname="stop-selenium"
src="http://${selenium.host}:${selenium.port}/selenium-server/driver/?cmd=shutDown"
dest="${project.build.directory}/selenium-shutdown.txt"
ignoreerrors="true" />
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
Okay, so we've configured Selenium to start and stop, now lets ask it to run our tests. We created an extremely simple TestNG suite in src/test/it/testng.xml:
<suite name="integration-tests">
<test name="Everybody Everybody">
<packages>
<package name="com.elasticpath.mdm.tests.it" />
</packages>
</test>
</suite>
By convention, the Maven Failsafe Plugin will run src/test/java/<above packages>/*IT.java tests when called with integration-test and TestNG will look for methods annotated with @Test. So back in the POM, we say:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.5</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/it/testng.xml</suiteXmlFile>
</suiteXmlFiles>
<systemPropertyVariables>
<seleniumUrl>${seleniumUrl}</seleniumUrl>
<appUrl>${appUrl}</appUrl>
</systemPropertyVariables>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
We bind it to the integration-test phase so it's run after Selenium starts. You'll notice that we pass seleniumUrl and appUrl into the TestNG tests. We pass those as parameters into a @BeforeSuite method; we'll touch on that in a moment.
Last bit, we're going to launch Xvfb, but only if we're on a UNIXy operating system, so we'll wrap it in a profile stanza:
<profiles>
<profile>
<id>xvfb-started</id>
<activation>
<os>
<family>unix</family>
</os>
</activation>
<build>
<plugins>
<plugin>
<phase>package</phase>
<goals>
<goal>xvfb</goal>
</goals>
<configuration>
<display>${selenium.DISPLAY}</display>
<options>
<option>${xvfb.option}</option>
</options>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Oh the hack. Ironically, Maven does not have a way to specify that a particular execution id depends on another execution id, so we had to bind launching Xvfb to the package phase. Maybe in Maven 3.0?
Getting into the actual code, earlier I mentioned the @BeforeSuite attribute, in that method we'll configure Selenium for our purposes:
.....
/*
* When running TestNG from within Eclipse, the plugin passes this value in to parameters it cannot substitute.
*/
private static final String PARAMETER_NOT_FOUND = "not-found";
.....
/*
* Passing in these properties will allow you to use the specified values if the parameters are not found by TestNG.
*/
public static final String SELENIUM_URL_PROPERTY = "selenium.url";
public static final String APPLICATION_URL_PROPERTY = "app.url";
private static WebDriver driver;
private static WebDriverBackedSelenium selenium;
private static String sessionString;
private static String appUrl;
.....
@BeforeSuite
@Parameters( { "seleniumUrl", "appUrl" })
public void startBrowser(String seleniumUrl, String appUrl) throws Exception {
if (PARAMETER_NOT_FOUND.equals(seleniumUrl)) {
seleniumUrl = System.getProperty(SELENIUM_URL_PROPERTY);
}
if (PARAMETER_NOT_FOUND.equals(appUrl)) {
appUrl = System.getProperty(APPLICATION_URL_PROPERTY);
}
driver = new RemoteWebDriver(new URL(seleniumUrl), DesiredCapabilities.firefox());
// This will cause all find-element operations to keep trying for up to 10 seconds automatically.
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
selenium = new WebDriverBackedSelenium(driver, appUrl);
/*
* Generate a unique string so if we update a field to a new value, we know it's going to be a new value (and can compare that it's so).
*/
sessionString = System.currentTimeMillis() + "";
selenium.open(appUrl);
}
....
@AfterSuite
public void stopBrowser() {
driver.close();
}
.....
This method is called before our test suite runs. We do some extra work to allow developers to run TestNG with -Dselenium.url and -Dapp.url if running using the TestNG plugin for Eclipse to pass in parameters. We also configure WebDriver to automatically wait 10 seconds for events to occur. Don't forget to tell people writing tests that they don't need to do this again in their code!
This configuration gives us a handful of options, developers can boot up a Selenium server by issuing "mvn -Dselenium.background=false integration-test", then run tests either in Eclipse or by calling surefire directly with "mvn surefire:integration-test".
Reporting the News
To recap, we can start a container, deploy our application, wait for it to start, then launch Selenium, run some tests and shut Selenium down when our tests have finished. Our final goal is to report an announce the results of our test. Generating reports is pretty easy, we'll just add the following to our POM:
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>2.5</version>
<reportSets>
<reportSet>
<id>integration-tests</id>
<reports>
<report>report-only</report>
</reports>
<configuration>
<outputName>failsafe-report</outputName>
<reportsDirectories>
<reportsDirectory>${project.build.directory}/failsafe-reports</reportsDirectory>
</reportsDirectories>
</configuration>
</reportSet>
</reportSets>
</plugin>
</plugins>
</reporting>
Wow, that was easy. In Hudson, we configure the selenium-tests project to run the following Maven Goals: integration-test site-deploy failsafe:verify. This asks Maven to perform the tests, deploy the site and then fail if the tests failed.
I've also configured Hudson to send an email out if this build fails, that email contains the URL directly to the Selenium Failsafe report!
Now we have an automated deployment mechanism, Selenium tests and reporting.
Awesome!
