Agile development is well suited to software projects for many reasons. It empowers developers to make decisions proactively and keep projects moving forward. On the down side, Agile projects are not known for their formal reporting practices, which can be disconcerting for customers more used to a traditional project management approach.

My traditional PMI training taught me how to use Earned Value Management. EVM is a set of formulas that give insight into the progress of a project. These formulas allow you to put a dollar value on each dollar spent; a value of less than 1 is bad, while a value greater than 1 is good. These techniques require a thorough upfront understanding of the project. While this can work well on waterfall projects, it is not going to be accurate on iterative projects, where a degree of uncertainty is the norm.

No matter how agile you are, you still need to know at any given time whether the project as a whole is on track. The most common tool for reporting progress on an Agile project is the burndown chart. The burndown chart shows if the project is on track for schedule, by showing how much scope has been completed at a given time.  It does not show if you're on track for budget or give a picture into quality. To fill those gaps, there are two other tools I like to use in addition to the burndown chart:

• The budget burn-up, which tells us how much have we spent against what we planned to spend, plotted over time.
• The bug count, which tells us how healthy the project is.

By using these three tools together, we get pretty much all the information that we'd get by using Earned Value. If we're on track for budget and scope, that's great, but if the bug count is high, the project is struggling and quality is suffering as a result. But just how accurate is the information we get from these tools?

Budget

Budget is straightforward. If billing at time and materials on a blended rate, the time spent working on the project multiplied by the rate is the cost. It is tracked and invoiced through the same tool.  You can plot this on a graph against the planned expendititure (like the baseline on a Waterfall project).

Budget Burnup Chart

The two dotted lines represent the original planned expenditure, and the budget amended with a scope change request.  If actual dollars spent is below the baseline or planned expenditure, the project is under budget.  When viewed independently, this chart tells us that we have spent more than planned, which in itself is not a good thing.  However, the project is also ahead of schedule so more work has been completed than was expected, this is a good thing.  This demonstrates how none of these visual aids can stand alone to give an accurate insight into project status.

Bug Count

Bug count is only useful if there is a rigorous QA process. Zero QA also means zero known bugs. In theory, an iteration should produce "production ready" code (we strive for "done, done, done"). In practice, there is usually a bug backlog after the iteration is complete. This can be shown as an exported table from the issue tracking system. (We use Jira, which allows you to select the most useful columns when exporting table data.)

Burndown Chart

I've used three types of burndown chart:

• Effort remaining (in hours) to complete work in a sprint. This produces the most subjective results, but arguably the most useful for a weekly status report.

• Stories completed during a sprint. Day to day, not much use, because a task could be 2 weeks long and 80% complete, and you don't see this on the chart.  In theory, if you break down stories into similar sized units of work and use a binary approach to tracking (done or not done), a "task complete" burndown chart would give you what you need. In reality, many stories don't break down into equal sized work packages.

• Stories completed during the entire project (iteration by iteration). This is a good way to look at velocity, but not useful for changing course mid-iteration when risks occur.

For status reporting week by week, the first chart is arguably the most useful. It is also the most challenging data to obtain because it relies on re-estimating work. Effort remaining is subjective; accuracy can vary depending on many factors.  But if you're in the middle of an 80 hour task, the only way to know if you're on track to deliver by the end of the iteration is by re-estimating the remaining work periodically. This could be in hours remaining, or percent complete, whatever works for the team.  It can be reported during a scrum or via a daily update to your task tracking software (we use Jira), or other tool such as an iteration tracking spreadsheet.

Feature Burndown by Iteration

A feature burndown, showing tasks completed per iteration tracked against the original plan. A binary burndown (done/not done) is a more accurate way of showing progress than the effort remaining burndown chart. It doesn't give visibility into the progress of a project mid-iteration, and therefore is not a good tool for monitoring progress on a weekly basis.  This chart shows the planned burndown against the actual.  At the end of Iteration 2, which is the current state, the actual is below the planned features completed on the burndown, so we are ahead of schedule with more scope completed than expected.

Effort Remaining Burndown Chart

An effort remaining burndown shows whether you're on track for schedule during an iteration.  This chart has hours remaining to complete the work, plotted against days.  This chart was generated using Excel, but Greenhopper is a Jira Plugin that will create this chart for you using data captured within each task, and assigned to an iteration, if this data is added and kept up to date at the task level.

The day to day reporting of time spent and effort remaining on a task allows the project manager to keep stakeholders informed of progress, and flag risks as they arise. The more accurate we are in reporting status (among the team and to our customers), the more the project's stakeholders will trust us. No project runs end to end without some bumps along the way, but if we can see them coming, we can respond proactively and ensure a smoother ride.

Selenium is a suite of free, community-developed tools for automated testing of web applications. It provides test recording and playback, and supports test playback in both Firefox and Internet Explorer. For our testing purposes, we use two tools in the suite: Selenium IDE and Selenium Core.

Selenium IDE is a Firefox extension that lets you record, edit, and debug Selenium tests. To start it, launch Firefox and choose Selenium IDE in the Tools menu. In the IDE window, you can create a test suite and record all the test cases you want to make, then save them to your hard drive in the file format of your choice. HTML, Java, C#, Perl, PHP, Python and Ruby are all supported formats, but we save the test cases as HTML so we can re-use them in our IE test suite (more on this later).

Selenium Core can do a lot of things, but we're only interested in the TestRunner tool, which we use to run our HTML test cases in Internet Explorer. To run the test cases, you need to create a test suite file in an HTML editor and add links to the HTML test case files created in IDE. To see how to do this, you can look at the example test suite files included in Selenium Core.

After that's done, run TestRunner by double-clicking TestRunner.HTA in the Core Dir\Core folder where you unzipped the Selenium Core package. Then enter the path and name of the test suite you want to run in the Test Suite box and click Go.

Real-World Selenium

For one large customer project, we run test cases against four different online stores in three difference locales. For each store, we've grouped the test cases into a test suite file. This test suite file is the master test case list and can be loaded into either Selenium IDE or TestRunner,depending which browser needs to be tested.

Installing Selenium IDE

1. In Firefox, go to http://seleniumhq.org/.

2. Click the Download tab.

3. Download the Selenium IDE.

4. Click Install Now. This will automatically install Selenium IDE in Firefox. You will need to restart Firefox to use IDE.

Installing Selenium Core

1. Go to http://seleniumhq.org/.

2. Click the Download tab.

3. Download the Selenium Core package to your local system.

4. Unzip the Core package. TestRunner application is located in the folder where you unzipped Core, under Core Dir\Core.

Elastic Path supports web services via the Simple Object Access Protocol (SOAP), and provides support for obtaining and updating information for orders. The problem is, how do we actually test the web services component? One could write a custom SOAP client that interacts with the the SOAP server. However, there is an excellent tool available currently (and free) that can simulate sending a SOAP request, getting a response from the server, and then examining the data.

*Drum roll*

Introducing SoapUI. SoapUI is a Java-based program that aims to automate web service calls and can integrate into your testing framework. In this article we will go through the steps and create a test case that can be run automatically after the initial setup. We will introduce scripting in Groovy, a derivative of Java language.

The goal of this exercise to get used to the SoapUI framework and making a test suite consisting of updating a SKU, then checking to make sure that SKU has been updated. It can be run from the command line, and can generate reports/results, which could potentially be used in build reports.

To get started, you will need to have the following things ready:

1. SoapUI software. To get SoapUI, please visit http://www.soapui.org and get the latest version (at the time of writing, 2.5 is the most current version and will be used in this article)

2. The address of the Web Service Description Language file (WSDL). In this example the address is https://<your_domain>/webservices/catalogwebservice?wsdl where <your_domain> refers to the domain of your deployment

3. Permission priliveges to access web services. To create permissions you have to select the Web Service Role when creating a new CM user in the Commerce Manager client.

4. Have available SKU code information for use by the web services.

Step 1 - Creating a project

1. Open SoapUI, and then click on File->New soapUI Project.

2. In the Project Name text box, put in example1

3. In the Initial WSDL/WADL, type in the URL of the web service definition file (bullet number 2 in the previous section)

4. Press OK

    

   

A list of operations should appear on the left hand column. For the purposes of this demonstration, we will only test two operations, getSku and updateSku. Next, we will need to create a new TestSuite. Right click on the project name and then click on New TestSuite. Name it TestSuite 1 for the purposes of this test.

Now create a new TestCase by right clicking on TestSuite1 and click on New TestCase. Name it TestCase 1 for the purposes of this test.

Step 2 - Configuring the test case

The next step is to configure the test case so we can run the test. First we need to set the global settings. Click on the key icon and set the credentials for the username and password.

This username/password combination should match the user that has Web Services permission set in the Commerce Manager. Leave the Domain blank in this case.

The next step involves setting a property that can be accessed by the whole test case. Click on the Properties button.

Now add a property name called 'start' by click on the left-most icon. For the purposes of this test, we will name it 'date'. Give it a value of '2050-01-01T12:12:12.643-08:00'.

Step 3 - Adding the first SOAP request

We need to add a few SOAP requests first. Add the first SOAP request by clicking the little SOAP icon.

Name that request 'getSku', press Ok and click on CatalogWebServicePort->getSku from the dropdown box in the next dialog. Finish by clicking 'OK'. Now request window will open. Now on the left pane, the XML generated using the schema from the WSDL will be present. For this test case, use a SKU code that exists in the system and enter it in the <SkuCode> element. I chose '1YESPE' for this particular test. If you know the catalog that the SKU belongs to, put it between the catalog tag. Next we have add an assertion that the response is successful. We do that by clicking on the Assertions button and then clicking on the '+' button, then select 'Xquery Match' from the dropdown box. Fill in the details based on the values from below.

Press Save. By doing this, every time the test step is run, a validation check will occur. It will pass the check only if we see that the status of the test is succesful.

Queries using XPath are invaluable because they test can expected node values coming from the response. Since EP's Web Services returns a result status, it is ideal to evaluate the result using XPath queries. For more information on how to use XPath queries, please goto http://msdn.microsoft.com/en-us/library/ms256086.aspx

Step 4 - Adding the first Groovy script

SoapUI uses Groovy, an agile dynamic language for the Java platform. I won't go into details about Groovy, but basically if you know Java, then you know Groovy. To learn more about Groovy, please visit http://groovy.codehaus.org/.

For the first step of this part, you will need to add a test step to the test case. Right click on the test step on the left hand column and then click on Add Step -> Groovy Script.

A new window will appear. Fill in the following information in the box.

//gets the xml utilities

def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )

//get the reponse part

def holder = groovyUtils.getXmlHolder( "getSku#Response" )

//get the original start date

def startDate = holder.getNodeValue( "//StartDate" )

//assign it to a session variable to be used later

testRunner.testCase.setPropertyValue('temp', startDate);

//now update the Request of the second request

def holder2 = groovyUtils.getXmlHolder( "updateSku#Request" )

//gets the start date node and set it to our date

def startDate2 = holder2.setNodeValue( "//StartDate", testRunner.testCase.getPropertyValue('date'))

//update the change

holder2.updateProperty()

What does this do? Basically this will retrieve the startDate timestamp from the previous soap request and assign it to a variable, which will be used later at the end. This script also sets the variable of the next request so that it can be sent to the server. To read more on how to script your tests with Groovy in SoapUI, please visit http://www.soapui.org/userguide/functional/groovystep.html.

Step 5 - Configuring the rest of the test

Next we configure the rest of the steps. The rest of the test consists of:

Adding a SOAP request to update the SKU, and then checking the results.

1. For this request, we once again use assertions to help us identify potential problems. Create a new Xquery Match assertion and copy the data below into the boxes and press Ok.

Adding a Groovy script to retrieve the time stamp that was saved previously, and set it in the subsequent request.

1. This step is similar to the one before, and it will update the next SOAP request with original value that we retrieved (and saved) in the first request. The code is listed below.

    //gets the xml utilities

    def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context )

    //now update the Request of the second request

    def holder = groovyUtils.getXmlHolder( "updateSku2#Request" )

    //gets the start date node and set it to our date

    def startDate = holder.setNodeValue( "//StartDate", testRunner.testCase.getPropertyValue('temp'))

    //update the changes

    holder.updateProperty()

Adding a SOAP request to update the SKU again, this time we replace the start date with the original date we retrieved from the first request (so that we don't have actually change any data in this test).

1. This step is identical to the previous updateSku request. We can actually make a clone of the last request and run that. The only difference is the value of StartDate, which actually will be changed by the Groovy script from above. To make a clone of the previous step, right click on the last SOAP request, and then click 'Clone TestStep'. Name the new step updateSku2.

Running the test case and making sure they pass

1. Run the test case by pressing the arrow button in the Test Case 1 window.
2. If you have done everything correctly, all the tests should be green when you run the test. Your screen should look similar to the following.

SoapUI is a powerful tool that is designed to test web services. It can be used as a functional testing tool, a regression testing tool and an unit testing tool. Furthermore, it has built in capabilities to run using command line and can generate reports after each run. This tutorial is just a fraction of the things that SoapUI can provide a QA team when it comes to testing web services.