Web services performance testing: a pilgrim’s progress. Part 4.

I have SoapUI Pro from SmartBear at my disposal at work, and I’m quite comfortable with it. So it was my tool of choice for creating load and recording details about the results.

SoapUI’s first cousin LoadUI, a free version of which comes with the SoapUI Pro install, was also an option. However, I chose not to explore it for this testing mission. (A road not travelled, admittedly.)

My first load tests sent requests to the middleware Web services described in Part 3 of this series. Because of our NTLM proxy at work I had to use Fiddler to do the handshake between my desktop and the Web services.

Fiddler records a lot of interesting information about Web service requests, including response times and bytesize. So I copied that data from Fiddler, pasted it into Excel, and created time-series graphs from that data. I was able to create some pretty graphs, but copying and pasting the data over and over got to be a real time sink. I am not and will never be a VBA whiz, so I knew I had to find a better way.

I was forced into finding that better way when it came time to test the performance of the .NET services that USED the middleware Web services. Because of the .NET Web server’s configuration, I could no longer route requests through Fiddler and see the data. What seemed to be a hindrance turned out to be a blessing in disguise.

The solution I arrived at was to use SoapUI to record several aspects of the request and response transactions to a flat file. I could then bring that flat file into R for graphing and analysis.

The SoapUI test case for the .NET services is set up as follows. Apologies for the blurriness of some images below: I haven’t done HTML-based documentation in quite some time.

  1. Initial request to get the service “warmed up.” I do not count those results.
  2. Multipurpose Groovy script step.
  3. // I am registering the jTDS JDBC driver for use later in the test case. See below for info on using third-party libraries.

    import groovy.sql.Sql
    com.eviware.soapui.support.GroovyUtils.registerJdbcDriver( "net.sourceforge.jtds.jdbc.Driver" )

    //Get the time now via Joda to put a time stamp on the data sink filename.

    import org.joda.time.DateTime
    import org.joda.time.LocalDate

    def activeEnvironment = context.expand( '${#Project#activeEnvironment}' )

    def now = new DateTime().toString().replaceAll("[\\W]", "_")

    // Construct the file name for response data and set the filename value of a Data Sink set further along in the test.

    testRunner.testCase.getTestStepByName("Response Size and Time Log").getDataSink().setFileName('''Q:/basedirectory/''' + activeEnvironment + '''_''' + now + '''_responseTimes.csv''')

    If your Groovy script uses third-party libraries like jTDS and Joda, you have to put the jar files into $soapui_home/bin/etc.

    Putting jars where SoapUI can see them.

    Note that jTDS has an accompanying DLL for SQL Server Windows-based authentication. DLLs like this go in $soapui_home/bin.

    Putting DLLS where SoapUI can see them.

    This is how you set an activeEnvironment variable: setup happens at the project level:

    Creating environments at the project level.

    Then you choose your environment at the test suite level.

    Choosing environment in the test suite.

  4. Run a SQL query in a JDBC DataSource step to get a random policy number from the database of your choice on the fly. You can create your SQL query for the DataSource step dynamically in a Groovy script.
  5. // State Codes is a grid Data Source step whose contents aren't shown here.

    def stateCode = context.expand( '${State Codes#StateCode}' )

    testRunner.testCase.getTestStepByName("Get A Policy Number").getDataSource().setQuery("SELECT top 1 number as PolicyNumber FROM tablename p where date between '1/1/2012' and getdate() and state = '" + stateCode + "' order by newid()")

  6. Here’s the JDBC data source for the policy numbers.The query (not shown) is fed over from the preceding Groovy script step.
  7. JDBC DataSource step.

    You will have to set your connection properties. Your connection string for jTDS might look something like this. For more information about jTDS, see the online docs.

    jdbc:jtds:sqlserver://server:port/initialDatabase;domain=domainName;
    trusted_connection=yes

  8. I feed the value of PolicyNumber returned by the SQL query to my SOAP request via a property transfer.
  9. I have a few assertions in the SOAP request test step. The first two are “canned” assertions that require no scripting.
  10. Assertions in SOAP request.
    The third SOAP request assertion, which is more of a functional script than it is an assertion, captures the timestamp on the response as well as the time taken to respond. These are built-in SoapUI API calls. The properties TimeStamp and ResponseTime are created and initialized in this Groovy script – I didn’t have to create them outside the script (for example, at the test step level).

    import org.joda.time.DateTime

    targetStep = messageExchange.modelItem.testStep.testCase.getTestStepByName('Response Size and Time Log')
    targetStep.setPropertyValue( 'TimeStamp', new DateTime(messageExchange.timestamp).toString())
    targetStep.setPropertyValue( 'ResponseTime', messageExchange.timeTaken.toString())

  11. Another Groovy script step to get the size in bytes of the response:
  12. def responseSize = context.expand( '${Service#Response#declare namespace s=\'http://schemas.xmlsoap.org/soap/envelope/\'; //s:Envelope[1]}' ).toString()

    responseSize.length()

  13. Yet another Groovy script step to save responses as XML in a data sink, with policy number, state code, and active environment as the filename:
  14. def policyNumber = context.expand( '${Service#Request#declare namespace urn=\'urn:dev.null.com\'; //urn:Service[1]/urn:request[1]/urn:PolicyNumber[1]}' )

    def stateAbbrev = context.expand( '${Service#Response#declare namespace ns1=\'urn:dev.null.com\'; //ns1:Service[1]/ns1:Result[1]/ns1:State[1]/ns1:Code[1]}' )

    def activeEnvironment = context.expand( '${#Project#activeEnvironment}' )

    testRunner.testCase.getTestStepByName("DataSink").getDataSink().
    setFileName('''B/basedirectory/''' + policyNumber + '''_''' + stateAbbrev
    + '''_''' + activeEnvironment + '''.xml''')

  15. Some property transfers, including a count of policy terms (see part 3 of this series) in the response:
  16. Property transfers.

  17. DataSink to write the response XML to a file. The filename is set in step 7 above.
  18. File DataSink step.

  19. DataSink to write information relevant to performance to a CSV whose name is set on the fly. The filenames and property values come from the steps above.
  20. Response properties DataSink step.

  21. Don’t forget a delay step if you’re concerned about overloading the system. (Note that you can configure delays and threading more precisely in LoadUI. This test case, like all SoapUI test cases that aren’t also LoadTests, is single-threaded by default.)
  22. Delay step.

  23. And of course a DataSource Loop.
  24. DataSource Loop step.

  25. The whole test case looks like this:
  26. Whole test case.

  27. And now I have a CSV with response times, bytesizes, policy risk states, and term counts to parse in the tool of my choice. I chose R. More about that later.
Advertisements

Web services performance testing: a pilgrim’s progress. Part 3

This particular blog series is probably going to take me as long to finish as it did the medieval Muslim residents of al-Andalus to make a Hajj. So the “pilgrim” in this blog series title is apt.

It’s easy enough to set up a “load test” in a testing tool. It’s a little more challenging to frame the questions you want to ask about performance.

The services whose performance I was testing request data on policies that vary by insured risk state. The policy data resides in an IBM z/OS mainframe system and Datacom databases. The architecture works something like this:

  • .NET Web service request for policy data is made
  • Request is routed through middleware Web services that scrape mainframe screens or query Datacom
  • Middleware Web service returns policy data, or a SOAP fault, to .NET
  • .NET passes back the data to the requestor as XML

The team was especially concerned about the performance of the components that scraped the screens. Screen scraping can be slow, and our code would be sharing the subsystem that scrapes the data with a finicky Java messaging framework. Also, the middleware in question is very much due for an upgrade.

After thinking about these issues as well as some consultation with the project team, I designed my performance tests to record:

  • Response time in milliseconds per request
  • The risk state of the policy data being requested
  • The number of policy terms in the response: the usual number is two but the minimum number is one when the request is successful. If two terms are present, the number of screens that needs to be scraped doubles.
  • The size in bytes of the response. It is possible that a single-term response could be as large or larger than a two-term response in some cases, depending on the amount of data per mainframe screen and whether certain screens were used for that policy.

I made a couple more decisions based on the fact that I was testing in production (see part 1 of this series).

  • Requests would be made every 30 to 60 seconds over a period of a couple of hours for a total of 200 or so. Earlier tests at a higher frequency did not always go well (the aforementioned Java messaging framework was a rather feeble canary in the coal mine).
  • The team felt that the volume of requests (one or two a minute) was a realistic prediction of actual production load. The static frequency is not realistic, but my concern was again to avoid interfering with other production usage.
  • I felt that a sample size of 200 was “decent.” Since prod support staff had to monitor the production systems as I ran the test, a run time of anything over a couple of hours would not have been reasonable.

In the next posts I’ll review how I recorded and reported on data using SoapUI, Groovy, and the R statistics language.

Web services performance testing: a pilgrim’s progress, part 2

Here is where I vent some frustration. I’m self-taught on all the tools I use and I am the only person in my shop who knows how to use them in this manner. What’s more, I am the only Java/Groovy person right now at my shop. Being an autodidact can make one feel nice and self-sufficient, but the (large) holes in my knowledge are starting to cause me trouble. Why does the tool I use appear to do something very different from what I expect after reading the documentation? How do I instrument the tool to put load on a Web site through the UI, or should I be using a different tool?

Fortunately there’s been a shift in leadership at my workplace so that if I need to make the case for training, I’m more likely to be heard. So I’ll probably be doing that shortly. That being said, if the training doesn’t exist, I’m not sure what I will do. Is there training for JMeter or do developers just pick it up on their own? And if you don’t work in a Java shop, what do you do? Should I start looking into LoadRunner? I suppose I could put this question out to sqa.stackexchange or Software Testing Club and see what I come back with.

The testing community needs good TESTING tools – which are not necessarily automation tools – and reliable documentation, training, and support on those tools. There is DEFINITELY a market for this.

Also: I need the support of prod support/sysadmin types who, frankly, have better things to do than monitor a service in test for hours a day.  After several stress/performance tests have yielded conflicting results, my best bet would probably be to learn how to use the monitors they are using and go conduct some tests on my own time.

Finally: it’s an education to watch how quickly other people and I can fall into the confirmation bias trap. Example: I ran a low volume performance test against the production region on Friday, one I’ve run several times before apparently without incident. Sure enough, the system that is the most likely to have trouble with the load I’m creating started going haywire during the test. It continued to do so after I shut down the script, which makes me start wondering if my tool’s UI is telling me what I need to know. (A tool that monitors incoming requests against the system under load would be helpful here.)

Right away, people (including me) wanted to ascribe the problems with the system in production to the load against the system under test.  However, there’s really no proof to support that unless it happens repeatedly. And repeated production failures are not something I would wish on anyone. Sadly, our test system is not an exact replica of our production system – shared resources need to be allocated first to prod – so I’m not sure what performance testing on the test system will tell me.

So I have some new goals today:

  • Get a monitor that does a different view of the system under test (incoming requests) and learn how to use that monitor
  • Seek out some REAL training on the tools I use, or different ones. First I have to find the right place to ask those questions, though.
  • See if performance testing under the test region will tell us anything useful.