Automation in Performance Testing: resources

Sooo, I've had an opportunity to speak at the Testing Automation Summit yesterday in Auckland. The event was organized and hosted by the Testing Mind, and they did a good job off it. I am certainly grateful to have had a chance to be a speaker once again - haven't done it in a few years, and it was nice (although nerve-wrecking, hi, social anxiety) to be back.

As promised at the conference, here I am sharing some resources from the talk for those interested:

1. Slides with notes and all in pptx format or same slides on the speakerdeck.com.
2. Datapot and Pritcel report, as well as few other R-scripts I regularly use in performance testing are on github: https://github.com/hali/Performance-data-processing

Please let me know if there are any questions you didn't get a chance to ask at the conference!

Concurrency graphs

Just last week we were discussing one of the JMeter test scripts in our team, and it occurred to me that it would be useful to see the concurrency of the requests, meaning which requests run in parallel. That should very roughly represent which threads on the server are doing the same thing at the same time, and it is interesting as another way to look at what's going on during the test.

I knew there wasn’t such a chart in JMeter, because I thought about it briefly couple of times before and couldn't find it, but I was surprised to find that there wasn’t any ready to use tool to make that kind of chart. The closest I found was Gantt charts, but they are not designed to deal with thousands of requests and hundreds of threads. So I set up to make my own tool. In R, of course.

Here’s how the result looks (generated from one of the internal tests, legend cut off not to disclose anything):

The chart above shows which requests are currently running on each of the concurrent JMeter threads. It gives a very high level picture of what’s going on with the concurrency (requests are shown in different colors each, so you can assess the concurrency just by looking here). It is kinda hard to read though, especially in a case like this where we have a lot of requests, long duration of the test and long user thinking times in between. So here's another way to look at the same data:

This chart shows calculated concurrency level for different points of time (points of time being request start times). Concurrency is calculated as number of “overlaps” - i.e. if in the first graph you were to draw a vertical line and count the number of same-colored horizontal blocks intersecting that line - that would be the concurrency level for the corresponding request, and that value is displayed on the chart as (request_start_time, concurrency). Calculation is pretty rough, for more precision we'd like to calculate concurrency at regular time intervals, not just at request start times - that does mean longer running time and even harder to read chart though, so I decided I'm happy with what I've got here.

Of course, even this chart isn't easy to read, so if you want to drill into the data, I suggest running R-scripts in interactive mode and either look at text data, or chart only the specific request you are interested in. The code is there, and is hopefully easy to understand.

• Use this script to make the first graph: https://github.com/hali/Performance-data-processing/blob/master/scripts/concurrency_graph.R 
• Use this script to make the second graph and to explore the data: https://github.com/hali/Performance-data-processing/blob/master/scripts/calculate_concurrency.R

It's all good and nice to look at the data from the tests, but for it to be really useful you want to know whether it resembles production. You can check that by producing similar graphs from production usage data, e.g. from access.log. Please note, that access.log can be much noisier than JMeter output files, since in JMeter we get to name the samplers, whereas access.log will have each request with all the URL parameters included. So before graphing or calculating concurrency, you'll need to clear the data and filter out requests you are not interested in (e.g. you might want to remove static resources and/or remove dynamic user- or session- dependent parameters from other urls). Example of such filtering is in the script.

https://github.com/hali/Performance-data-processing/blob/master/scripts/access-log_concurrency.R

Feel free to grab the scripts, and please let me know in the comments if they were useful, or if you see some error in the calculations.

Oracle troubles and findings: tuning experience

I am not an Oracle DBA. But with performance testing I more often than not end up creating the whole environment, which means setting up Oracle server as well. Since I am not testing Oracle server specifically, I usually only tune it enough for it to not be the bottleneck. Lucky for me Oracle server is actually pretty cool compared to applications I test, and only gets to be a bottleneck in scalability testing where it serves multiple application servers. Still... recently I've bumped into a set of correlated problems that led me to tuning effort on the Oracle server itself.
Sponsored by Internet - meaning that all that I've done was googled in the internet and applied with fingers crossed.

So, here it goes.

First problem was the following: when I went from 4 application server nodes to 6 application server nodes, response times high rocketed. It was obviously an infrastructural problem, and after a while Oracle was the only culprit. Unlike usually, CPU usage wasn't that high on the oracle, so I had to dig a bit deeper, and guess what I found: concurrency issues such as "cursor pin S on X"!

To avoid doing hard parses, oracle puts any new query and it's execution plan into a shared cursors tree. Access to that tree is controlled by mutex pins algorithm. Only one session can grab a mutex pin for a specific cursor at a time. Also, similar queries are being put as leaves with a common root, and my understanding is the whole root is being pinned during any updates in the tree...

Anyway, it was happening for two reasons:
1. Application under test was using queries with literals where it should've been using prepared statements.
2. My oracle version (11.2.0.1) had known issues around shared cursors tree.

So I've updated to 11.2.0.4 and set CURSOR_SHARING=FORCE. What this option does is it replaces all literals in all queries by system variables, which effectively means that all the queries that only differ in literals are now treated as the same query, they have the same cursor, and cursors tree doesn't need to be constantly updated. This took care of concurrency issues. It also created another problem.

Suddenly one of my other queries which was never a problem before, a very simple and well behaved query, became a huge bottleneck. It would take thousands of CPU cycles to execute where before it was tens of cycles! This one took days of my time, numerous experiments that slightly improved the situation but didn't solve the main issue, and in the end I had to go to DBAs for help.

Turned out that innocent "1=2" in that query (which was there because the query was dynamically generated with optional conditions) was replaced by something like ":SYS_0=:SYS_1", and that meant Oracle was grabbing those variables and evaluating the clause again and again for each row in a huge table (I would think it would do it once, understand it's FALSE and leave it at it - but no).
This was of course the result of CURSOR_SHARING=FALSE. I'll say in advance, that I got exactly the same behaviour with CURSOR_SHARING=SIMILAR.

The suggested fix in my case was either to switch to prepared statements everywhere so that we don't need to use CURSOR_SHARING=FALSE/SIMILAR, or to remove "1=2" from the query that suffered from that setting. Can't have it both.

Other useful tuning:

• Increasing shared_pool_size.
• Increasing session_cached_cursors.
• Weirdly enough, locking statistics on selected columns helped.

R for processing JMeter output CSV files

So, I'm working as a performance engineer, and I run a lot of tests in JMeter. Of course most of those tests I run in non-GUI mode. To get results out of non-GUI mode there are two basic ways:

1. Configure Aggregate report to save results to a file. Afterwards load that file to a GUI JMeter to see aggregated results.
2. Use a special JMeter plugin to save aggregated results to a database.

I already wrote about the second way, so today I'll write about the first one. There are probably better ways to do it, but until recently this was how I processed results:

1. Run a test, have it save results to a file.
2. Open that file in Agregate Report component in GUI JMeter to get aggregated results.
3. Click "Save Table Data" to get new csv with aggregated results.
4. Edit that new CSV to get rid of samplers I am not interested in (mostly the ones that I didn't bother to name - e.g. separate URLs that compose a page), and to also get rid of the columns I am not interested in.
5. Sort the data in the CSV by Sampler - this is because I run many tests, and I need to compare results between runs. For that reason I create a spreadsheet and copy response times and throughput data to that spreadsheet, adding more and more columns for the table with rows labeled as samplers. Whatever, works for me.
6. Copy results from csv to a big spreadsheet and graph the results.

At some point I used macros and regexps in Notepad++ to do stage 4. Then my laptop died and I lost it, couldn't be bothered to write it again, even though it was big help. Still, even with the macro there were a lot of manual steps just to get to meaningful results.

But hey, guess what, I've been learning stuff recently - in particular Data Science and programming in R. So I used little I know and created this little script in R to do steps 2-5 above for me.

Now all I have to do is to place JMeter output files in a folder, start R Studio (which is a free tool, and I have it anyway) (you can probably do it with pure R, no need in R Studio even), set working directory to the folder with files and run the script. Script goes through all csv files in the folder (or you can setup whatever filenames list you want in the script), and for each file:

• Calculates Median, 90% Line and Throughput per minute for each sampler
• Removes the samplers starting with "/" - i.e. samplers I didn't bother to give proper names, so I am probably not very interested in their individual results.
• Removes delays (that's the thing with our scripts - we use Debug sampler usually named as "User Delay" to set up a realistic load model).
• Orders results by sample name.
• Saves results to a separate file.

One button and voilà - all the processing done. Now I only need to copy data to my big spreadsheet and graph it as I choose.

Script is in github, feel free to grab and use.

Clouds for performance testing

Cloud computing has been a buzz word in IT few years ago, and now it is rapidly becoming industry standard rather than some new thing. Clouds have matured quite a lot since they got public. Now, I do not know the full history of clouds, but in the last few months I had an opportunity to work with few of them and to assess them for my very particular purpose: performance testing in the cloud. What you need for performance testing is a consistent performance (CPU, memory, disks, network) and if that's a cloud - also an opportunity to quickly and easily bring environment up and down.  

These are the clouds I tried out:

• HP Cloud
• Rackspace
• AWS
• MS Azure

Without going much into details (to avoid breaking any NDAs), lets just say, that in each cloud I deployed a multi-tiered web application using either puppet master or (in case of Azure where I only really looked at the vanilla Cassandra database, see explanation below) internal cloud tooling. Then I loaded the solution and monitored resource utilisation, response times, throughput, and I noted down any problems that got in the way.

And here’s what I’ve got.

HP Cloud. The worst cloud I’ve seen. The biggest problems I’ve encountered are the following:

• Unstable VM hosts: two times a VM we used suddenly lost the ability to attach disks, which practically caused DB server to die, and us to lose extra day on creating and configuring a new DB server.
• Unstable network: ping time between VMs inside the cloud would occasionally jump from 1ms to 16-30ms.
• High steal CPU time - which means that VM would not get the requested CPU time from the host. During testing it got as high as 80% on the load generating nodes, 69% at the database server, 15% on the application servers.

There were also minor inconveniences such as:

• It is impossible to resize a live VM: you’ll need to destroy it, and then to recreate, if you need to add RAM or CPU.
• There is no option to get dedicated resources for a VM.
• Latest OS versions were not available in the library of images, which means that if you need a new OS version, you’ll have to install it manually, create a customised VM image, and pay separately for each license.
• Sometimes HP Cloud would have a maintenance that puts VMs offline for several hours.

HP Cloud was the starting point of my cloud investigation, and it was obvious we cannot use it for performance testing. So next I moved to Rackspace - another Openstack provider, more mature and powerful than HP Cloud. More expensive, as well. In Rackspace I didn’t have any problems with steal CPU time, nor with resizing VMs on the fly. It was a stable environment allowing to do benchmarking and load testing. However, it also had a bunch of problems:

• Sometimes a newly provisioned VM wouldn’t have any network connectivity but through Rackspace web console. Far more often a new VM wouldn’t have network connectivity for a limited amount of time (2-5 minutes) after the provisioning, which caused our Puppet scripts to fail and thus caused a lot of trouble in provisioning test environments. Rackspace tech support has been aware of the issue, but they weren’t able to fix it in the time I was on a project (if they fixed it later, I wouldn’t know).
• There were occasional spikes in the ping times up to 32 ms.
• Hardware in Rackspace wasn’t up to our standards: CPU we got didn’t have a lot of cache, so our application would stress out CPU much more than on the hardware we used “at home”. That practically meant that to get the performance we wanted we’d need at least twice as much hardware, which was quite expensive.

After Rackspace we moved onto AWS (my colleagues did more stuff on AWS, than me, thus “we”), and we were amazed at how good it was. In AWS we didn’t have any of the problems we had in Openstack. AWS runs on good hardware (including SSD disks), allows to pay for dedicated resources (but we didn’t have to do it, because even non-dedicated VMs gave consistent results with zero steal time!), shows consistent small ping times between VMs, has a quite cool RDS service for running Amazon-managed easy-to-control relational database servers.

Yet, AWS is not cheap. So we thought we'd quickly try MS Azure to see if it can provide comparable results for a lower price. Because I was to compare Azure vs AWS in few specific performance-related areas (mostly I was interested to see CPU steal times, disk and network performance), I ran few scalability tests for the Cassandra database. Cassandra is a noSQL database, that is quite easy to install and start using. What was cool for my purposes, it has a built in performance measuring tool named cassandra-stress. It's a fast to setup and extremely easy to run test, and also Puppet just wouldn't work with Azure, so instead of the multi-tiered web application I went with Cassandra scalability test.

MS Azure wasn’t actually that bad, but it is nowhere near AWS as an environment for running high loads:

• The biggest problem seemed to be network latency. Where AWS was doing perfectly fine, Azure had about 40% failures on timeouts on high loads. Ping times between nodes during tests were as high as 74 ms at times (compared to 0.3 ms in AWS under similar load). From time to time my SSH connection to this or that VM would break for no apparent reason.
• Concurrently provisioning VMs from the same image is tricky: part of the resources is actually locked during VM creation, and no other thread can use it. That caused few "The operation cannot be performed at this time because a conflicting operation is underway. Please retry later.” errors when I was creating my environment.
• Unlike AWS, Azure doesn’t allow you to use SSD, which means a lower disk IO performance. Also in Azure there are limitations on the number of IOPS you can have per storage account (though to be fair there is no practical limitation on how many storage accounts you can have in your environment). Even using RAID-0 of 8 disks didn’t allow me to reach the performance we easily had in AWS without a RAID.
• For some reason (I am not entirely sure it was MS Azure fault) CPU usage was very uneven between the Cassandra nodes, even though the load on each node was pretty much the same.
• I was not able to use Puppet because the special Puppet module for MS was out of sync with the Azure API.

This being said, Azure is somewhere near Rackspace (if not better) in terms of performance, and is quite easy to use. For a non-technical person who wants a VM in the cloud for personal use I’d recommend Azure.

For running performance testing in the cloud, AWS is so far the best I've seen. I also went through few of Amazon courses, and it looks to me like the best way to utilise AWS powers is to write an application that would use AWS services (such as queues and messages) for communicating between nodes.

As a summary: from my experience I would recommend to stay away from the HP Cloud, to use MS Azure for simple tasks, to use AWS for complicated time-critical tasks. And if you are a fan of Openstack - Internet says Rackspace is considered to be the most mature of the Openstack providers and to run the best hardware.

A bit about measuring client-side performance for web applications

What the heck is client-side performance for web applications? Well, it's about the time browser on your user's computer takes to download all the elements on a page and render that page. It's also about the speed of JavaScript on the page. It's not, strictly speaking, about the network between a user and a server, but that usually gets measured too.

Why do you care about it? Because no matter how fast are your servers in, well, serving responses to user's requests - if user still ends up with a slow application, (s)he wouldn't care about the reasons and would rather just go to someone else's site. Sometimes milliseconds matter in how much revenue you'll get, or how many clients your site will bring to you.

I've been doing a bit of research on client-side performance of web applications today, and I found quite a lot of interesting stuff that I didn't know about earlier. I thought it can be useful to put it all in one place. :-)

First of all, amongst all the cool tools I found these two free online analyzers:

• http://www.webpagetest.org/
• http://developers.google.com/speed/pagespeed/insights/

Both of them provide pretty detailed statistics and analysis of the page performance, and they also provide simple recommendations on how to fix the discovered issues. First tool has a fair amount of settings allowing you to emulate different browsers, different locations, network throttling, browser settings etc. Second tool doesn't have that much settings, but it shows results for both desktop and mobile site versions. Pretty cool, look for yourself!

Awesome as they are, these two tools can only be applied to publicly available pages. Luckily there are also few simple non-intrusive tools that can help developers and testers in measuring client-side performance of their applications in testing environments - i.e. without having to build some libraries into a code base (as you have to do with boomerang, Google.analytics or countless paid frameworks with similar functionality).

Here we go:

• Fiddler for any Windows browser (Charles for Mac browsers?) - measures time spent on downloading different resources, can show a timeline for a page. If you select few requests and look at Timeline tab, you'll see which requests are called in parallel, and which are not.
• Google Insights browser extensions (for Chrome and Firefox) - show detailed analysis on page performance, advice on actions to improve it.
• AJAX View - ancient proxy from Microsoft, gives performance statistics for JavaScript on a page down to the function level. Looks like the project has been abandoned in 2009, but the downloadable package is still there.
• APM DynaTrace AJAX edition - paid tool with free trial, that gives page performance analysis down to function level.

As everyone knows, it's much cheaper to prevent than to fix, so here's few articles I bumped into that can help developers in designing a page so as to maximize perceived client-side performance:

• Recommendations from Yahoo
• Recommendations from Google
• Top ten client-side performance problems according to DynaTrace stats

Cool thing about both of these lists is that it's all easily applicable on a stage of functional testing. And it should be! In my experience functional testers are perfectly good in noticing when the software under test is slow, but it rarely gets measured or reported, so it stays on the level "I feel this could be faster". With these tools it's quite easy to look under the hood, get some measurement and communicate with developers in a language of facts rather than feelings.

To go further down the rabbit hole you'll need active participation from development team. There are plenty frameworks out there that allow you to gather information on performance on real users' computers. The only catch is that you have to actually include some additional code into your application to do that. The amount of code would differ between frameworks, and so will the price. I haven't actually tried to use any of those frameworks myself so I cannot compare them, but development team should be able to choose the one that suits your case the best.

That's just those I found today, free ones are marked green:

• https://github.com/HubSpot/BuckyClient - "Bucky is a client and server for sending performance data from the client into statsd+graphite, OpenTSDB, or any other stats aggregator of your choice." That's the only self-hosted tool I found - i.e. tool which won't send any information whatsoever to a third party servers.
• Google Analytics
  - https://developers.google.com/analytics/devguides/platform/
  - https://developers.google.com/analytics/devguides/collection/analyticsjs/
• http://newrelic.com/real-user-monitoring - New Relic Browser, has free limited features offering and 2-weeks full features trial
• App Dynamics - http://www.appdynamics.com/ - a paid tool, similar to google analytics
• http://www.compuware.com/en_us/application-performance-management/products/user-experience-management/real-user-monitoring-web-and-mobile.html
• http://www.compuware.com/en_us/application-performance-management/products/dynatrace-free-trial.html
• http://www.lognormal.com/boomerang/doc/
• http://www.real-user-monitoring.com/

Aaaaand that's probably it. I hope it'll give someone a start. :-)

How to make JMeter save results to a database

Our team is responsible for performance testing in the company, and JMeter is one of our major tools. Recently we understood that to make our job easier we need to save aggregated test results to a database instead of a file. There are different use cases for that, the most important being to run periodic automatic performance testing and graph the results to see if a new build dropped in performance. You can do it manually, sure, but saving results to a database makes it so much easier.

Turns out there is currently no plugin for JMeter (that we know of) that would allow you to do that. So a collegue of mine made a plugin to do just that. Then I got interested, and decided to do it a bit differently, and found a problem in his plugin, and... to make the long story short now we have two slightly different plugins that save aggregated results to a database.

I'll probably try and share source code in JMeter's Github once I'm not too lazy to figure out how to do that, but for now I just want to share few things I learned on the way. And boy, was it a harsh way. JMeter has the most unfriendly API I've ever seen (though to be fair I haven't seen much), it looks crazy, and I couldn't find answers to my questions in the internet when I needed them, so here you go. Maybe the next person who decides to make a fancy visualizer will benefit from this post.

So, you decided to write a visualizer that does something more than just calculating statistics differently or showing the data you already have in a prettier graph. Here's some tips that go beyound common sense and general ability to write code in Java and google.

• Do not do any logic in your visualizer class, because in non-GUI mode JMeter will behave like your class doesn't exist, so you will get no logic whatsoever. Think about standard Aggregate report: in non-GUI mode all you can do is save every and each request to a file, you cannot just get an aggregated table. That's because aggregation happens in the visualizer code, and visualizer doesn't get initialized in non-GUI mode.
• In non-GUI mode samples are being processed by ResultCollectors. This is where you want to put all your logic. To do that you need to implement two classes and integrate them with each other:
• implement your visualizer (extend AbstractVisualizer)
• implement your result collector (must extend ResultCollector, or it won't get started)
• override "createTestElement" method in the visualizer and create your result collector. You must also override "modifyTestElement" and "configure" methods and make sure the proper constructor of your result collector is called. See the example 1 for "createTestElement" in the end of the post. If you don't do it, JMeter won't know that your two classes are connected. Basically this is where you say: okay, my GUI shows information which really comes from this TestElement, so please create this TestElement even when you skip the GUI.
• In your result collector override the "sampleOccured" method and put all your aggregating results logic here. You also probably want to call "super.sampleOccured" in the beginning - this way you will get the standard logic (check if sample is okay, send it to visualizer) as well as your new one.
• If you need to get settings from the GUI, make sure they are saved as properties of your result collector (and not as properties of your visualizer).
• Keep in mind that you can only access properties after the constructor. Yep, cannot to it in constructor even if your result collector is fully initialized at that point. JMeter runs the "give test elements their properties from the jmx file" process only after it initialized those elements.
• If you want to use JMeter variables (e.g. you have a field "Test run description", and you want to put some context dependent info there), be aware: you only get access to parsed variables in result collector, but not in the visualizer. In other words, getPropertyAsString will give you "${host}" if called from visualizer, though it will give you "192.168.7.15" if called from result collector for the same property.
• If you want to know active number of threads (load level) you will have to calculate it. See example 2 below for what to put in the "sampleOccured" method in result collector or in the "add" method in visualizer to do that.

Promised examples.

example 1:

@Override

public TestElement createTestElement() {

            if (collector == null || !(collector instanceof DBResultCollector)) {

                collector = new DBResultCollector();

            }

            modifyTestElement(collector);

            return collector;

}

example 2:

loadLevel = Math.max(res.getAllThreads(), loadLevel);

//where res is SampleResult

JMeter non-GUI distributed testing

Since I'm now in performance testing, I've been using JMeter quite a lot in the recent months. I've also spent a lot of time on google after I read the whole JMeter reference guide. No reason not to share little things now that the problems are solved.

Distributed testing with JMeter is supposed to be a very easy task according to the official guide. Yet, for me it didn't work straight away. I blame it on the way we deployed the nodes (using puppet master that screwed up configuration at first), and you may also get this if you deploy JMeter by copy-pasting JMeter folder between nodes. Solution to all my problems turned out to be really simple, but maybe writing it down will save someone else time on googling.

My environment: 

- few CentOS 6.3 VMs, one of them arbitrary chosen to be JMeter master (others are slaves). No GUI access to any of the nodes;

- my local machine with GUI JMeter.

Steps to make it work and get results back

1) on each JMeter node modify $JMETER_HOME/bin/jmeter

- make sure heap size is configured as you want it

- if you want Visual JVM monitoring, make sure host is set to a proper IP address on each node

2) on each slave node modify jmeter-server

- make sure that RMI_DEF_HOST is explicitely set to that node's IP address

3) on JMeter master modify jmeter.properties file

- set remote_hosts to a list of all JMeter nodes except the master itself

4) on each slave node start/restart JMeter server. Either use service jmeter restart (if you installed JMeter server as a service) or kill current JMeter server thread, and then start it with $JMETER_HOME/bin/jmeter-server

5) if your testscript uses any external files, you have to put those files on each slave node into the same place, which is specified in testscript. From my experience it's safer to specify full path rather than relative (i.e. /etc/testscripts/in/userlist.csv instead of in/userlist.csv), because relative paths are resolved based on the folder from which you are executing jmeter, and not from the testscript location

6) in your testscript configure listeners to save results to files. Again, it is safer to use full paths instead of relative. JMeter will gather results from all slave nodes and save them on master node

7) place your testscript on JMeter master and run it with a command like this:

$JMETER_HOME/bin/jmeter -n -t <test script> -r

- additionally if you want to get some graphs, enable appropriate listener in your testscript and run jmeter with additional parameter -l <filename.jtl>. Note that this creates significant additional load on JMeter nodes, so it probably shouldn't be used when it is not absolutely necessary.

- additionally you can configure JMeter Summarizer to save and show aggregated responce times (one set of results for all samplers) during the run. This blog post does a good job describing how to use it. For me it wasn't of much use because I needed to get aggregated response times separately for each of the many samplers.

8) to see aggregated results, copy files that were generated by listeners from JMeter master to your local machine, run JMeter in GUI mode, open your testscript, choose appropriate listener and point to the file. JMeter will process the file and give you aggregate table as if you ran the script in GUI mode from the beginning. Same works for graph listeners - just point them to a generated *.jtl file.

9) oh, and if you need to stop the distributed test in non-GUI mode, use $JMETER_HOME/bin/shutdown.sh script on the JMeter Master - it will send "shutdown all" to slave nodes.

Hope that was helpful. I know I would be glad to find this when I first googled for "jmeter distributed testing get aggregate response times", or "jmeter master cannot connect to slaves", or "jmeter master doesn't get results back from slaves". :-)