HAProxy balancing https backends

Recently I needed to configure load balancing in my environment, where I needed to balance between few https servers with sticky sessions enabled. I looked in the haproxy manual, I googled, I asked - and for days there was no making it work.

Most of the haproxy configuration examples out there are for the case when client connects to haproxy via https, and then haproxy decrypts it and balances requests between http backends. Few examples around https backends assumed that no sticky sessions are needed, so they all sit on top of tcp. To this day I have not found a guide or an example of how to configure what I need, so once I figured out how to do that, I thought I'd share.

So the way you do it is:
0) You need haproxy 1.5+. haproxy before that did not support https on its own.
1) A client connects to haproxy via https. There need to be a certificate+private key combination (that client would trust) on the haproxy server.
2) HAProxy decrypts the traffic and attaches a session cookie. If the cookie is already there, it knows where to send the request further.
3) HAProxy encrypts the traffic again before sending it to backend (where backend can decrypt it).
4) and the other way around.

And the configuration for that is:

• For both backend and frontend you should have mode http.
• In the bind line you need to add ssl cert <path to haproxy certificate + private key file>.
• In the backend section you need to set load balancing algorithm - e.g. roundrobin or leastconn.
• In the backend section you also need to set a cookie - e.g. cookie JSESSIONID insert indirect no cache.
• For each server you need to say "ssl" after the ip, and then also set a cookie.

For me the one part I couldn't find in any guides was to put "ssl" in the server line (as well as in the bind line). I might have missed it somewhere in the not-so-helpful haproxy manual.

One thing I didn't go into was setting up a proper certificate on a backend servers in my environment, because of course in test environment they are self signed and all that. In order to work around it, just add another global setting to the haproxy settings: ssl-server-verify none.

And here's the example of the config file frontend & backend sections to make it work:

frontend  main

    mode http
    bind :443 ssl crt /etc/haproxy/cert.pem
    default_backend app

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app

    mode http
    balance     roundrobin
    option httpchk GET /concerto/Ping
    cookie JSESSIONID insert indirect nocache
    server  app1 10.0.1.11:443 ssl check cookie app1
    server  app2 10.0.1.12:443 ssl check cookie app2
    server  app3 10.0.1.13:443 ssl check cookie app3
    server  app4 10.0.1.14:443 ssl check cookie app4

A bit about measuring client-side performance for web applications

What the heck is client-side performance for web applications? Well, it's about the time browser on your user's computer takes to download all the elements on a page and render that page. It's also about the speed of JavaScript on the page. It's not, strictly speaking, about the network between a user and a server, but that usually gets measured too.

Why do you care about it? Because no matter how fast are your servers in, well, serving responses to user's requests - if user still ends up with a slow application, (s)he wouldn't care about the reasons and would rather just go to someone else's site. Sometimes milliseconds matter in how much revenue you'll get, or how many clients your site will bring to you.

I've been doing a bit of research on client-side performance of web applications today, and I found quite a lot of interesting stuff that I didn't know about earlier. I thought it can be useful to put it all in one place. :-)

First of all, amongst all the cool tools I found these two free online analyzers:

• http://www.webpagetest.org/
• http://developers.google.com/speed/pagespeed/insights/

Both of them provide pretty detailed statistics and analysis of the page performance, and they also provide simple recommendations on how to fix the discovered issues. First tool has a fair amount of settings allowing you to emulate different browsers, different locations, network throttling, browser settings etc. Second tool doesn't have that much settings, but it shows results for both desktop and mobile site versions. Pretty cool, look for yourself!

Awesome as they are, these two tools can only be applied to publicly available pages. Luckily there are also few simple non-intrusive tools that can help developers and testers in measuring client-side performance of their applications in testing environments - i.e. without having to build some libraries into a code base (as you have to do with boomerang, Google.analytics or countless paid frameworks with similar functionality).

Here we go:

• Fiddler for any Windows browser (Charles for Mac browsers?) - measures time spent on downloading different resources, can show a timeline for a page. If you select few requests and look at Timeline tab, you'll see which requests are called in parallel, and which are not.
• Google Insights browser extensions (for Chrome and Firefox) - show detailed analysis on page performance, advice on actions to improve it.
• AJAX View - ancient proxy from Microsoft, gives performance statistics for JavaScript on a page down to the function level. Looks like the project has been abandoned in 2009, but the downloadable package is still there.
• APM DynaTrace AJAX edition - paid tool with free trial, that gives page performance analysis down to function level.

As everyone knows, it's much cheaper to prevent than to fix, so here's few articles I bumped into that can help developers in designing a page so as to maximize perceived client-side performance:

• Recommendations from Yahoo
• Recommendations from Google
• Top ten client-side performance problems according to DynaTrace stats

Cool thing about both of these lists is that it's all easily applicable on a stage of functional testing. And it should be! In my experience functional testers are perfectly good in noticing when the software under test is slow, but it rarely gets measured or reported, so it stays on the level "I feel this could be faster". With these tools it's quite easy to look under the hood, get some measurement and communicate with developers in a language of facts rather than feelings.

To go further down the rabbit hole you'll need active participation from development team. There are plenty frameworks out there that allow you to gather information on performance on real users' computers. The only catch is that you have to actually include some additional code into your application to do that. The amount of code would differ between frameworks, and so will the price. I haven't actually tried to use any of those frameworks myself so I cannot compare them, but development team should be able to choose the one that suits your case the best.

That's just those I found today, free ones are marked green:

• https://github.com/HubSpot/BuckyClient - "Bucky is a client and server for sending performance data from the client into statsd+graphite, OpenTSDB, or any other stats aggregator of your choice." That's the only self-hosted tool I found - i.e. tool which won't send any information whatsoever to a third party servers.
• Google Analytics
  - https://developers.google.com/analytics/devguides/platform/
  - https://developers.google.com/analytics/devguides/collection/analyticsjs/
• http://newrelic.com/real-user-monitoring - New Relic Browser, has free limited features offering and 2-weeks full features trial
• App Dynamics - http://www.appdynamics.com/ - a paid tool, similar to google analytics
• http://www.compuware.com/en_us/application-performance-management/products/user-experience-management/real-user-monitoring-web-and-mobile.html
• http://www.compuware.com/en_us/application-performance-management/products/dynatrace-free-trial.html
• http://www.lognormal.com/boomerang/doc/
• http://www.real-user-monitoring.com/

Aaaaand that's probably it. I hope it'll give someone a start. :-)

Lessons learned in non-functional testing

Yesterday I gave a lightning talk at my favorite WeTest Auckland meetup hosted by Assurity. It was a five minutes (okay, maybe 6 minutes) long presentation with the 15 minutes of discussion afterwards. So, for five minutes I had to make it really short, yet making sense.

The main idea I was trying to express is really simple: plan for non-functional quality characteristics right from the beginning.

As an IT graduate I had this assumption when I first started to work as a software tester, that developers would do proper technical and database design according to the task at hand, and then perform testing of the product from the very beginning. In other words, I thought it was obvious that someone would start testing as soon as you get your first requirements from a customer. In reality it wasn't like this at all. Testing usually started at the latest stages of development, which cost projects lots of work hours, effort and money, and often reputational losts. As years passed by, situation seemed to improved. Nowadays most IT companies accept that testing is important, and that it should begin as soon as possible. With one "but" - it only applies to functional testing.

When I switched from functional to performance testing last September, I didn't have that naive assumption about functional testing beeing done the right way everywhere... turns out, I still had even more naive assumption that non-functional testing is done the right way. I haven't done any serious programming in 7 years, yet when occasionally I think about implementing this or that, I still operate by the algorithm I learned in uni: I ask myself questions about a problem I'm trying to solve, and about the future of the software that would solve it. I thought it was obvious: when you write e.g. a web application, you think about bunch of stuff on top of functionality: how would it scale, how would you plug in new functionality, how would you localize it, how would you customize it, what questions will users ask about it, how would you test it, and so on. Sometimes most of the answers would be "it doesn't matter because..." - but important thing is to ask questions and know that for sure.

Well, in real life it happens rarely. I'm not talking about companies like Google or Amazon - surely those guys know what they are doing if only because they had enough experience in the area. But most companies aren't Google. What I see in real life is fire fighting: application is created, then it goes into production, and then you have problems in production, which are extremely expensive to fix. Some issues are obvious: e.g. security breaches and performance and/or scalability breakdowns. Some issues are more subtle: e.g. bad supportability would mean a lot of time and effort spent by field agents (support line, or operators, or whatever you call people who talk with unhappy users). Bad maintainability would mean additional problems when you need to install updates or reconfigure something in live system. Bad testability would mean more time necessary for testing e.g. new release, and potentialy worse quality because some areas testers won't be able to reach. Bad initial usability will bite you in the ass when you decide to implement shiny new GUI which would be different from the existing one, because users will be resistant to changes (remember MS Office 2003 -> 2007 outcry). And what's even worse, to fix non-functional issue often requires to implement fundamental architecture changes.

To summarize, non-functional quality is important, it costs a lot to fix, it isn't being given enough attention.
So, the question is, if you work in one of those companies, what can you do?

As part of the team, regardless of your role, you can communicate and underline the importance of the following:

• When doing tech design for a new application or a new feature, think about the future. What would happen with your application in a year? What new functionality it might have? What would be the environment and the load (e.g. number of users per hour)? Are you gonna store any sensitive data? Would you need to create additional applications (e.g. mobile) to coordinate with it? What additional data you might need to store? How would you localize/customize it if necessary? How often would you need to release a new version?
• Do risks analysis. Always do risks analysis - better be prepared for the trouble that might come your way.
• Involve specialists in specific fields if your team lacks those: there are consultancies, that can review tech design, or do security testing, or performance testing, for you.
• Start the actual testing as soon as possible - not only functional testing, but non-functional too. On early stages it would look differently from how it loos on later stages: e.g. it makes little to no sense to do a full performance testing for a half-made release, but it is possible to do performance testing of existing separate components, especially of the core ones. It pays off because the earlier you fix the issue, the cheaper it is.
• Treat non-functional issues fair. If the team doesn't fix e.g. testability issues in time, you will pay for it later with much more complicated testing and higher price for building testability into the project. If performance or security issues are not being treated with respect because "it still works, no one will notice it's slow/insecure", you will pay with your reputation and money when it manifests on production. And if your product is a success, it will manifest in production sooner or later. #too_much_painful_experience

After I did the talk, one of the testers on the meetup asked if all of this makes sense for startups and such - she implied that in the beginning you don't want to spend your time thinking about all this stuff, because you are under pressure to deliver fast. This is another point of view, but I strongly disagree with it. The only case in which you can forget about quality is when you are making proof of concept, then throw it out the window and start anew. If you are gonna use the codebase of your first release further, it pays off greatly to spend few extra hours (and in the very beginning you would only need few extra hours) on analysing the future of your application and doing your tech design (database design and application design) properly.

Remember: there is nothing more permanent than a temporary solution.

Slides for the presentation are shared here:
https://docs.google.com/file/d/0Bxi4eMT3I97ea3luRU9nemRpODg/edit

How to make JMeter save results to a database

Our team is responsible for performance testing in the company, and JMeter is one of our major tools. Recently we understood that to make our job easier we need to save aggregated test results to a database instead of a file. There are different use cases for that, the most important being to run periodic automatic performance testing and graph the results to see if a new build dropped in performance. You can do it manually, sure, but saving results to a database makes it so much easier.

Turns out there is currently no plugin for JMeter (that we know of) that would allow you to do that. So a collegue of mine made a plugin to do just that. Then I got interested, and decided to do it a bit differently, and found a problem in his plugin, and... to make the long story short now we have two slightly different plugins that save aggregated results to a database.

I'll probably try and share source code in JMeter's Github once I'm not too lazy to figure out how to do that, but for now I just want to share few things I learned on the way. And boy, was it a harsh way. JMeter has the most unfriendly API I've ever seen (though to be fair I haven't seen much), it looks crazy, and I couldn't find answers to my questions in the internet when I needed them, so here you go. Maybe the next person who decides to make a fancy visualizer will benefit from this post.

So, you decided to write a visualizer that does something more than just calculating statistics differently or showing the data you already have in a prettier graph. Here's some tips that go beyound common sense and general ability to write code in Java and google.

• Do not do any logic in your visualizer class, because in non-GUI mode JMeter will behave like your class doesn't exist, so you will get no logic whatsoever. Think about standard Aggregate report: in non-GUI mode all you can do is save every and each request to a file, you cannot just get an aggregated table. That's because aggregation happens in the visualizer code, and visualizer doesn't get initialized in non-GUI mode.
• In non-GUI mode samples are being processed by ResultCollectors. This is where you want to put all your logic. To do that you need to implement two classes and integrate them with each other:
• implement your visualizer (extend AbstractVisualizer)
• implement your result collector (must extend ResultCollector, or it won't get started)
• override "createTestElement" method in the visualizer and create your result collector. You must also override "modifyTestElement" and "configure" methods and make sure the proper constructor of your result collector is called. See the example 1 for "createTestElement" in the end of the post. If you don't do it, JMeter won't know that your two classes are connected. Basically this is where you say: okay, my GUI shows information which really comes from this TestElement, so please create this TestElement even when you skip the GUI.
• In your result collector override the "sampleOccured" method and put all your aggregating results logic here. You also probably want to call "super.sampleOccured" in the beginning - this way you will get the standard logic (check if sample is okay, send it to visualizer) as well as your new one.
• If you need to get settings from the GUI, make sure they are saved as properties of your result collector (and not as properties of your visualizer).
• Keep in mind that you can only access properties after the constructor. Yep, cannot to it in constructor even if your result collector is fully initialized at that point. JMeter runs the "give test elements their properties from the jmx file" process only after it initialized those elements.
• If you want to use JMeter variables (e.g. you have a field "Test run description", and you want to put some context dependent info there), be aware: you only get access to parsed variables in result collector, but not in the visualizer. In other words, getPropertyAsString will give you "${host}" if called from visualizer, though it will give you "192.168.7.15" if called from result collector for the same property.
• If you want to know active number of threads (load level) you will have to calculate it. See example 2 below for what to put in the "sampleOccured" method in result collector or in the "add" method in visualizer to do that.

Promised examples.

example 1:

@Override

public TestElement createTestElement() {

            if (collector == null || !(collector instanceof DBResultCollector)) {

                collector = new DBResultCollector();

            }

            modifyTestElement(collector);

            return collector;

}

example 2:

loadLevel = Math.max(res.getAllThreads(), loadLevel);

//where res is SampleResult

Exploratory testing is about exploration - unedited version of article for "Trapeze"

(That's what I initially sent to the journal. It got quite edited and rewritten on the way. I still like initial version better, so here it is.
Edited version can be found here: http://www.testingcircus.com/testing-trapeze-2014-february-edition/)

Exploratory testing seems to become more and more of a a big thing lately, which fills my heart with joy, but in the recent years I realized that people mean completely different things when they talk about it. Sometimes this difference is amusing, sometimes it's enriching and eye-opening,  and sometimes it's annoying (e.g. when your colleagues refuse to try something new, claiming that they are already "doing this exploratory testing thing"). I don't claim to have an ultimate answer, of course, but I'd like to talk about what I mean under "exploratory testing", and why I love it so much.

For me, exploratory testing is all about the idea of exploration. It's not about using charts and session-based testing, it's not about agile environment, and it's definitely not about some list of heuristics (becoming the new "best practices") you absolutely must use. It's about asking questions, experimenting and gaining knowledge about the world (and the software under test in particular). And as a good explorer, of course you are also bound to keep good notes of your deeds. I like to think about it as if I were the Curiosity rover. Or even better, a crew member of the starship Enterprise (LLAP to all you fellow geeks out there): exploring brand new worlds and happily startrekking across the universe. Never knowing all of it, but having the tools, the desire and the attitude to acquire new knowledge.

As a tester, I have knowledge and assumptions about the software I'm about to test: business requirements, technology description, knowledge of the environment, common sense, etc.. I also have a map, based on my knowledge: test cases, test strategy, process I'm about to follow. And of course I have many tools to help me on my job: from special software to heuristics and ol' good test analysis techniques. All of that gives me a good place to start, but if I let it to define what I do, my job will become mechanical and it really wouldn't need the power that is human brain that much. When I'm doing exploratory testing (and I'm always doing exploratory testing), I have to keep asking questions and remember to readjust my assumptions. It absolutely blows my mind how much you can gain from such a simple idea!

First of all, it gives you efficiency, because by asking questions you gain understanding of software, you make sure you don't use outdated documentation, you get to know people on a team and what they can do, you are providing fast feedback, and you are helping everyone on the team to do better job with having more up-to-date information about the project than they would otherwise have. It also makes the job more fun, which you can't underestimate if you get bored as easily as I do. Another thing exploratory testing encourages you to do is to use techniques from other fields -  humans have been exploring the world since forever, and there is heaps of historical experience and wisdom waiting to be applied to testing software. Also, if you agree that testing is about asking questions and gaining knowledge, it can help you with the project roles that others try to enforce on you. I found that this is a common occurrence, that the product manager (or whoever is responsible for the application) presses the testing team into answering the "is it ready to go production", and "when it will be ready" questions. The problem here is that usually testing team don't really have the power to influence the situation much: they can't decide what is acceptable to be in production, they can't assign additional time for testing or for development teams, and they can't rush developers, designers, translators, etc. into doing their jobs. And if you have no power, you can't take the responsibilities.

All these seemingly abstract ideas form a perfectly practical approach, that I've been more or less successfully applying on my daily job for the last 6 years, even before I knew the terminology to talk about it and to understand exactly what I'm doing. I call that approach exploratory testing. Let me share with you its main points in a series of statements.

1. The mission of software testing isn't to "provide quality", it's to gather and provide helpful information to those who make decisions about the product and it's quality.
2. Exploratory testing is an approach that doesn't depend on SDLC, it can be applied to any situation (even outside of testing itself).
3. You can never know for sure how much time will it take to finish the testing. The challenges of planning and performing testing are rooted in the "work in the unknown" part. It's just like scientific research in that regard.
4. Exploratory testing consists of many iterations of two very different steps: discover and investigate. Each step has it's own challenges and goals. On the "discover" step you are concentrated on finding the issue (issue being a problem or a question), and on the "investigate" step your goal is to gain information you need to deal with that issue.
5. It is important to ask questions (no such thing as a stupid question!), to look at problem from different angle from time to time, to perform root cause analysis, to use new tests instead of old ones, and to keep notes.
6. Use automation to help with mundane tasks and free your precious time for smart tasks - not to replace manual testing of everything.
7. Give feedback as soon as you can make it useful. Give clear feedback, and make sure it isn't personal or insulting.
8. Prioritize your work (which features to test first, how much time to spend on a problem, which risks to mitigate and which tasks to do first, etc.).
9. Know your tools well: heuristics, practices, diagram notations, software that can help on the job, etc.
10. Know software you are testing from business and technological points of view: what problems is it supposed to solve, who are the stakeholders, and how does it work (on a system level at least).
11. Don't spend time on documentation no one will read or use.
12. Keep documentation that is in use up-to-date. Don't let your experience and knowledge of the project to stay only in your head.
13. Use test ideas to guide your testing rather than test cases to define it.

There is one more thing I feel necessary to say out loud. Exploratory testing (the way I understand it) doesn't require you to be an experienced tester in order to be good at it. A completely new to a product and/or software testing itself person can learn exploratory testing, use it right away and be awesome. The only thing that is absolutely required is the passion to explore. Everything else will come with the answers to the question you ask. I tested that with few new testers way back by teaching them to do exploratory instead of the classical "do step-by-step test cases until you learn them by heart" approach, and it worked, they are now totally awesome.

That's all, folks. Once again, I don't claim to know what exploratory testing is really about, but just one way of looking at it. I'd like to thank all testers, whose blogs I read for the last 8 years, because I definitely learned a lot by doing that. And specifically, the biggest thank you goes to James Bach and Michael Bolton for their absolutely brilliant "Rapid Software testing" course, which gave me ideas to think about, confidence that I'm not going the wrong way and most of all terminology I use to vocalize my thoughts on testing.

JMeter non-GUI distributed testing

Since I'm now in performance testing, I've been using JMeter quite a lot in the recent months. I've also spent a lot of time on google after I read the whole JMeter reference guide. No reason not to share little things now that the problems are solved.

Distributed testing with JMeter is supposed to be a very easy task according to the official guide. Yet, for me it didn't work straight away. I blame it on the way we deployed the nodes (using puppet master that screwed up configuration at first), and you may also get this if you deploy JMeter by copy-pasting JMeter folder between nodes. Solution to all my problems turned out to be really simple, but maybe writing it down will save someone else time on googling.

My environment: 

- few CentOS 6.3 VMs, one of them arbitrary chosen to be JMeter master (others are slaves). No GUI access to any of the nodes;

- my local machine with GUI JMeter.

Steps to make it work and get results back

1) on each JMeter node modify $JMETER_HOME/bin/jmeter

- make sure heap size is configured as you want it

- if you want Visual JVM monitoring, make sure host is set to a proper IP address on each node

2) on each slave node modify jmeter-server

- make sure that RMI_DEF_HOST is explicitely set to that node's IP address

3) on JMeter master modify jmeter.properties file

- set remote_hosts to a list of all JMeter nodes except the master itself

4) on each slave node start/restart JMeter server. Either use service jmeter restart (if you installed JMeter server as a service) or kill current JMeter server thread, and then start it with $JMETER_HOME/bin/jmeter-server

5) if your testscript uses any external files, you have to put those files on each slave node into the same place, which is specified in testscript. From my experience it's safer to specify full path rather than relative (i.e. /etc/testscripts/in/userlist.csv instead of in/userlist.csv), because relative paths are resolved based on the folder from which you are executing jmeter, and not from the testscript location

6) in your testscript configure listeners to save results to files. Again, it is safer to use full paths instead of relative. JMeter will gather results from all slave nodes and save them on master node

7) place your testscript on JMeter master and run it with a command like this:

$JMETER_HOME/bin/jmeter -n -t <test script> -r

- additionally if you want to get some graphs, enable appropriate listener in your testscript and run jmeter with additional parameter -l <filename.jtl>. Note that this creates significant additional load on JMeter nodes, so it probably shouldn't be used when it is not absolutely necessary.

- additionally you can configure JMeter Summarizer to save and show aggregated responce times (one set of results for all samplers) during the run. This blog post does a good job describing how to use it. For me it wasn't of much use because I needed to get aggregated response times separately for each of the many samplers.

8) to see aggregated results, copy files that were generated by listeners from JMeter master to your local machine, run JMeter in GUI mode, open your testscript, choose appropriate listener and point to the file. JMeter will process the file and give you aggregate table as if you ran the script in GUI mode from the beginning. Same works for graph listeners - just point them to a generated *.jtl file.

9) oh, and if you need to stop the distributed test in non-GUI mode, use $JMETER_HOME/bin/shutdown.sh script on the JMeter Master - it will send "shutdown all" to slave nodes.

Hope that was helpful. I know I would be glad to find this when I first googled for "jmeter distributed testing get aggregate response times", or "jmeter master cannot connect to slaves", or "jmeter master doesn't get results back from slaves". :-)