Lessons learned in non-functional testing

Yesterday I gave a lightning talk at my favorite WeTest Auckland meetup hosted by Assurity. It was a five minutes (okay, maybe 6 minutes) long presentation with the 15 minutes of discussion afterwards. So, for five minutes I had to make it really short, yet making sense.

The main idea I was trying to express is really simple: plan for non-functional quality characteristics right from the beginning.

As an IT graduate I had this assumption when I first started to work as a software tester, that developers would do proper technical and database design according to the task at hand, and then perform testing of the product from the very beginning. In other words, I thought it was obvious that someone would start testing as soon as you get your first requirements from a customer. In reality it wasn't like this at all. Testing usually started at the latest stages of development, which cost projects lots of work hours, effort and money, and often reputational losts. As years passed by, situation seemed to improved. Nowadays most IT companies accept that testing is important, and that it should begin as soon as possible. With one "but" - it only applies to functional testing.

When I switched from functional to performance testing last September, I didn't have that naive assumption about functional testing beeing done the right way everywhere... turns out, I still had even more naive assumption that non-functional testing is done the right way. I haven't done any serious programming in 7 years, yet when occasionally I think about implementing this or that, I still operate by the algorithm I learned in uni: I ask myself questions about a problem I'm trying to solve, and about the future of the software that would solve it. I thought it was obvious: when you write e.g. a web application, you think about bunch of stuff on top of functionality: how would it scale, how would you plug in new functionality, how would you localize it, how would you customize it, what questions will users ask about it, how would you test it, and so on. Sometimes most of the answers would be "it doesn't matter because..." - but important thing is to ask questions and know that for sure.

Well, in real life it happens rarely. I'm not talking about companies like Google or Amazon - surely those guys know what they are doing if only because they had enough experience in the area. But most companies aren't Google. What I see in real life is fire fighting: application is created, then it goes into production, and then you have problems in production, which are extremely expensive to fix. Some issues are obvious: e.g. security breaches and performance and/or scalability breakdowns. Some issues are more subtle: e.g. bad supportability would mean a lot of time and effort spent by field agents (support line, or operators, or whatever you call people who talk with unhappy users). Bad maintainability would mean additional problems when you need to install updates or reconfigure something in live system. Bad testability would mean more time necessary for testing e.g. new release, and potentialy worse quality because some areas testers won't be able to reach. Bad initial usability will bite you in the ass when you decide to implement shiny new GUI which would be different from the existing one, because users will be resistant to changes (remember MS Office 2003 -> 2007 outcry). And what's even worse, to fix non-functional issue often requires to implement fundamental architecture changes.

To summarize, non-functional quality is important, it costs a lot to fix, it isn't being given enough attention.
So, the question is, if you work in one of those companies, what can you do?

As part of the team, regardless of your role, you can communicate and underline the importance of the following:

• When doing tech design for a new application or a new feature, think about the future. What would happen with your application in a year? What new functionality it might have? What would be the environment and the load (e.g. number of users per hour)? Are you gonna store any sensitive data? Would you need to create additional applications (e.g. mobile) to coordinate with it? What additional data you might need to store? How would you localize/customize it if necessary? How often would you need to release a new version?
• Do risks analysis. Always do risks analysis - better be prepared for the trouble that might come your way.
• Involve specialists in specific fields if your team lacks those: there are consultancies, that can review tech design, or do security testing, or performance testing, for you.
• Start the actual testing as soon as possible - not only functional testing, but non-functional too. On early stages it would look differently from how it loos on later stages: e.g. it makes little to no sense to do a full performance testing for a half-made release, but it is possible to do performance testing of existing separate components, especially of the core ones. It pays off because the earlier you fix the issue, the cheaper it is.
• Treat non-functional issues fair. If the team doesn't fix e.g. testability issues in time, you will pay for it later with much more complicated testing and higher price for building testability into the project. If performance or security issues are not being treated with respect because "it still works, no one will notice it's slow/insecure", you will pay with your reputation and money when it manifests on production. And if your product is a success, it will manifest in production sooner or later. #too_much_painful_experience

After I did the talk, one of the testers on the meetup asked if all of this makes sense for startups and such - she implied that in the beginning you don't want to spend your time thinking about all this stuff, because you are under pressure to deliver fast. This is another point of view, but I strongly disagree with it. The only case in which you can forget about quality is when you are making proof of concept, then throw it out the window and start anew. If you are gonna use the codebase of your first release further, it pays off greatly to spend few extra hours (and in the very beginning you would only need few extra hours) on analysing the future of your application and doing your tech design (database design and application design) properly.

Remember: there is nothing more permanent than a temporary solution.

Slides for the presentation are shared here:
https://docs.google.com/file/d/0Bxi4eMT3I97ea3luRU9nemRpODg/edit

How to make JMeter save results to a database

Our team is responsible for performance testing in the company, and JMeter is one of our major tools. Recently we understood that to make our job easier we need to save aggregated test results to a database instead of a file. There are different use cases for that, the most important being to run periodic automatic performance testing and graph the results to see if a new build dropped in performance. You can do it manually, sure, but saving results to a database makes it so much easier.

Turns out there is currently no plugin for JMeter (that we know of) that would allow you to do that. So a collegue of mine made a plugin to do just that. Then I got interested, and decided to do it a bit differently, and found a problem in his plugin, and... to make the long story short now we have two slightly different plugins that save aggregated results to a database.

I'll probably try and share source code in JMeter's Github once I'm not too lazy to figure out how to do that, but for now I just want to share few things I learned on the way. And boy, was it a harsh way. JMeter has the most unfriendly API I've ever seen (though to be fair I haven't seen much), it looks crazy, and I couldn't find answers to my questions in the internet when I needed them, so here you go. Maybe the next person who decides to make a fancy visualizer will benefit from this post.

So, you decided to write a visualizer that does something more than just calculating statistics differently or showing the data you already have in a prettier graph. Here's some tips that go beyound common sense and general ability to write code in Java and google.

• Do not do any logic in your visualizer class, because in non-GUI mode JMeter will behave like your class doesn't exist, so you will get no logic whatsoever. Think about standard Aggregate report: in non-GUI mode all you can do is save every and each request to a file, you cannot just get an aggregated table. That's because aggregation happens in the visualizer code, and visualizer doesn't get initialized in non-GUI mode.
• In non-GUI mode samples are being processed by ResultCollectors. This is where you want to put all your logic. To do that you need to implement two classes and integrate them with each other:
• implement your visualizer (extend AbstractVisualizer)
• implement your result collector (must extend ResultCollector, or it won't get started)
• override "createTestElement" method in the visualizer and create your result collector. You must also override "modifyTestElement" and "configure" methods and make sure the proper constructor of your result collector is called. See the example 1 for "createTestElement" in the end of the post. If you don't do it, JMeter won't know that your two classes are connected. Basically this is where you say: okay, my GUI shows information which really comes from this TestElement, so please create this TestElement even when you skip the GUI.
• In your result collector override the "sampleOccured" method and put all your aggregating results logic here. You also probably want to call "super.sampleOccured" in the beginning - this way you will get the standard logic (check if sample is okay, send it to visualizer) as well as your new one.
• If you need to get settings from the GUI, make sure they are saved as properties of your result collector (and not as properties of your visualizer).
• Keep in mind that you can only access properties after the constructor. Yep, cannot to it in constructor even if your result collector is fully initialized at that point. JMeter runs the "give test elements their properties from the jmx file" process only after it initialized those elements.
• If you want to use JMeter variables (e.g. you have a field "Test run description", and you want to put some context dependent info there), be aware: you only get access to parsed variables in result collector, but not in the visualizer. In other words, getPropertyAsString will give you "${host}" if called from visualizer, though it will give you "192.168.7.15" if called from result collector for the same property.
• If you want to know active number of threads (load level) you will have to calculate it. See example 2 below for what to put in the "sampleOccured" method in result collector or in the "add" method in visualizer to do that.

Promised examples.

example 1:

@Override

public TestElement createTestElement() {

            if (collector == null || !(collector instanceof DBResultCollector)) {

                collector = new DBResultCollector();

            }

            modifyTestElement(collector);

            return collector;

}

example 2:

loadLevel = Math.max(res.getAllThreads(), loadLevel);

//where res is SampleResult

No old tests in regression testing?

Just the other day I was in Assurity, giving my favorite talk about efficient testing (based on "Rapid Software Testing" course and my personal experience). One of the arguments I made (after Michael Bolton in his time) was "use new tests other than old ones". Then someone from the listeners asked: but how do you do regression without using old tests? It got me for a second, because that was an awesome question, a new way to look at it. I mean, no one ever asked that in my presence. When you get question like that, it means that audience listened carefully, processed and understood what you were saying. It also means you get a chance to look at something from a new point of view.

So, I thought for a second, and then I realized: yep, I'm totally doing regression testing without using old tests. And that's how I do it: I don't use scripted test cases at all. What I use is a list of test ideas that structures my testing and serves as a backbone, as a starting point for my testing. So each time I do regression testing, I do it differently, because I don't have steps to follow. Each time I have to think: how would I test this thing, knowing what I know about it right now? So it's always a new test, and it's never out of date.

Another awesome thing was that after my answer people who asked me this realized that they do practically the same thing! They do have scripted testcases, but they never stick to them, and often they don't even read them. So they are actually doing new testing each time. "So why do you need steps written down?", - was my next question. I hope it will do some good. ;-)

I absolutely love it when I can actually say something useful to another tester and understand it immediately that it was useful. Far too often I get this feeling that a person listens to be polite and then goes about their old business, and maybe even uses some of my insights - but I never get to know about it.

Automation in functional testing

When people hear the word "automation" near the word "testing" they usually think about automated testing, meaning "automatic execution of test scripts". But while automated testing is a big awesome area of interest, it's not the only way you can use automation to aid testing. I'm saying nothing new here, probably, that's just a topic that I find fascinating and practical. Soooo... what else can you do automatically to make your life easier if you are a software tester?

I shall probably start with a rude description of the stuff that a functional tester does more or less often in his or her daily job (this is based on my personal experience, of course), and then I can think of what of that it would make sense to automate.

As a tester I:
- analyze requirements for the product;
- create test strategy for the product and test design for separate features of the product;
- set up testing environment;
- prepare test data;
- test according to test strategy and test design;
- do exploratory testing outside of test design;
- do regression testing and bug fixes verification testing;
- keep notes during any kind of testing;
- report bugs;
- communicate with the team to resolve questions and keep up to date on what's going on on the project;
- report and analyse testing results.

Going top to bottom, I can see few obvious candidates for automation outside of automated testing:
1) setting up test environment
There are plenty tools that can help with that (especially if the environment is Unix/Linux based), from bash scripts to Puppet, to cloning VMs, to restoring database to a snapshot, to all-around continuous delivery systems that can do stuff all the way from building a new build from source code to deploying it on a newly provisioned environment, to running predefined set of automated tests and reporting on results.

At the very least setting up a test environment for long-living project should be documented and aided with prepared configuration files, so that it takes the minimum amount of time for anyone to pick up the instruction and create a fully operational test environment.

2) generating test data
This is especially important for performance testing, but functional testing can also benefit from having a way to automatically generate necessary amount and variety of test data instead of doing it manually each time. Or at least if test data has been generated manually once, it's a good idea to make a snapshot of that data in order to be able to return to that state (or recreate it on new environment) whenever needed.
Excel and similar tools are pretty cool for preparing randomized test data in csv files that can then be used by tools like JMeter, or SoapUI, or any scripting language to actually populate data storage. Sometimes it's easier to do it via emulation of user activity (e.g. with JMeter and HTTP requests), and sometimes it's easy enough to just go straight to the database and do some SQL scripting. If speed is the priority, I recommend to start from the lowest level with the less amount of processing logic and only go up if adding test data on that level is overly complicated. On the other hand, emulating user actions in order to add data has a side effect of testing those actions in the process.

3) generating/using templates for test documentation
This includes test strategy document, test design documents, bug reports and test run notes/sessions (or whatever you use to keep notes on the testing). All these documents have structure (or at least they are supposed to have it), so they would probably have some content that doesn't change from document to document. Prepopulate it to save time on actually typing that stuff! Also it can be useful to generate test runs/sessions descriptions from the test design documents automatically.

4) keeping notes during testing
It highly depends on the application you are testing, but sometimes it's useful to have a tool that sits on the background and records some information for you.
Some examples would be:
- automatic screenshot-maker set to take screenshots every 10 seconds or so;
- a program to record system events - I developed that about 6 years ago for testing few Windows applications, not sure if there are any tools or any wide demand for these tools;
- logs built in to the application. Logs are great and highly useful! Make sure your dev team is aware of that, and that they implemented few levels of logging (at least Info, Warning and Error).

5) generating documents from testing notes
If you keep testing notes in a specific format, they can be later automatically parsed and used for generating other documents: e.g. bug reports, reports on test results, templates for future testing sessions or list of test ideas.

6) analysing application logs
Sometimes it is necessary to monitor application and server logs during testing and to analyse them. When you are doing it for a single event, no reason not to use just tail -f, but if the monitoring is more or less regular task, it's worth while to set up some monitoring system to do most of the job for you. Splunk or Logstash-ElasticSearch-Kibana would do splendidly. They also allow to store and access historical data, so you can return to a point of time when you noticed fault or weird behavior and see what was going on in that exact moment. Isn't that cool?
Also in case of JMeter results you sometimes get results in a for that requires manual processing before they can be used. For example, aggregate report would contain every HTTP sampler, while you might only need few. It's easy enough to write a macro using regexp for Search/Replace in Notepad++ to do this for you in matter of seconds.

Less obvious would probably be:
1) aid in test design
For example, a macro that goes through product requirements and colors all verbs one color and all adjectives a different color. It's useful with some of test design techniques to have this colorization, but usually you do it manually.

2) script to go to a certain place in the application
It makes sense when you are testing a stateful application for example multypage registration applications with lots of fields, or a game where you need to do certain amount of actions to get to a certain location/level. If there is no backdoor available that leads directly to a point where you need to be (e.g. page 5 to test how fields on that page work together), you might lose time to get there. Or you might use automated test scripts to emulate getting there, and just do what you are really after - testing.

In the end I'd say that any repetetive action is a candidate for automation, if you don't absolutely need a human being to do it, and if it's repetetive enough. I.e. don't automate something if you are only gonna do it few times, and/or if automating it is a complicated time consuming task that just isn't worth it. In every other case - why not make your (and everyone else's) life easier?

(update from the comments):
3) tools to gather usage statistics in production
Example would be Sentry. I'm sure I also used a tool named "captain" for android applications, but I can't find it right now. Anyway, these tools can help to deal with client bugs, so you won't have to actually try and get steps for reproducing the issue froma client - you will have those steps gathered and reported automatically.