Introvertic ramble on the trap of openspaces and office spaces in general

Hi, my name is Viktoriia, and I’m an introvert.

The weekend before last I spent two awesome days socializing with some of the best testers in New Zealand. After that I spent another three days trying to recover from all the joy. I was exhausted emotionally and physically, and had to spend full Sunday being sick and miserable because that’s how my body reacts to over-socialization - it goes to hibernate. Humans are not built to spend time in hibernate. That got me thinking…

Every day working in the office I get a bit more socialization that I would voluntarily choose to. And then when I get one little spike (like a testing conference), it becomes a butterfly that broke the cammel’s back.

Don’t get me wrong, co-location is awesome and critical for agile teams and all that. But there are also problems that come from the way we implement it (by placing everyone into these huge openspaces), and not only problems relevant to introverts exclusively:

• The constant humming noise. Even if we forget about people who talk loud because that’s how they talk - the typing, and moving, and clicking, and talking, and whatelse is always there. Noise is stress. We even had it as a topic in school and university in Russia: even though human brain is pretty good with filtering out non-changing signals, human-produced complicated noise still makes it to do a lot of work to maintain those filters. Nervous system is always working extra hard just to save you the ability to concentrate. 
• The cold going round. When someone is sick, everyone is sick. Someone is always sick. Sneezing and coughing never really stops. It’s like a kindergarden for IT - if you don’t have iron-made immune system, you are bound to go in and out of colds non-stop. Nothing serious, but pretty annoying. 
• The temperature. Since we are all sharing the same space, we cannot possibly set temperature so that it’s good for everyone. For me it’s always freezing in the office. Judging from the number of people in jackets around, I guess I’m not the only one. 
• The socialization itself. For introverts like me it’s additional stress just to be around this many people all the time. It makes it harder to concentrate, and it means that I’m always under just a little extra bit of stress. Immune system works badly when you are under stress, so that feeds into constantly being in and out of sickbay, which feeds into concentration problems again.
• Commuting. This one applies to working from office in general, not just to openspaces. Every day so much time is being lost on getting from home to office and back. This makes roads overloaded, makes air worse, makes us all spend our precious time doing what really isn't necessary. Would be cool to free up roads for people who actually do have a good reason for being there. In IT in many cases it can be avoided - we have enough collaboration tools to go from 5 days a week working side by side to 1 day when everyone's physically in the office to align their actions and adjust plans as necessary and 4 days when everyone is where they choose to be, being online and connected via internet.
• Multitasking. There have actually been research done* about the efficiency of office workers in different settings. It was shown that even extraverts work more efficiently and more creatively when they have a little bit of privacy (even if that’s a cubicle or a smaller room with just your team - but not the openspace). We also all know that exploratory testing recommends uninterrupted test sessions. The thing is, humans suck in multitasking. We can only really do one thing at a time. We can switch between tasks fast, that’s true, but imagine the overhead! When part of your resources is spent on ignoring the noise (I guess, headphones somewhat help, but in my experience you just get touched a lot when people want to talk to you), part on fighting the cold and part on switching between different tasks (passersby wanting to chat, for example) - you cannot possibly work at your fullest.

While having separate rooms for each team instead of openspace would make things much better, I personally would still prefer to have a choice to work from home. I found that few days when I was sick and worked from home turned out to be no less productive than an average day in the office, and most times even more productive. Always more comfortable.

It would be awesome to have an oportunity to work from home and be judged by results, not by hours in the chair. Especially since many IT companies seem to be already evaluating performance by results. Company I currently work for has a thorough system of logging and evaluating successes and results, and no one really sticks for hours as far as I know. Yet it is not a common practice to allow employees to work from home, aside from emergencies and special cases. I wish it was. One of the reasons I want to go to contracting in few years is to have an opportunity to live out of Auckland in a nice house with good internet and do all the work from there. In my book it beats both living in the center of Auckland to be near office and living outside of Auckland and spending few hours every work day on commuting. I'd rather work 9 hours from home than work 8 hours in the office and spend another hour on getting there and back.

*about research and more, there is an awesome book “Quiet: The Power of Introverts” by Susan Cain. It quotes and references quite a lot of scientific research in the area. I highly recommend it to anyone who’s interested in how people work.

#KWST4, day two

It just so happened, that I was presenting the first ER of the day. Now let me explain: when I'm on any kind of stage, I go autopilot. Very realistic and well-behaved autopilot, but autopilot nonetheless. So I kinda panicked through the presentation and the discussion, and because I wasn't able to take notes, I don't remember much. On the other hand, it was my ER, so I'm gonna write about it in details.

I talked about my experience of dealing with emergencies on my last job in Russia, where I was testing mobile applications for android - think "google maps" but better. I won't mention the name of the company not to attract searching bots and unnecessary attention to this blog post, but if you're interested, it's on my LinkedIn account. Now, this company is amazing in many ways, and it's got huge user auditory and a reputation to maintain. From time to time due to different circumstances I was in a situation when I get a new build, it goes into production in few hours, and full proper retest takes about a week. Bear in mind, I still had enough time to do the testing between the craziness, but the craziness still happend from time to time.

The way I dealt with it was using these three tools:

• Cluster (environments, contexts, test cases).
• Prioritize.
• Parallelize.

The secret is that you cannot possible use these tools if you are not ready. So the solution really is "Be prepared". This is what you need to be able to use those tools:

• Know your environments (what are they, how are they different, which differences matter and why, are there any special reasons to be doing testing on a particular environment).
• Know how your application is being used and how is it likely to be used after the release (popular workflows, past statistics, core users, geek users, marketing effort).
• Know high risk areas (bug-rich areas, understanding of the codebase, functions that make application useless or annoying to use when broken, what changed since last release and since last build)
• Easy to read documented coverage blocks - you should be able to give a coworker who doesn't know your application well a peace of paper and ask to test areas mentioned there. I like to use coverage matrix in a spreadsheet where each row is a short test idea e.g. "make sure the field doesn't let through special characters" or "try two users doing this concurrently".
• Know on a deep level how your application works (architecture in diagrams and mindmaps, dependencies between different parts, server-client protocols).

Few tips on how do you get there:

• Ask questions early - make sure you understand whats, hows and whys about every feature. Or at least whats and whys.
• Document knowledge in diagrams and bullet-point lists: structure!
• Do risks analysis with the team, and use results both in development and in testing.
• Keep up to date with sales/marketing. If they spent a month advertising some shiny new feature you wanna make sure that feature is spotless.
• Gather and analyze usage statistics - there are plenty of the libraries nowadays (google analytics and the likes) that allow you to do it. You wanna know how users are actually using your application, not just how you think they are using it.
• Work close with development - they are irreplaceable source of information. They can also back you up when you are explaining to higher powers why you absolutely need to test this feature, but can release without testing that one.
• Learn differences between environments.

I talked more, of course, but hopefully these lists make sense on their own. To give you an example of how this worked in real life, this is how it usually went. Instead of testing the full scope of the functionality on all representative devices in different contexts and conditions (on the move, in building, in different map setups, with network fluctuations, etc.) I would choose 3-5 devices that are most suitable for the final round (hardware differences, OS versions, popularity, known problems), I would choose a subset of tests, I would cluster my tests by the context. I would e.g. go and take one ride on a bus to the subway and back, carrying three devices with me and switching between them to do all the testing that needs to be done on the move. I would ask a coworker to do the easier part of the testing to increase the scope. I would only test functionality that had a chance to be affected by the new build (because I tested everything else in the previous build), and also core functionality. I would do part of the testing with stubs - e.g. on a test environment or using server simulators like Fiddler - because I understood the impact of that and how to emulate the server properly. I would make sure severe bugs that were fixed and tested few builds back did not come back (happens sometimes when the code is merged from one branch to another in Git).

And after I am done, I would write a letter to a PM (CCed to certain team members) where I would specify which areas are left untested, what are the risks involved, what is my recommendation (safe to release/need more time to test/need to fix known bugs) and why.

There was certainly an interesting discussion after I was done with my ER, but as I said, I don't remember it very well. This is what I do remember.

• Few people were worried that after seeing a project successfully going into production without going through full testing, PM would decide that full testing isn't actually necessary. We did not have this problem in my company, but I think it's a valid point. I can only suggest to educate your managers each release on what are the risks, and why they matter (e.g. what would be the consequences of the failure).
• I think it was Adam Howard who asked whether I used any of the crisis techniques in normally paced testing. I totally did, especially the clustering: I mean, once you figure out a way to do something efficiently, you can't really go back. Well, I am personally too lazy to go back from being efficient. :-) And also, once you went through the emergency, you learn to be ready to the next one, so it's a self-feeding circle, really.
• We talked a bit about how important is co-location, in particular for developers, testers and product owners. I also mentioned, and I think few people agreed, that it's also important to have the right to not be disturbed when you choose to. Sean also mentioned that in their company they dealt with teams being geographically distributed by setting up a constant video thread between locations through huge TVs on the wall.

Next ER was presented by Rachel Carson. She talked about speeding up testing (and the whole project) by switching from Waterfall to "Jet Waterfall" - kind of a Waterfall-Agile hybrid if I understood that correctly: cross-functional teams, co-location, more communication within the team. According to Rachel, testing activities didn't change much, they were pretty good to begin with. But the mindset of testers shifted to the more realistic side: now instead of reporting all the bugs, they started to think whether it's useful to report the bug. Sometimes it was better to talk to development and either get it fixed right away or learn that it will never be fixed with a reasonable explanation. They also started to use the Definition of Done rather than pure guts to drive the testing.

These are some of my notes from that ER and discussion afterwards:

• Important developer's skill - to be able to share information with non-technical people (Rachel).
• When you are forced into defending design decisions, be sure to stay critical about them (Rachel).
• When there are many existing bugs on a project, you can use them as a set of data rather than one-by-one, to extract useful info. E.g. areas where bugs are clustering might need more attention and/or redesign. (Adam and Chris).

The last ER on the KWST was done by Adam Howard. He talked about speeding up the whole process of development on a very challenging bound-to-fail project. Which thanks to Adam and his team did not fail in the end. :-)

This is what I got from it:

• They used visual models (mostly mindmaps) to see/show the big picture - that allowed the team to make a decision to rewrite some functionality rather than to try and fix numerous existing bugs there.
• Fixing all the bugs didn't work because the results of those fixes did't play well together - another reason to use visual models.
• Adam tried to switch testing team to the ways of exploratory testing. To make it easier on them, Adam didn't require them to switch - he just showed the way. Some teams accepted new way, some didn't.
• It's very challenging to change when you have to do it fast. As a result, after the project was out of the door, and Adam and his team went back to Assurity (it was a contract project), remaining testers went back to the old ways. It's not easy to make even good practices stick if they are new.
• Adam and his team partially took BA role on themselves in order to create the big picture view of the project. Some (e.g. Oliver and Katrina) argued that instead of doing it we should rather push BAs to do their job better.

And I believe that's about it. We discussed the exercises from the day before, played some games, said goodbyes and went our ways. It was a lot of fun, and I've learned from every single person who was there. Not everyone is quoted in my two posts (I think I might have missed Ben here, but mentioned him on twitter. Or vice versa), but everyone was brilliant.

I'd like to finish this with mentioning mazing people who made it happen. Thanks to Oliver, Rich, Janice, Nikki (I hope I spelled their names right) and to our sponsors:  @AST_News @SoftEdMan and @AssurityNZ!

#KWST4 day one

I just got back from the awesome two-days long testing workshop KWST4. It was a bit overwhelming for me, with the amount of intense thinking activity and socialization, and I'm still a bit out of it, but nonetheless I'll try to write down all the highlights while I still remember them.

There were at most 17 people in the room, KWST is a pretty small workshop, but that's likely what made it so good. Everyone was participating in discussions, and everyone got to share their experiences and pains. :-) We had four ERs (ER = experience report) on the first day, and three more ERs on the second day. There were also exercise, testing games and a lot of socialization on coffee/lunch breaks. The way ERs worked was someone presented their experience, and then we all had a facilitated discussion around that experience. Main topic of the workshop was "How to speed up testing, and why we shouldn't".

First ER was presented by Sean Cresswell, a test manager from Trademe. Sean shared his experience on how changing the way Risks based testing was perceived in the team allowed to speed up testing. Business unit was already doing what they called Risks analysis, but somehow in the end tests were still prioritized with specification coming first. That led to most important bugs being found closer to the end. Changing the way BU did Risks analysis and reordering tests allowed to change the perception of the testing. It might have taken about the same time as before, but now most critical bugs were found in the beginning of the testing, which gave the impression the testing itself sped up. And come on, we all know that finding serious bugs early is good for so many reasons besides the perception of testing. :-)

Some ideas that came out of this discussion, were:

• Risks analysis should affect priorities in development as well as in testing (I think that was by Rachel Carson, but not sure).
• Plan the work so that everyone (devs, BAs, testers) are busy all the time - as a planning strategy (Oliver Erlewein).
• Instead of step by step howto guides it's sometimes useful to write down lists of questions, answers to which will lead a person through the howto (Andrew Robins).

There were also some pretty heated up discussions around definition of "risk" and such. I personally can recommend Rex Black's book on Risks based testing. Funny enough, when I twitted about the book and jokingly mentioned that it's good despite Rex's evilness (of being involved with ISTQB. ISTQB is evil), I found myself in a twitter argument around Rex's evilness. Huh. Sometimes life is weird.

Moving on, second ER was from Chris Rolls, and he talked about examples of successful and unsuccessful usage of test automation. One of the examples I found interesting was using automation to quickly do regression testing of a web app, after security patch was applied.

Chris also formulated a pretty cool (if you ask me) approach to test automation: when automating existing tests, first transfer test objective. He talked about the common perception of automated tests as step by step repetition of existing manual tests. It's easy to lose the "why" when thinking like this: why are we automating this test, and why do we test it this way. It might very well be that after transferring test objective first, you'll find you can reach that test objective in a different, more efficient way in automation.

Discussion went mostly about automation in general, and testing roles. I liked this idea from Oliver: he said he uses roles to protect people from being pulled away from their main tasks. Example he gave was one of his testers was responsible for developing a testing framework, and Oliver had to protect him from being used as a manual tester when there were lack of those. Apparently, naming the guy "test automation architect" or something like that makes a huge difference to the management. :-)

After the lunch break Thomas Recker presented his ER on coming to do the test automation late in the project. He talked about his experience of being called to "come and automate" when it was too late to influence test design or decisions around testability of the application. He was also asked both to do the testing and to fit some existing list of functionality coverage which didn't play well with how the tests worked. In the end he had to balance actual testing with the bureaucratic part and with the "make it stick together" parts of the job. I think we could all agree that coming into project early and having the opportunity to do test design and such with automation in mind helps with implementing test automation.

There were an interesting discussion around test automation strategies: when does it make sense to automate, and how do you choose what to automate. Few interesting for me things came out of that:

• Andrew suggested to try and see the existing test as a measurement: if it can be seen as such, it's a good candidate to automate.
• Till Neunast mentioned it makes sense to automate testing around business logic when it's implemented in files/modules that change often. No sense in having automated test that always succeeds because part of the application it checks never changes.
• Aaron Hodder insists that you cannot automate testing, because "once it's automated, it's a different 'thing'".
• Oliver described the system they have in his workplace: an internal twitter-like tool that is connected with the test automation framework. It works both ways: you can twit to the system to make it start some tests, and system constantly twits about what it's doing, and what are the results of the testing. Pretty cool!
• Joshua Raine raised a question about differentiating between test automation that helps you here and now and test automation that requires investment now with a potential pay off in future, and about maybe finding some other types of test automation in the same classification.

In the end of the day Andrew Robins presented the ER number four. His topic was "Enabling the team as a technique to speed up testing". Andrew worked in a challenging environment where the company produced not just software, but hardware as well. That meant that test environment would consist of specially made prototypes, and would be extremely expensive. Previously (before Andrew did his magic) that led to a huge bottleneck where testers only had one environment, and it had to be reconfigured for different tests. Reconfiguration took three days. You can see how it's not good to make 40 people wait while the only existing test environment is being unreachable.
The way Andrew solved that and sped up testing significantly was to plan for testing in advance. Months before the testing was to begin, he started on the task to get more environments. Because there was enough time, the company was able to budget and plan for two test environment, which meant far less reconfiguration bottlenecks as well as opportunity to run different tests in parallel.

Important lesson here for me was to plan for testing as well as plan testing. To be fair we are doing it in OrionHealth, but to have it as a consciously formulated strategy is so much better than just do it intuitively!

One interesting part of the discussion that I managed to note down was about other ways of handling limited environment when there is no way to get a new one. The consensus was that one great technique is to stub/emulate parts of the environment you need. I might also add that it might be a good idea to challenge the "we cannot get another environment" notion and go into the cloud. Clouds allow to spin up additional environments pretty fast, secure and really cheap, so unless some specialized hardware is needed, no reason not to use it.

There was also plenty of stuff tracked on twitter, I do recommend to go through #KWST4 hashtag there. And of course discussions were pretty intense, so I didn't have a chance to note down/twit most of it, and probably left out a lot. It would be interesting to compare with other blog posts on the event. :-)

In the end of the day one we also did a group exercise: we had 10-15 minutes to come up with a common answer for each of the given 4 questions. I teamed up with Adam Howard and Nigel Charman. I like the answers our team gave, and I would totally hire anyone who demonstrates kind of thinking we demonstrated. But funny enough it turned out that part of anonymous test managers who assessed our answers called asking reasonable questions "being evasive" and were more concerned about the formatting and punctuation than about the actual content and what skills it showed. Well, what can I say... I wouldn't hire those test managers and wouldn't go work for them either. It also interestingly shows that hiring process in the testing industry is not at its top game in many companies.

Unrelated to the exercise, sadly enough some are still looking for drones instead of testers and wouldn't even consider a really good tester if he doesn't have ISTQB certification, or N years of experience with some tool. Luckily for us, there are also plenty of smart IT companies on the market who hire people to do the job, not to fit in a cell. :-)

Day 2 to follow.

Darkwing Duck about testing

"Darkwing Duck" is my favorite Disney TV series. It's awesome. It has superheroes, cool variety of villains, puns, cultural references, a very active extra smart girl and crazy plots.

Recently I've been watching episode 7 "Dirty money" again first time in few years, and realized: that's a saga about ISTQB vs Context-driven testers! :-D Agent Grizzlikof tries to force DW to do everything by the book, but instead our hero succeeds using his own genuine methods and situational awareness.

And in the end of the episode Darkwing Duck shows us one of the few ways in which ISTQB Foundation textbook can be useful. Yep, that's how it goes - it's only as useful as the paper on which it has been printed. Ignore the letters, use the paper... works for crime fighters!

And more on this topic, I believe that detective work is a very cool area to learn from for a tester. It's a similar business: go into the unknown and find information you seek using scientific methods, reasoning and wide domain and technical knowledge.

P.S.: I'm sure in actual law enforcement structures there are a lot of useful rules, because in their case people get hurt when someone makes a mistake, so inventing the bicycle each time has price too high to pay. And there are plenty of useful rules in testing (e.g. "Ask questions to understand") too, but ISTQB book is not about those nice flexible recommendations-rules. So my metaphor stands. :-)

Fair pay for IT specialists

I have this dream to get my own testing consultancy going in few years. Find good thinkers, find clients in need of smart testing, do the right thing... nothing fancy, just good ol' testing. There are plenty of challenges ahead of me on the way to this dream (it's not even a SMART goal yet), of course, but from time to time I like to think about little details as if I was already there.

Today I got to this: how would I organize payment system in my dream company?

I've been in the industry for almost 11 years now, and everywhere I see more or less similar picture:
- people get hired from outside - vs promoting capable people on the inside;
- a newly hired specialist gets higher salary that someone on a similar position who's already working for the company;
- companies don't work hard enough to keep their people as they consider them easily replaceable.

Now, I don't know about you, but to me it seems unfair and straight dumb. IT is an industry mostly getting its profit from innovation, and people are the basic material making that profit possible. People are important. Software company is not a factory! It can never work as a factory for a very simple reason: factories deal with repetitive tasks, whereas software development always deals with new tasks in a constantly changing environment.
An equally simple fact follows: if you want to be successful, you cannot treat people who work for you as replaceable parts of the system. It means both high expectations and high rewards.

So, going back to the initial problem... this is what I'm thinking to apply to my future business:
- a newly hired specialist will get salary not higher than someone already working in the company who is doing the same job. Which automatically means that salaries for anyone inside the company are at least on par with the market;
- all salary levels will be transparent as well as requirements for moving between those levels;
- structure of profits and spends will be available for anyone inside the company to see and comment on;
- when we need someone to fill a position inside the company with the higher level of competence than there already exists, we would first look for anyone who is ready to upskill and take it as a challenge inside the company.

I strongly believe that if a company wants loyalty, it has to be loyal to its employees first. And it also makes perfect commercial sense to me: the more happy, motivated and high qualified people I gather - the more efficient and high quality work we can do, the more happy clients will be, the more clients will want us. Profit!

So, does anyone think my payment rules would work? Do you think it's fair? Would you like to work in a company with rules like these? Do not hesitate to tell me if you think that's an awful idea.

Lessons learned in non-functional testing

Yesterday I gave a lightning talk at my favorite WeTest Auckland meetup hosted by Assurity. It was a five minutes (okay, maybe 6 minutes) long presentation with the 15 minutes of discussion afterwards. So, for five minutes I had to make it really short, yet making sense.

The main idea I was trying to express is really simple: plan for non-functional quality characteristics right from the beginning.

As an IT graduate I had this assumption when I first started to work as a software tester, that developers would do proper technical and database design according to the task at hand, and then perform testing of the product from the very beginning. In other words, I thought it was obvious that someone would start testing as soon as you get your first requirements from a customer. In reality it wasn't like this at all. Testing usually started at the latest stages of development, which cost projects lots of work hours, effort and money, and often reputational losts. As years passed by, situation seemed to improved. Nowadays most IT companies accept that testing is important, and that it should begin as soon as possible. With one "but" - it only applies to functional testing.

When I switched from functional to performance testing last September, I didn't have that naive assumption about functional testing beeing done the right way everywhere... turns out, I still had even more naive assumption that non-functional testing is done the right way. I haven't done any serious programming in 7 years, yet when occasionally I think about implementing this or that, I still operate by the algorithm I learned in uni: I ask myself questions about a problem I'm trying to solve, and about the future of the software that would solve it. I thought it was obvious: when you write e.g. a web application, you think about bunch of stuff on top of functionality: how would it scale, how would you plug in new functionality, how would you localize it, how would you customize it, what questions will users ask about it, how would you test it, and so on. Sometimes most of the answers would be "it doesn't matter because..." - but important thing is to ask questions and know that for sure.

Well, in real life it happens rarely. I'm not talking about companies like Google or Amazon - surely those guys know what they are doing if only because they had enough experience in the area. But most companies aren't Google. What I see in real life is fire fighting: application is created, then it goes into production, and then you have problems in production, which are extremely expensive to fix. Some issues are obvious: e.g. security breaches and performance and/or scalability breakdowns. Some issues are more subtle: e.g. bad supportability would mean a lot of time and effort spent by field agents (support line, or operators, or whatever you call people who talk with unhappy users). Bad maintainability would mean additional problems when you need to install updates or reconfigure something in live system. Bad testability would mean more time necessary for testing e.g. new release, and potentialy worse quality because some areas testers won't be able to reach. Bad initial usability will bite you in the ass when you decide to implement shiny new GUI which would be different from the existing one, because users will be resistant to changes (remember MS Office 2003 -> 2007 outcry). And what's even worse, to fix non-functional issue often requires to implement fundamental architecture changes.

To summarize, non-functional quality is important, it costs a lot to fix, it isn't being given enough attention.
So, the question is, if you work in one of those companies, what can you do?

As part of the team, regardless of your role, you can communicate and underline the importance of the following:

• When doing tech design for a new application or a new feature, think about the future. What would happen with your application in a year? What new functionality it might have? What would be the environment and the load (e.g. number of users per hour)? Are you gonna store any sensitive data? Would you need to create additional applications (e.g. mobile) to coordinate with it? What additional data you might need to store? How would you localize/customize it if necessary? How often would you need to release a new version?
• Do risks analysis. Always do risks analysis - better be prepared for the trouble that might come your way.
• Involve specialists in specific fields if your team lacks those: there are consultancies, that can review tech design, or do security testing, or performance testing, for you.
• Start the actual testing as soon as possible - not only functional testing, but non-functional too. On early stages it would look differently from how it loos on later stages: e.g. it makes little to no sense to do a full performance testing for a half-made release, but it is possible to do performance testing of existing separate components, especially of the core ones. It pays off because the earlier you fix the issue, the cheaper it is.
• Treat non-functional issues fair. If the team doesn't fix e.g. testability issues in time, you will pay for it later with much more complicated testing and higher price for building testability into the project. If performance or security issues are not being treated with respect because "it still works, no one will notice it's slow/insecure", you will pay with your reputation and money when it manifests on production. And if your product is a success, it will manifest in production sooner or later. #too_much_painful_experience

After I did the talk, one of the testers on the meetup asked if all of this makes sense for startups and such - she implied that in the beginning you don't want to spend your time thinking about all this stuff, because you are under pressure to deliver fast. This is another point of view, but I strongly disagree with it. The only case in which you can forget about quality is when you are making proof of concept, then throw it out the window and start anew. If you are gonna use the codebase of your first release further, it pays off greatly to spend few extra hours (and in the very beginning you would only need few extra hours) on analysing the future of your application and doing your tech design (database design and application design) properly.

Remember: there is nothing more permanent than a temporary solution.

Slides for the presentation are shared here:
https://docs.google.com/file/d/0Bxi4eMT3I97ea3luRU9nemRpODg/edit

SFIA Skills framework

There is this cool skills framework which helps you in structuring your professional skills and knowledge, and our company adopted it for internal use. So when I recently decided to review what I know and where are the biggest gaps in my professional knowledge (which I love to do from time to time), I got really inspired.

It probably doesn't work for everyone, but for me it's a perfect fit. I use my imagination a lot to get the spirit of each skill level and apply it to the exact area of expertise I'm looking at (in this case - performance testing). It works kinda like Satisfice Heuristic Test Model: you don't get any instructions - instead you get directions to think about and tools to categorize and/or quantify this seemingly unquantifiable ball of knowledge, skills and experience you keep in your head.

So long story short, after I assessed myself against the next competency level for my job (that is, going from Intermediate to Senior), I got an impressive todo list for self-development. It's funny how I sometimes miss doing functional testing, but at the same time I'm glad I switched to performance testing, because it gave me so much motivation for learning. If I want to get as good in performance testing as I am in functional, I need to learn a lot, and I need to get all sorts of experience.

When I learn, I like to do it in big leaps of reading/trying, but I also like to switch between several different topics and give myself time to process new information on the background. This being said, currently my focus is Java performance. That includes both JVM tuning for performance and knowing how to write highly performant code in Java (algorithms, data structures, concurrency). I feel like when I'm finished I might be better in programming than a lot of actual Java developers. Or maybe not. Anyway, that would be awesome. I like keeping the balance between testing and developing skills shifting. And it's cool that some jobs actually require you to have it this way. E.g. Google's test engineers, who are pretty much really good highly technical testers. Or performance engineers. Or software developers developing testing frameworks.

No old tests in regression testing?

Just the other day I was in Assurity, giving my favorite talk about efficient testing (based on "Rapid Software Testing" course and my personal experience). One of the arguments I made (after Michael Bolton in his time) was "use new tests other than old ones". Then someone from the listeners asked: but how do you do regression without using old tests? It got me for a second, because that was an awesome question, a new way to look at it. I mean, no one ever asked that in my presence. When you get question like that, it means that audience listened carefully, processed and understood what you were saying. It also means you get a chance to look at something from a new point of view.

So, I thought for a second, and then I realized: yep, I'm totally doing regression testing without using old tests. And that's how I do it: I don't use scripted test cases at all. What I use is a list of test ideas that structures my testing and serves as a backbone, as a starting point for my testing. So each time I do regression testing, I do it differently, because I don't have steps to follow. Each time I have to think: how would I test this thing, knowing what I know about it right now? So it's always a new test, and it's never out of date.

Another awesome thing was that after my answer people who asked me this realized that they do practically the same thing! They do have scripted testcases, but they never stick to them, and often they don't even read them. So they are actually doing new testing each time. "So why do you need steps written down?", - was my next question. I hope it will do some good. ;-)

I absolutely love it when I can actually say something useful to another tester and understand it immediately that it was useful. Far too often I get this feeling that a person listens to be polite and then goes about their old business, and maybe even uses some of my insights - but I never get to know about it.

Art of balancing

I love how in the process of making and releasing software you (as a team) constantly have to balance technical and business risks. You can't have it all, you most likely will have to go to production with a list of known issues, because if you don't do it, you might lose a client or a market. World moves fast and doesn't wait for anyone. I find that some testers and developers don't really get it, and they insist on blocking the release until all major (in their opinion) issues are resolved.

I say: don't block a release, but make sure that product owner knows of all the technical risks you are aware of, their likelyhood and severity, and the risk mitigation plan (which btw might include install/user guide instructions for workarounds). Let product owner decide if this risk is worth being taken. It might cost more to get rid of the issue than it would cost to fix it even if it indeed appears in production.

As a technical specialist I might strive for perfection, but as a problem-solver I always come back to practical solutions.

Maturity of testing process and all that jazzz

Today I spent some time discussing this thing that the other person called "Holistic Testing Maturity model". Now, I probably didn't understand everything he was saying properly, but that got me thinking, and I want to share. I'm not saying that I present that other person's position true and fair, it was just a starting point for my thoughts.

What I got from the conversation is the idea that you want to have a separate testing process maturity model, that only talks about finding and reporting bugs, but not about dealing with them. So you can be very mature according to this model if testing team on your project is all shiny - but at the same time you may have an awful product because you don't actually do much with the information your shiny testers provide you. You will still be able to claim you have a mature testing process. *facepalm*

Now, maybe I spent too much time as a test lead, business analyst and product owner (yep, I wasn't always hands-on testing), but that just doesn't make any fracking sense to me. Excuse my French. Of course it is nice to have a good process of finding and reporting bugs, but what goal do we achieve with that? To have a good testing? But (a) it's not a goal (it's means), and (b) this is not a good testing. At least not what I call good testing.

Thing is, testing doesn't exist in vacuum. It exists to add value for the overall product: to provide information that helps to make product (and process of creating that product) better. If that information is not being used - why should it even exist? If you create perfect bugs, but they stay forever unfixed - how can this be called a mature testing process? Testing isn't about finding bugs, it's about finding useful information (part of that information are bug reports, surely), so if we cross out the word "useful", that's a crappy testing process from my point of view.

Be sure, I am not saying that product quality is a responsibility of solely testing team. Neither am I saying that having a nice and clear process of getting to a point where information is provided (but not yet used) is a bad thing. What I'm saying is: a process should be there for a reason, so if improvements on the process don't bring value to the end product, they are kind of useless.

To be even more clear I'll just list few examples from the top of my head of what I consider a valid reason/goal for an improvement of testing process:

• to test more features in breadth and depth at the same (as currently) amount of time (either less work hours or more value in the same total number of hours - reduces cost of testing -> valid business goal);
• to involve testing team on the earliest reasonable stage (reduces overall time needed to complete a project because some stuff is being done in parallel + the earlier problem is found, the cheaper it is to fix -> valid business goal);
• to improve reporting on test results (gives product owner and developers necessary information faster and in better shape -> enables product owner to make business decisions in time + enables developers to fix issues faster -> saves everyone time and money, valid business goal);
• to reduce the time for onboarding new team members (less time means project can start benefiting from new addition faster; and it also improves stability of the team, which enables product owner to make proper delivery estimations -> valid business goal);
• and so on and so forth.

It should all trace back to business value, to a reason why we create the product at all (and no, it's not to "make money" - it's to solver problems our clients have in a timely manner and with satisfying quality). Think of it as of agile vertical stake: until the feature is released, nobody cares how much effort you spent on it. If it's not released, you don't get payed. If testing didn't bring value - it doesn't matter how shiny it is. You don't want to do a process for the sake of doing a process (and if you think I'm wrong, you've probably been ISTQBfied).

And if you are making a maturity model for testing that only goes until the bug is reported - awesome, just don't call it "testing maturity model", call it "maturity model of reporting bugs" (and think why do you even want a separate maturity model for that). Because that's what it's about: reporting bugs. Testing goes beyond that. A good tester enables the team to make a better product. Mature testing process would consider at the very least:
- keeping good (bug not excessive) testing documentation (from notes, to test strategy, to bugs, to test results);
- having a process of onboarding new test team members;
- reporting bugs;
- following up with the bugs (triaging/estimating/confirming the fix/updating test data and test ideas list/etc.);
- acceptance testing: requirements and conditions on which the pass is granted, and clarification on what the "acceptance testing passed" would mean;
- approach to testing (when do we start testing, what do we do, how do we do it, when do we stop doing that, how we report on results and what do we do after);
- team collaboration stuff (by what means, with who and how often do we exchange information);
- what is the goal of having testing on the product at all.

So, just to reiterate, from my point of view a mature testing process eliminates the situation when nothing is being done with the results of testing. If nothing is being done - you don't need to spend testers' time on the project, you already know that it's crap.

Auckland WeTest meetup, Mobile testing

Today I was attending my favorite event for testers, WeTest Auckland - a meetup, hosted by Assurity, where testers and people interested in testing share their experience and discuss challenges that arise in testing. That was my third time, and I'm definitely coming again, because each time I go there I have so much fun, and I also get inspired, and learn stuff, and meet cool people who I would've never met otherwise... gosh, it sounds like I'm doing an advertisement because I was told to... but that's totally not the case, I'm genuinely enthusiastic about this thing. I love the atmosphere and the facilitated discussion format. I am quite a shy person actually, but on these meetups I feel comfortable talking about testing. More comfortable in fact than in my own company, where most of the time I don't feel that people understand what I'm trying to communicate, so I get nervous, and my English gets awful, and I end up just feeling stupid. Which probably happens because in my company we talk about products a lot (and they are crazy complicated as a whole ecosystem, and I don't really know them that well as of yet), whereas on meetups we talk about testing in general... aaaanyway, back to the meetup.

First ever WeTest was about automation, second one was about exploratory testing (with me presenting actually, which was a special brand of fun because of how enthusiastic and accepting the audience was), and today we talked about mobile testing. Morris Nye did an awesome job on presenting his experience report on mobile testing and automation in that area, and as always there were people asking questions and sharing their experience. Even though I spent most of the last 4 years doing mobile testing, I learned few things today. Mostly just the names of the tools (or rather the fact that they exist and the ways they behave), but still. And of course it was awesome to meet new people (which I'm usually terrified of) and hang out with those I knew already. Can't express how much I value this opportunity. Way to go, Assurity!

Hopefully from now on the WeTest Auckland meetup will become a monthly event (as opposed to bimonthly before now). :-)

Automation in functional testing

When people hear the word "automation" near the word "testing" they usually think about automated testing, meaning "automatic execution of test scripts". But while automated testing is a big awesome area of interest, it's not the only way you can use automation to aid testing. I'm saying nothing new here, probably, that's just a topic that I find fascinating and practical. Soooo... what else can you do automatically to make your life easier if you are a software tester?

I shall probably start with a rude description of the stuff that a functional tester does more or less often in his or her daily job (this is based on my personal experience, of course), and then I can think of what of that it would make sense to automate.

As a tester I:
- analyze requirements for the product;
- create test strategy for the product and test design for separate features of the product;
- set up testing environment;
- prepare test data;
- test according to test strategy and test design;
- do exploratory testing outside of test design;
- do regression testing and bug fixes verification testing;
- keep notes during any kind of testing;
- report bugs;
- communicate with the team to resolve questions and keep up to date on what's going on on the project;
- report and analyse testing results.

Going top to bottom, I can see few obvious candidates for automation outside of automated testing:
1) setting up test environment
There are plenty tools that can help with that (especially if the environment is Unix/Linux based), from bash scripts to Puppet, to cloning VMs, to restoring database to a snapshot, to all-around continuous delivery systems that can do stuff all the way from building a new build from source code to deploying it on a newly provisioned environment, to running predefined set of automated tests and reporting on results.

At the very least setting up a test environment for long-living project should be documented and aided with prepared configuration files, so that it takes the minimum amount of time for anyone to pick up the instruction and create a fully operational test environment.

2) generating test data
This is especially important for performance testing, but functional testing can also benefit from having a way to automatically generate necessary amount and variety of test data instead of doing it manually each time. Or at least if test data has been generated manually once, it's a good idea to make a snapshot of that data in order to be able to return to that state (or recreate it on new environment) whenever needed.
Excel and similar tools are pretty cool for preparing randomized test data in csv files that can then be used by tools like JMeter, or SoapUI, or any scripting language to actually populate data storage. Sometimes it's easier to do it via emulation of user activity (e.g. with JMeter and HTTP requests), and sometimes it's easy enough to just go straight to the database and do some SQL scripting. If speed is the priority, I recommend to start from the lowest level with the less amount of processing logic and only go up if adding test data on that level is overly complicated. On the other hand, emulating user actions in order to add data has a side effect of testing those actions in the process.

3) generating/using templates for test documentation
This includes test strategy document, test design documents, bug reports and test run notes/sessions (or whatever you use to keep notes on the testing). All these documents have structure (or at least they are supposed to have it), so they would probably have some content that doesn't change from document to document. Prepopulate it to save time on actually typing that stuff! Also it can be useful to generate test runs/sessions descriptions from the test design documents automatically.

4) keeping notes during testing
It highly depends on the application you are testing, but sometimes it's useful to have a tool that sits on the background and records some information for you.
Some examples would be:
- automatic screenshot-maker set to take screenshots every 10 seconds or so;
- a program to record system events - I developed that about 6 years ago for testing few Windows applications, not sure if there are any tools or any wide demand for these tools;
- logs built in to the application. Logs are great and highly useful! Make sure your dev team is aware of that, and that they implemented few levels of logging (at least Info, Warning and Error).

5) generating documents from testing notes
If you keep testing notes in a specific format, they can be later automatically parsed and used for generating other documents: e.g. bug reports, reports on test results, templates for future testing sessions or list of test ideas.

6) analysing application logs
Sometimes it is necessary to monitor application and server logs during testing and to analyse them. When you are doing it for a single event, no reason not to use just tail -f, but if the monitoring is more or less regular task, it's worth while to set up some monitoring system to do most of the job for you. Splunk or Logstash-ElasticSearch-Kibana would do splendidly. They also allow to store and access historical data, so you can return to a point of time when you noticed fault or weird behavior and see what was going on in that exact moment. Isn't that cool?
Also in case of JMeter results you sometimes get results in a for that requires manual processing before they can be used. For example, aggregate report would contain every HTTP sampler, while you might only need few. It's easy enough to write a macro using regexp for Search/Replace in Notepad++ to do this for you in matter of seconds.

Less obvious would probably be:
1) aid in test design
For example, a macro that goes through product requirements and colors all verbs one color and all adjectives a different color. It's useful with some of test design techniques to have this colorization, but usually you do it manually.

2) script to go to a certain place in the application
It makes sense when you are testing a stateful application for example multypage registration applications with lots of fields, or a game where you need to do certain amount of actions to get to a certain location/level. If there is no backdoor available that leads directly to a point where you need to be (e.g. page 5 to test how fields on that page work together), you might lose time to get there. Or you might use automated test scripts to emulate getting there, and just do what you are really after - testing.

In the end I'd say that any repetetive action is a candidate for automation, if you don't absolutely need a human being to do it, and if it's repetetive enough. I.e. don't automate something if you are only gonna do it few times, and/or if automating it is a complicated time consuming task that just isn't worth it. In every other case - why not make your (and everyone else's) life easier?

(update from the comments):
3) tools to gather usage statistics in production
Example would be Sentry. I'm sure I also used a tool named "captain" for android applications, but I can't find it right now. Anyway, these tools can help to deal with client bugs, so you won't have to actually try and get steps for reproducing the issue froma client - you will have those steps gathered and reported automatically.

About ISTQB certification

ISTQB certification sucks on so many levels that I can't even bring myself to be nice about it.