Mittwoch, 26. Juli 2017

Pathway Exploratory Testing

Katrina Clokie has a very good section on her blog she calls “pathways”, which are curated link lists and training exercises you can use to get started with certain software testing aspects. Katrina offers these pathways for Security Testing, Mobile Testing and many other topics. You should definitively check them out.

When a colleague asked me to send him an introduction into Exploratory Testing a while ago I created a list of useful blog posts I think are helpful to get started with Exploratory Testing. His answer was something along the lines “oh my, this is a complete pathway”. I think it was not, but it was not that far off from being one.

Therefore I added some more links and thought of some exercises to evolve this into a proper pathway. In the meantime, more people asked me for information about exploratory testing so from now on I can answer with a link and an offer for a joined coffee.

Just as Katrina's pathways, this is a rabbit hole: A lot of the articles below link to other articles, which will forward you to even more sources. I hope this will be a fun ride for you.

STEP 1: What is Exploratory Testing?

The questions often begin with what Exploratory Testing actually is. And perhaps, more importantly, why would you use it in the first place? Sometimes people have some negative bias as if exploratory testing is just toying with the software and can only be an addition to test case based approaches. Sometimes people seemingly have heard great things and now want in on this cool "new" method. Here are a few articles, that describe what Exploratory Testing is and how you can use it in your project:

Exercise: If you are currently not using exploratory testing techniques, but are testing based on test cases, watch your next test execution very carefully: Are you really performing only the specified steps and checking the specified expected results or are you doing more? Are you looking “left and right”? How do you tell people what you saw when you moved away from the script? How do you remember it yourself? Do you take notes that go beyond the test steps of your test cases? Are your cases/steps answering questions about risks?

STEP 2: How do I manage my Exploratory Testing?

Session-Based Test Management is the most popular method to introduce Exploratory Testing in projects. It helps to make Exploratory Testing manageable and reportable, especially towards people outside of the testing team. Michael Bolton used the metaphor of putting the clouds, that are bursts of performed Exploratory Testing, into boxes so you can count those boxes. 

Here are some articles about and experience reports on Session-Based Test Management:

STEP 3: How do I find test charters?

A topic, that troubles a lot of people starting out with Exploratory Testing and Session-Based Test Management is generating test charters in comparison to generating test cases. Most of them are used to derive test cases from any form of specification document, which verifies that the software works “as specified", which is not necessarily “as intended”. 
But how to create missions for exploration in addition, or instead of, test cases? 

A good idea is to think about the goal you want to achieve with your respective testing mission, Simons examples for different charter types can help you with this. I listed some more models and texts, which can help you in finding test charters:
Exercise: Chose one of the above approaches and write down down three test charters for the software you are currently working on. You can use the charter template provided in the link from STEP 2.

STEP 4: How do I come up with test ideas?

Once you have your testing mission for a session down, the interesting part is having ideas of what to actually test to serve your mission. I have seen testers, who suffered from some form of writer's block when they started their first test sessions. Especially when they were not involved with designing test cases and only executed them in the scripted testing world they knew.

Fortunately, there are a lot of methods and tools out there to help you come up with test ideas. I just listed a few down there, which you will hear or read often when deeper exploring the Exploratory Testing world, or which help me specifically. If you really, really want to go deep into this topic follow the link to Erik Brickarp's text below and be blown away. 
Exercise: Remember the test charters from the last STEP? Now it is time you perform these test sessions. As a first step, you don’t need to test the whole 90 minutes as described in the linked introduction tests, start with 30 minutes if you like to.

STEP 5: I've got to the end of my session. Now what?

After one or more sessions a Debriefing has to take place. I met several people, who skipped on these and asked: "Do I really need that Debriefing?". Short answer: yes. Long answer: Although a lot of people find it tedious to add yet another meeting to their schedule, the Debriefing is still absolutely crucial. It helps to identify new test charters, spread the testing results across teams, identify holes in documentation and processes, improve the overall testing. Regular Debriefings also help the testers to develop their testing skills. All in all, Debriefings belong to the important meetings you really should attend. Here are some helpful links how you can structure a debriefing:
Exercise: A Debriefing is usually a meeting and you should not do it alone and by yourself. I still want to ask you that you take your time and reflect on the test sessions you performed during the last exercise. Write down answers to the following questions: 
What happened during testing? What did you find out? Were there things, you wanted to test but couldn’t and why couldn’t you? What is left to test in maybe another session or did you find interesting new session ideas? What were they? And what were you feeling during the session as a tester or potential user of your software? Was something annoying you or were you positively surprised?

STEP 6: Are there tools, that can help me?

The most obvious tools for an Explorer is that of a classical Explorer: A notebook and a pen to write down what she finds during exploration. Still a lot of people ask about software tools.
There are tools, that help with Session-Based Test Management, for example by aggregating the session metadata in the TASK BREAKDOWN section. Since taking notes is a crucial part during a test session I choose a note-taking tool, Evernote, for my projects. This solution works best for me at the moment, especially with the Mac Client App.

I want to emphasise one tool in particular and that is TestBuddy: TestBuddy is still under development and is being designed specifically for Exploratory Testing and note taking by people, who really love this style of testing. The prototypes I saw look very promising. The link below will bring you to a “waiting list”. Please get in contact with the folks at Qeek, they are eagerly waiting for your feedback and insights. 

STEP 7: How do I document my testing?

In an environment that heavily uses scripted testing and test cases, testers usually document their testing by ticking off the steps of a test case as either "passed" or "failed". A test session in Session-Based Test Management does not work that way instead it reflects much more how a detective, a journalist or scientists take notes during an investigation or experiment. A lot of testers switching to Session-Based Test Management are quite surprised at the amount of writing they have to do during test execution and they struggle in finding the right balance.
My personal belief is that you do not write more documentation than you do when using test cases because those have to be written, too. Test cases tend to be heavily documented, a lot of people just don’t connect this to their test execution since they wrote them weeks before not close to or even during testing as they do with a test session. 
Another thing I want to add is that in test case based approaches testers often don’t correctly document their actual testing, for example by ticking all test steps in a test case because they “kind of did this” although they skipped several steps due to routine. They often don’t write down strange things (not bugs!) outside of the current test case scope and things get lost.

It's important to find a healthy balance when it comes to documenting your sessions. Here is a list of ideas or experience reports and don’t worry: Even experienced testers constantly question the way they take note as the last three links prove.
Exercise: Alan's 10 experiments in the last link are a great way to get started. How about giving it a go?

STEP 8: Will Exploratory Testing pass an audit?

I often hear people writing Exploratory Testing off as a just playing with the software, a non-structured testing approach, that does not survive the scrutiny of an audit. This is not true at all. Testers have been using Exploratory Testing techniques in heavily restricted environments. Here are a few links, which can help you to report your Exploratory Testing beyond a single session and to make it audit-prone:
Exercise: Try to come up with a low tech testing dashboard for your application and discuss it with your team members.


In addition to all these blog posts, pdfs or online articles there are two books specifically about Exploratory Testing, which I recommend to you:

Mittwoch, 21. Juni 2017

CDMET: a mnemonic for generating exploratory testing charters

I gave a workshop about exploratory testing a few weeks ago. Furthermore, some colleagues want to use session based testing in another project and don’t have much experience with it so far. One topic both groups were eager to know more about is how to generate test charters: How do I find missions I want to explore during a test session.

 My short answer to this is “focus on the perceived risks in your software and on unanswered questions”. This statement alone is not very helpful, so I came up with various sources, which can point to interesting testing areas and charters. They are also good to start figuring out the perceived risks.

While I clustered these sources I found that there is a little mnemonic to remember them: CDMET. 
Conversation, Documentation, Monitoring and Earlier Testing. Alongside these four clusters, I listed various ideas, which can help you find test charters. I find them especially useful when combined with other oracles, FEW HICCUPS from Michael Bolton for example.

My list is by no means exhaustive, I still think it can help finding new test charters.


Conversation means every form of people speaking to each other. This can span from water cooler talk over regular meetings you join up to meetings you specifically create to talk about testing and risks.
  • talk to various people involved in the product
    • business departments; marketing; product development
    • developers; architects; security people
    • user; customers; managers
  • listen closely and make notes during every meeting you attend
    • daily; retrospective; grooming; planning; debriefing
    • product demonstrations; training
    • status meetings; jour fixes; risk workshops


Documentation is everything that is written down and yes this includes source code. There are a variety of official specification documents, which you can use but you should not end there. There are also emails, group chats, user documentation, etc.
  • official specification documents
    • requirement documentations; use cases; business cases
    • user stories; customer journey; personas 
    • UX designs; mock-ups; feature descriptions
  • technical documentation
    • class diagrams, architecture descriptions
    • sequence diagrams, interface descriptions
    • source code
  • general project documentation
    • wiki or confluence for everything someone deemed worthy of writing down
    • chat logs; emails; project plans; risk lists; meeting protocols 
  • test documentation
    • test cases; test notes; test protocols; test reports
    • bug descriptions; beta feedback
    • automation scripts, code and reports
  • user documentation
    • manuals; handbooks; online help; known bugs
    • tutorials; product descriptions; release notes; FAQs
    • flyers; marketing campaigns


Monitoring encompasses everything that I connect with the actual usage of the product because this is a very powerful input for generating new test charters. Therefore I use the term a bit more loosely as people usually do.
  • technical monitoring
    • crash reports; request times; server logs
    • ELK Stack (Elasticsearch, Logstash, Kibana); Grafana
  • user tracking
    • usage statistics for features, time of day, any other contexts
    • interaction rates; ad turnovers
    • top error messages the user faces regularly
  • user feedback
    • App or Play Store Reviews; reviews in magazines or blogs
    • customer services tickets; users reaching out to product team via email
    • social media like Twitter, Facebook, LinkedIn, etc

Earlier Testing

Earlier Testing is very useful to inform your future testing, it basically means: Use what you figured out yesterday to know what you should look at today. This feedback loop can be even faster when you come across an interesting part of the software while already performing a test session. Note this down and create new charters afterwards. 
If you played your hand right Earlier Testing should blend in with some of the other clusters, because you should document your testing and tell other team members about it.
  • results & artifacts
    • test case execution reports; test notes; status reports
    • bug descriptions; beta feedback; debriefings
  • hunches & leftovers
    • whenever you went “huh?” or “that’s interesting" in the past
    • known bug lists; “can’t reproduce” issues
    • unfinished business (“there is so much more I can look at if I had the time")

Whatever helps

You see that some items in the clusters are not strictly separated, a meeting can have a written down protocol for example. It does not really matter if you remember the meeting you had because you recall the talks you had or because you are flipping through the email protocol you wrote down. 
The important part is that thinking about conversations you had, the documentation you can read, the testing you already performed or the monitoring you can access can help you in figuring out what to test next. It surely helps me.

Dienstag, 16. Mai 2017

Mingling Conferences

Recently a thread started at twitter originating with this tweet:

What then happened were developers asking for testers to join their conferences and vice versa.
We then started gathering developer conferences and testing conferences, which actively want both disciplines mingling. On some of these testing conferences I personally met developers, some are completely built around people meeting, some state they want everybody to join them on their web page.

As twitter is a fleeting medium I want to use this blog post to collect these conferences. I will state which conference it is, where and when it takes place and if it is a developer or tester conference. This turns out to be pretty easy to guess since testing conferences love to put the word “test” in their name.

I want this list to grow so please DM me on twitter, write me a mail or leave a comment so I can curate and update. I am eager to get this blog post out before the energy to do so leaves me so I will start with the mentioned conferences on the twitter thread and over time research more, e.g. by crawling through the respective web pages for partnering conferences. I will also try my best to keep conference dates and places updated in the future, and the table below will hopefully become less ugly, too.
Wish me luck.

Oh, and if you are a UX designer, requirements engineer, product manager or something completely else and think: “Hey, why don’t you want to meet me? I want developers and testers on my conference, too!” Then also contact me, I will add you here. I will add anyone, who wants to help all disciplines mingling more.

Here is the list:

2017-08-24 to 2017-08-27
European Testing Conference
2018-02-08 to 2018-02-10
Amsterdam, Netherlands
2017-05-11 to 2017-05-12
Bucharest, Romania
2017-05-10 to 2017-05-12
2017-10-06 to 2017-10-06
2017-11-13 to 2017-11-17
2017-07-21 to 2017-07-22
2017-01-15 to 2017-01-19
2017-04-06 to 2017-04-09
Gran Canaria,
2017-03-25 to 2017-03-27
2017-11-09 to 2017-11-12
La Roche-en-Ardenne,
2017-10-26 to 2017-10-29
2017-03-09 to 2017-03-12
2017-11-13 to 2017-11-17
2017-06-15 to 2017-06-18
2017-10-20 to 2017-20-21
2017-03-23 to 2017-03-24
2017-01-26 to 2017-01-27
TestBash Dublin
May 2018
2017-10-26 to 2017-10-27
2017-11-09 to 2017-11-10
2017-11-06 to 2017-11-10
2017-05-19 to 2017-05-20
2017-10-12 to 2017-10-13
2017-09-25 to 2017-09-26
2017-11-26 to 2017-11-28
Zwartkop Mountains,
South Africa
2018-02-05 to 2017-02-09
2017-10-24 to 2017-10-26
2017-04-20 to 2017-04-21

I hope a lot of you people start going to the "other" conferences now. If not we have to take extreme measures:

Montag, 22. August 2016

A tester’s thoughts on characterization testing

Michael Feathers just recently posted something about characterization testing on his blog. The term is not new, in fact it is used at least since 2007, still I stumbled over something in this particular blog post. Since I also read Katrina Clokies post about human centered automation at the same time the two topics kind of merged a bit together in my head and got me thinking.
So what is this blog post going to be? Basically it is my stream of thought about characterization testing to see if I can make sense of my thoughts. Hopefully someone else benefits from this, too.

characterization testing is an exploratory testing technique

Let’s start with what characterization testing actually is, at least to my understanding. Characterization testing is a technique, which facilitates writing unit tests to check and document what an existing source code actually does and is therefore especially useful when dealing with legacy code. Note that it is not important if the checked behaviour is also the wanted behaviour.
The created checks are used to find out what a system does and then automatically check that it still works as before while you refactor the code base. If you want to dig deeper on characterization tests and how to create them I suggest you read Michael’s inital blog post or go to Alberto Saviola’s four piece article series, which starts here and ends with a tool, that can create characterization tests automatically.  

Michael starts his blog with the following statement before he moves on to characterization testing itself: "We use [the word testing] for many things, from exploratory testing and manual testing to unit testing and other forms of automation. The core issue is that we need to know that our code works, and we’ve lumped together a variety of practices and put them under this banner.” 

I have a problem with that statement and this was the kicker, which started my thoughts. I namely disagree with stating exploratory testing is there to make sure “that our code works”, because this is not how I see exploratory testing. I use exploratory testing to find out how a system works. If my findings represent desired behaviour or if they result in a series of bugs is often up to debate with my teammates. 

The ultimate reference for exploratory testing to me is Elisabeth Hendrickson’s book Explore It!. I own a german translation therefore I cannot quote here and will summarise instead. Right in the beginning of the book she writes a test strategy should answer two questions:
  1. Does the software behave as planed under the specified conditions?
  2. Are there further risks?
The first one deals a lot with knowing “our code works” as Michael puts it. The second one goes further and also explores (sic!) the system in more detail than just checking it against a specification. Risks are found by learning what the system actually does and using this as an input for even further exploration. 
I think you already know where I am going with this: If exploratory testing is there to learn about risks by learning how the system at hand behaves doesn’t this mean that characterization testing is an exploratory testing technique? Elisabeth’s book even has a whole chapter dedicated to exploring existing (aka legacy) systems, which is precisely what Michael uses characterization testing for.

In this case I think the terms black box testing and white box testing are helpful: While Elisabeth describes mainly black box testing techniques in her book I see characterization testing as a white box testing technique for exploration on unit level. Combine Elisabeth’s techniques with Michael’s characterization testing and you have a very powerful framework to start working on a legacy system, still I see characterization testing more as a part of and not an addition to exploratory testing.

You can read Meike Mertsch’s blog post Exploratory testing while writing code to see how a tester with an exploratory mind works with code while testing, although it might not be characterization testing in the most strictest sense. Meike was also the translator of Explore It! to german.

If you look at characterization testing as a white box exploratory testing technique they have a very unique property when being compared to all the black box techniques in Elisabeth’s book: they create automated checks, which can be seen as a form of documentation of the current system behaviour.

characterization tests are fascinating for testers

This is the point where I have to say that I am a big fan of characterization testing when dealing with legacy systems. Developers, who have to refactor the system, benefit from them directly, because they give them confidence that they did not change the system behaviour in unexpected ways. Testers can use existing characterization tests as a starting point for finding out more about the system.

I don’t know you, but to me finding or writing characterization checks begs the question why the system behaves that way. What is is this behaviour good for and what does it lead to if you put it in the bigger picture of the overall system? Characterization checks can be an input for exploratory testing sessions or fuel discussions with developers, product managers or users. They are an invitation to explore even when they don’t fail and therefore are a good example of checks that help you learn about the system even if the build is green. 

As a tester there are two fallacies regarding characterization tests, I have encountered in the past. The first one is not fixing bugs, because the bugfix breaks a characterization test. Remember that you cannot know if the checked behaviour is correct or wrong. I saw it happen that someone wanted to commit code, but reverted it because it broke some checks. Only later did we find out that the checked behaviour was actually faulty.
The second one is the exact opposite: You know that they are just checking the current state and you are very confident your new code works better than the old one, when the checks break you adjust them to your code and commit everything together. Guess what: The old behaviour was correct and you just introduced a bug.
Since characterization testing comes with all the pros and cons of unit testing (fast & cheap vs. checking only a small part of the system) the situation can even change over time: the checked behaviour is correct until the implementation of a new feature, now the checked behaviour is wrong. The build however stays green. 

ageing characterization and regular checks 

Characterization checks do not just come into existence, in fact Michael and Alberto both wrote down some rules when and how to create them. Now while developers work on a legacy system characterization checks are not the only unit checks they create. There are also regular checks for new code, which are created using TDD and check for a desired behaviour. Both kind of checks end up in the code base and in the continuous integration. In time you may not know anymore if a check stems from characterization testing or TDD. In this sense characterization checks itself can become legacy code, which is hard to deal with.

Imagine entering a project finding 1000 automated checks, 250 of which are characterization checks and the rest are regular checks. If one of the characterization checks fails it is not necessarily a bug, if one of the others fails it most certainly is. Only you cannot see which is which. if the person, who wrote the check, is not on the project anymore you have to treat every failing check as a characterization check and always have to investigate if you found a bug or not. A way to mitigate this is following up on Richard Bradshaw’s  advice to state the intent of a specific check. If you do this you know if a check is a characterization check or not.

Furthermore I have the feeling that a lot of checks become characterization checks over time. When they were written in the first place there was a reason for creating them exactly like they are, checking for a specific behaviour. Now, one or two project member generations later, they are there and document a specific system behaviour. The people, who know why they were created and why the system behaves like this, are gone. The checks have become characterization checks.

This is maybe what Katrina is facing in her project. She writes about a test suite, which is longer with the project than all of the testers, hence they don’t know why there is some certain logic coded into it. Katrina uses this as an example why they do not automate after mastery. I tend to disagree a little bit: The initial team members might very well have automated after mastery, I cannot know for sure, yet knowledge of why has been lost over time. Moving away from Katrina’s example this happens quite often: testers inherit checks from previous testers.

I like to think of a project as body of knowledge, not just the people, but the project itself. There is a lot of knowledge about the system, the users, the workflows in the project’s confluence, in the specific build setup and in the automated checks. From the project’s perspective I see the automated checks as a form of codified prior knowledge.
The current team is left with this form of prior knowledge and now has the problem of finding out why the system behaves like that. Otherwise they risk running into one of the two problems I mentioned earlier: being reluctant to change behaviour that needs changing or introducing bugs by ignoring the checks. This is actually a tough exercise, because finding out why a systems does what it does is usually very challenging. 


Characterization testing is a white box exploratory testing technique and a very powerful tool when dealing with legacy systems. As a tester you should make sure characterization checks are marked as such and try to find out why a system behaves as a characterization check says it does.