Months ago, I’d promised a blog post on the kind of documentation I create / use when testing. Since then, I’ve started working with a different company, which brings with it a reasonable possibility that the kind of documentation I create and use might change, to adapt to the needs of my new employer, its clients and how they each work. However, I’ve been reading a lot about session-based test management recently (with a view to try out the more structured exploratory testing approach and the session reports that come with it), which prompted me to pick up the topic again.
More specifically, I was reading an article by Paul Carvalho where he mentions Test Guides; the idea of which I feel has similarities to the documents I’ve been working with thus far.
Since it’s taken me a while to get round to writing this post, I won’t spend too much time on it. However, I’d like to share this because I haven’t seen any other templates like it (other than what Paul describes) and because I want the origins to still be fresh in my mind when writing. This blog is a way for me to reflect, learn and grow, and so I like the idea of reading this back at some point in the future and seeing if / how things might have changed.
A Brief Word on Other Templates
When I first started testing less than two years ago at my previous company, I was told I’d be responsible for all test documentation. I was given the test case template below that one of the developers had come up with. I think I actually added the Excluded and Variables sections.
I did not enjoy using this. Here are just some of the problems I came across very early on: extremely time consuming to create, difficult to maintain, and most importantly – never actually used. For all the time it took me to create and update documents like this, I never actually used them for anything. They were documents for documents’ sake, and they were taking up valuable testing time.
I Googled around for other templates; to try and find out what other testers were doing. I felt that testing at the company was at too much of a developing stage to invest in any formal test management tool and, to be honest, the look of those didn’t appeal to me either. Sadly, it seemed that all anyone else was using (as far as I could see at the time) were Word documents similar to the one I’d be given, or Excel spreadsheets. On reflection, perhaps I was Googling for the wrong thing. I was actually introduced to this template for “test scripts”. I then spent some time Googling the difference between test cases and test scripts, and then those were the templates for which I was searching. I learned about SBTM much later (at least it seemed much later, within the grand two year total).
So, thinking that perhaps I just needed to make a few adjustments and try a little harder, I took inspiration from some templates online and came up with this Excel template.
Admittedly, I liked the layout of this one better, but it had all the same problems.
By this point, I’d been the sole tester in the business for a few months and I was more confident in my feeling that these documents just weren’t useful, at least for me. With the blessing of my CEO, who I worked with closely, I decided to scrap these test case documents altogether and come up with something completely of my own making; something I would actually use and get value from.
Introducing HISToW
And so, HISToW was created: How It’s Supposed To Work.
Some of you might have noticed the word choice used in the opening sentence: “the documentation I create / use when testing”.
I deliberately avoided phrases such as, “test cases” or “test documentation” or “how I document testing”. Although (as far as I can remember) I was yet to discover the great debate of pre-written test cases vs exploratory testing, I knew very early on that written test cases weren’t for me. They were more counter-productive than helpful, and they were so restrictive! The first few times I tried to use test case documents in my testing, I noticed that I was more pre-occupied with trying to follow the exact steps detailed than focussing on the actual software in test.
And what if I saw something that looked like it might be a bug but wasn’t in the documentation? Was I supposed to ignore it? Was I “allowed” to deviate from the scripted steps to investigate? It was awful!
The strange thing, too, was that I wrote all those test documents, so I could add in the investigation steps if I wanted to. But that somehow felt like cheating. I was bound by my own creation…
The new HISToW documents weren’t to be anything like that. They were designed to be useful reference guides that I could use or not; that developers could refer to or not; that the product owner side of my role could check or not. They contained all the useful knowledge I’d gained from testing so far, including how it (the feature) is supposed to work, the accepted issues that still existed and weren’t to be raised as new issues, and any cases / pitfalls to watch out for.
The new HISToW documents served as great, succinct and combined summaries of requirements, learnings, use cases, scenarios and issues. They can actually be used, and not adhered to. And they are much easier to update, since they’re not granularly prescriptive, and are much shorter than test scripts or test suites. You can skim to the parts you’re really interested in without the worry that you’ve missed some important detail buried in step after step.
They concentrate much more on what we know about the product, allowing testers to use their own intelligence and creativity to test however they see fit, as opposed to restricting testers to only the cases documented, in only the documented variations.
It might also be worth noting that HISToWs are intended to be created or updated after testing and not before it. They’re designed to show what you’ve learnt from testing and act as a reference guide, not dictate what you “should” or shouldn’t be looking at. However, so long as you have sufficient information to start documenting how something is supposed to work and what you want to remember to test, I suppose there’s no reason why you couldn’t also use this to aid test planning.
Obviously I’m biased, but I really like the HISToW format / template. As a word play / way of remembering, you could even call it a HISToWry (as in “history”), since it’s based on a history of testing, discovery and learning, and it also tells a story of what that product or feature is supposed to do and how it might be used. But maybe I’m just grasping at straws trying to come up with a cool name =]
I’d love for you to try this out and let me know your thoughts on it. You can access the HISToW template here. Have you come up with any other templates for documentation in testing that you haven’t seen elsewhere? Let me know in the comments.
Hi,
So I came across this through one of your Linkedin posts. First thing, please set the “Share your thoughts” comment feature of your blog to appear at the top or an action button that leads the user to this 😅
Second, reading this feels like everything I went through. I highly agree with scripted test cases being a pain to maintain. I too found myself in similar situations and more so in situations where, after writing the cases, I’d test the whole thing and then circle back just to check what I missed. That led me to ditching them (Testrail costed my company 80 EUR a month 😬) and writing one liners instead, like a brainstorming free writing thing. My research on the matter then led me to RST and test activity charters which are my go to for test planning now.
Sounds great! What was your company’s reaction to the idea of ditching scripted test cases? Did you have to convince them?
Thank you Cassandra, the more people that use approaches like this make it easier for others to “sell” to their organizations. I
Do you record / capture any of the tests you run under this?
While I couldn’t convince TPTB with regards to not having test plans as an artifact ,as we can be subject to audits, I have at least gained acceptance that recording the testing sessions I do is a defacto 100% accurate representation of the tests. Similarly any automation is self-documenting.
I was lucky enough to arrive in a place that had already bought the QA tool ( Qtest) which is set up for session based testing , though it wasn’t being used for that. Had I not had the tool I could certainly argue for using HISToWs where the business didn’t have a compelling legal need to keep certain docs.
Hi,
Thanks for your kind words.
For the situation that you describe, I’d recommend that you continue using session reports for your audits but that the HISToWs are created as a kind of companion summary for the product as a whole, rather than to record details of specific sessions or test cases. HISToWs focus on the facts of the product, as seen through testing, rather than the actual testing carried out.
Am I making sense? I hope that helps!
Thanks,
Cassandra
I’m probably going to go with my misreading of this 🙂 E.G., I’m using an analogue at more than one level. Overall, at the build level and then one for each feature so each feature gets it’s own session ( or as many as needed)
Hi again
Maybe I should start where I am right now, to give an idea why what you’re presenting here interest me.
Currently my testing is done around a story, task or bug. Basically any unit of work that can be deployed, as when one piece of work is “complete”, it usually get pushed into production.
So I document my testing as all observations I have done on the system around a piece of work during. Basically how the system work, like a form of reverse requirements gathering. Also I include observation related of testing around activities related to this specific changes (a classic case for me would be validating a data migration script).
I could simplifying by saying that my documentation groups multiple HISToW document + other observations.
From time to time (to not say rarely), I extract some information of this into a wiki to serve as reference. Each “document” I produce represent a “session” of test of all testing activities I have done for a specific build, as to represent the coverage and specific observation I have done each time. The documentation will be added simply to the ticket related to the deliverable.
All this to say that right now I’m questioning myself if I should not take time more time to extract documentation of a feature and put it in a structure similar to HISToW in idea. I’m juggling to use some form of online tool like a wiki, documents on google drive or even maybe a git project with basic text file (using some sort of a markdown format). I’m searching a balance to easily search info, history per feature and light management.
But I’m trying to reconcile upfront how I would like this view of the system independent of the system itself, as for me each commit could be seen as a new version of the system. Not sure how much how I would love to create a version of the whole system documentation to represent a build/version of the system and I how I would like to link changes and feature’s documentation to a piece of work.
At least, all this make me realize is that I would still love to keep my documentation related to a changes request, as it’s the easiest way to see my coverage and effort, plus a good communication tool for stakeholder for this piece of work.
Hi Francis,
I think that whatever people choose to do for documentation should be tailored to their circumstance, so it’s good to get that extra info.
I certainly wouldn’t advocate for creating a whole new set of HISToWs per build, but just to update them as those parts of the system change, or you discover new information about them. For this reason, some sort of versioning might be useful so you can track the changes.
The way I used the version numbers on the documents was to confirm that the document was accurate, as in version X. But that’s not to say it would need to be updated every version. However, I can see how that might lead to questions over whether the document is still accurate in the current version or if it’s simply yet to be updated. As I was the sole tester in my last role where I came up with HISToWs, it was easier for me to keep track of all this because I didn’t have to coordinate with anyone else, but I can see how that can get confusing with more people involved.
Sorry, I’m not sure if that response was of any help at all!
Thanks,
Cassandra
Hi Cassandra
Thanks for sharing this.
I’m intrigue to know how the HISToW are organize and store. If they are link to other sources.
Hi Francis,
Good question!
The way I stored them was pretty rudamental. I used folders within the file explorer to separate first into modules, the high level categorisation / division of the system from the perspective of the user, then used sub-folders to separate specific areas of the system within those modules.
You can see there’s a “Feature” heading in the HISToW; this starts the grouping of headings within the document that can be repeated to record several linked features in one document.
Perhaps I should have called that heading “Sub-Feature” or something denoting the smallest reasonable portion for testing and explaining, as I also named the document itself to reflect the separator between area and feature.
That might seem rather nested, but it worked for the system I was testing.
The HISToWs can certainly link to other sources when relevant – there’s no use duplicating data unnecessarily – but if you find that HISToWs are full of references and not much else, I’d question why that is and if the overall documentation strategy needs to be reviewed.
I hope that helps!
Do you have any other ideas about how to organise HISToWs that would suit your context?
Thanks,
Cassandra