While trying to construct a test strategy for a difficult project, I was recently given this advice: “Don’t test thoroughly”.
The droning siren already in the back of my mind loudened to a shrill alarm bell that it made it hard to concentrate.
Don’t test thoroughly? Why do you hate me? Don’t you know me at all? Tell me that’s a joke! – the knee-jerk remarks I did well not to verbalise.
In truth, there were valid reasons for giving this advice that I grudgingly and uncomfortably accepted after further discussion, on the basis of us agreeing pre-test that this code would not be shipped without more testing than what I was about to do; what I felt to be far from adequate.
With the same feeling one might have leaving their cat alone for three days while visiting a sick relative, I returned to my desk still feeling bad about what I’d just agreed to. Kind of necessary, although maybe not…?
Knowing deep down that a “thorough test” is not always the right test, and looking for re-assurance from fellow testers, I tweeted:
To me, "don't test thoroughly" = "half-ass your job". Please remind me why that isn't always the case? Hate feeling I'm not doing my job.
— Cassandra H. Leung (@Tweet_Cassandra) February 9, 2017
The responses I received made me feel a little better. I wouldn’t instantly be banished from the testing circle for failing to uphold the moral code!
I reflected further on, “don’t test thoroughly,” and what the “real” message behind this might be, considering suggestions from those that had responded to my appeal.
“Don’t test thoroughly” could actually be a way of saying:
- We’re ready for some happy path tests, but not exploring
- The current scope for testing is narrow, but will widen before we release
- We’re aware of outstanding bugs that aren’t to be addressed at this time
- We’re aware of outstanding bugs for which a fix is in progress
- We want you to report new issues in a different way
- We’re not ready to address any other issues yet
- Only critical bugs are being looked at
- As long as there are no errors, unexpected behaviour is okay just now
- We just need the core to work right now, and will branch out later
- We don’t want a lot of time spent on this
- We’re under time constraints
- The goal for test coverage is much lower than you’ve planned
- The areas you’re testing don’t impact the value of the product
- Your tests are too geared towards edge cases
- We need more speed than perfection
The list goes on…
Looking back on how I’ve approached testing previously, I can see how some of these messages could be relevant to the project in question, and my tendencies in general.
Although the initial conversation from which this stemmed was somewhat difficult and went against the grain, it’s been a great opportunity to consider not only the underlying messages people (including myself) are trying to get across, but also what I can do to keep improving how I test and ultimately deliver value to the business.
When I likely forget all this the moment I next hear, “don’t test thoroughly,” I’ll be able to refer back to my own list of possible translations, and hopefully ask the right questions to find out what’s really being said.
At the heart of it, testing needs to be fit for purpose. So, perhaps the next time I get a request that doesn’t seem aligned with my thoughts on testing, I could start by asking, “what’s the purpose of this testing?” I may find that by more clearly identifying and understanding shared goals, a request that was initially startling might turn out to be completely reasonable.
Have you thought of any other possible translations, whether genuine or pure comedy / sarcasm? Share them in the comments!
Thanks for the reminder!
Usually, when I hear (or say) something like that, it means one of the following (not surprisingly, some are in your list):
1) It’s a low-risk change\feature. We assume it will either break completely or work well enough.
2) we don’t care if it’s rough around the edges – fixing it would cost us more than leaving it as is.
3) We will have time to do thorough testing later, quick results now, please.
4) We assume you are doing magic, so make your magic faster.
With the exception of the 4th case, those are valid reasons to work shallow. The 4th one usually comes with a “why didn’t you find this?!” pointy finger attached.
Hey,
Thanks for commenting, those are great additions! Number one is definitely one I’ve had a few times… Four means your cover’s been blown and you need to create a distraction 😛
Cassandra
Yeah when in the USA i use their spelling, should have changed it for UK blog:)
Hit enter on that last one too soon. It was my employer at the time that wanted me to check the testing that the third party co had been paid for. It was a fun time:)
Ah, seems like they’re not the best resource for you then if you need to retest their work and find more issues!
For me the context is who is saying it, when they are saying it and hopefully explained by some of the bullet points above. E.G, They are all acceptable to me if the person saying it has the authority to do so and owns the result outwith anything we agreed upon. Should it be to mask the results from any of the other stakeholders it’s probably time to find another place to work..
I can add one I was removed for a week from a project because too many defects were being located and it was —–demoralizing for the third party company— creating the work I was testing.
It became annoyingly political because no one wants to hear that they paid 7 figures for a site that had 1/3 of it’s deliverables unfit to publish in the app store. Yes they could have used “we’re under time constraints, but so is everyone alive and fundamentally doesn’t change the fact that the app didn’t solve the problem it was created to do. What’s weird is my company went along with their request..
Most of the other bullets I’ve seem and could be covered by “As a project manager I need something at this location that appears to work by this date, E.G, item is in place but is switched off , soft launched, awaiting final UAT etc mollifies CXO suite , they had a bullet for Q3 and that bullet was ‘technically’ met… ergo we’re all happy :/
Hey,
Thanks for your comment!
You’re right, context definitely plays a part.
About the “demoralising” reason… I think I’d have a much harder time accepting that! Maybe in that case, it was the client who needed to ask themselves the purpose of testing and the potential consequences of not testing. Also, seems weird that a client would be unhappy that you’re doing what they paid you too! But clients are most definitely weird like that.
Cassandra