Documentation: General Quality and Testing Concepts

Reading Time: 4 minutes

 

When starting on a new project, it’s very common for me to be the first and / or only tester.  I find myself frequently wanting to document the same things, rewriting them from scratch.  To help with this, I’ve decided to include some of this documentation on my own blog, so I no longer have to start from scratch.  I’ll probably edit and improve them over time.  You can view all posts in the documentation series.

 

 

Foreword

 

As in many domains, there’s a lot of discussion around what specific terms mean in the context of quality and testing.  Here, I provide brief working definitions only to help teams I work with know what we’re referring to when we use each of these terms.  This doesn’t doesn’t mean they are the only, full, or definitive definitions.  This list is not exhaustive.

 

 

Basic Terminology

 

What is Quality?

• Expanding on Gerry Weinberg’s definition, a popular definition is: “Quality is value to some person who matters.”
• Who matters? Customers, end users, stakeholders, project managers …?
• What is value?  Solving a problem, providing an ability, improving a process …?
• Quality is subjective, and can differ greatly from project to project and person to person; that’s why I like to use RiskStorming to align on a working definition of quality as soon as possible

 

What is a Bug?

• A bug is anything that detracts from the quality of a given subject (usually an SUT); an inability to complete a process, a performance issue, a poor user flow …

 

What is Testing?

• Testing is the pursuit of information (of which there are many forms), an objective of which could be to assess the quality of a system under test (SUT)
• Testing is more than bug hunting; it includes investigating, experimenting, questioning …
• Testing is a type of activity, not a person
• Testing is diverse, including many activities, approaches, techniques …
• The “testing vs. checking” distinction is an important, but advanced topic; don’t worry too much about it until you’ve learned the basics

 

What are End to End (E2E) Tests?

• This term is commonly used for two different things, so it’s important to check which one is meant
• It’s usually a way to refer to tests that test some “completeness”
• Complete user journey, from start to finish
• Complete system architecture, including lowest to highest layers / levels

 

 

Concepts Compared

 

Scripted vs. Unscripted vs. Exploratory Testing

• Scripted testing is prescriptive, rigid, and confirmatory
  • Can be preformed by humans or machines
  • Only possible when expected behaviour is fully known
  • Ideal for tests that are frequently repeated
• Unscripted testing does not follow a predefined set of instructions
  • Performed by humans
  • Commonly referred to as “manual testing”*
  • Can be confirmatory or informative
• Exploratory testing is investigatory, flexible, and informatory
  • Performed by humans
  • Commonly referred to as “manual testing”*
  • A type of unscripted testing, which can be managed using session-based test management (SBTM)
• * I reference the term “manual testing” only because it’s so commonly used, but it does lend itself to a lot of misconceptions and outdated ideas
  • “Manual” is only a method of execution, not an approach or technique
  • There is no “manual vs. automated” testing conflict

 

Testing Layer / Level vs Testing Target

• Testing layers / levels refer to where in the system architecture testing is taking place, e.g.:
  • Unit level (usually the lowest, cheapest** possible)
  • Service level (usually involving several interacting units)
  • GUI level (usually the highest, most expensive** possible)
• Testing targets refer to what you’re trying to test, e.g.:
  • Individual units or blocks in isolation (e.g., individual methods)
  • Integrations between multiple units (i.e., how they behave when interacting with one another)
  • GUI (e.g., what the user can interact with)
• People often confuse these two concepts (e.g., when looking at a test automation pyramid); see TUTTU below
• ** When talking about how “cheap” or “expensive” tests are, we’re talking about factors like run-time and the precision of feedback they’ll give you

  • Cheaper tests: The lower the level of tests, the more efficient they will be, both in terms how long it takes to run the test, and how long it takes to find exactly where the issue lies

  • More expensive tests: The higher the level, the more layers of the system architecture are exercised, and the more variables there are to comb through in order to pinpoint the source of the issue

 

Testing the UI or Testing Through the UI (TUTTU)

• When testing the UI, the graphical elements rendered are important (e.g., list of saved items is visible to the user)
• When testing through the UI, we’re not really interested in how things are rendered (e.g., list of saved items is stored in the database)
• When thinking about which level to perform tests on, think about what you’re really trying to test, and use the lowest layer possible

 

“Whole Team Approach” to Quality vs. Testing Silos

• With a whole team approach to quality, quality is everyone’s responsibility and is built in from the start; quality and testing are part of the development process (please do this)
• With testing silos, all quality and testing topics are considered to be outside of the development process, and associated activities are usually only carried out by a single role or an entirely different group of people (please don’t do this)

 

Verification vs. Validation

• Verification is about checking conformance to a set of requirements or specifications
• Validation is about assessing value by using realistic scenarios

 

Heuristics vs. Oracles

• According to BBST, a heuristic is a decision-support tool that is sometimes wrong, but still useful
• Oracles are a type of heuristic that help us determine whether something is correct
• e.g., How can you know what time of day it is?
  • Heuristic
    • Useful: Check to see how light or dark it is outside
    • Sometimes wrong: It depends on the time of year, weather, location …
  • Oracle
    • Useful: Look at a watch or clock
    • Sometimes wrong: Could be set to the wrong time, doesn’t differentiate between day and night time …

 

Testing vs. Checking

• This is an advanced topic that’s worth diving into once you’re comfortable with the basics
• TL;DResearch: Testing uncovers; checking confirms

 

---

 

Find this useful?  I’m happy for you to use it as a basis for your documentation too.  Please just add appropriate attribution (e.g., linking to this post).