One of the things that every sorcerer will tell you is if you have the name of a spirit, you have power over it.
Gerald Jay Sussman quote, 6.001 Structure and Interpretation of Computer Programs – Lecture 1B: Procedures and Processes; Substitution Model, July 1986 (link)
Since I started with the AST BBST Foundations course a few weeks ago, I thought it would be interesting to publish my current terminology for thinking about testing. Hopefully, this will be followed up by a “What I learned from BBST” post four weeks from now.
I do not claim that any of these represent some kind of “agreement” between testers, especially not an international one – these are just the words that I use for shaping my own mental model of what testing means to me. Most of the terms originate from conversations in the Context-Driven Community, and some belong to the RST namespace according to James Bach and Michael Bolton.
Namespaces – is a vocabulary of terms that are confined to a project or a group. Michael Bolton has a great blog post on the topic. Besides the RST namespace, I like to think of BBST course terminology and possibly the ISTQB glossary as namespaces. Additionally, most companies as a whole as well as any project in isolation probably have their own explicit or implicit namespaces.
The cool thing about namespaces is that we can create new ones whenever we want to, as long as we remind ourselves that these words might lose their meaning (or mean something completely different) in a different context.
One of the tricks to using namespaces is to make conscious switches when you change contexts. For example, a “Test Strategy” might have a very specific meaning in one company and a whole different meaning in another – and “out in the open” it is not that well defined at all.
Approaches – which can be thought of as “syles”
- Scripted Testing
- Exploratory Testing (a term with interesting history in its own right)
- Tool-heavy style, Analytical style, Well-documented style, etc
Methods – sometimes called x based techniques
- Black box – which is kind of the same as ‘specification based techniques’ for some people
- White box – which is kind of the same as ‘structure based techniques’ for some people
- Experience based – is sometimes included here but just seems terribly wrong to me.
I’m currently trying to figure out whether I find this distinction useful or not. Interestingly, these terms have been completely removed from the RST class materials.
General test techniques (According to HTSM) – A test technique is a heuristic for creating tests.
- Function, Domain, Stress, Flow, Claims, User, Risk, Scenario, Automatic
Test design techniques – Interestingly, these seem be to almost “factory school” exclusive
Test levels – For me, a test level is the layer at which our testing is performed. It is not really about coverage – we need to test at all levels – but about testability. It might be easiest to test certain elements of the product at the unit level.
- System – also known as end-to-end
- Acceptance – Sometimes included as a test level for reasons uknown. For me, the key difference between system and acceptance testing appears to be the performer. Acceptance testing should be done by the Customer. System testing by the Developing Organization. As such, it is not really a testing level for me. Acceptance testing focuses more on whether the application really performs as required, not as requested.
Test types – This category might as well be called “unsorted”. Wikipedia lists among others: Installation testing, Compatibility testing, Smoke and sanity testing, Regression testing, Acceptance testing (again!), Alpha/Beta Testing, Security, A/B testing.
Some of these seem to be about coverage level, some are non-functional requirements, yet others seem to imply who does the testing.
(Testing) Paradigms – An organizing worldview; a model. CDT is a paradigm. So is Agile. Interestingly, Cem Kaner notes that he has stopped using this term, as noted in a debate video with Rex Black (Paradigms mentioned at 24:45)
(Testing) Methodologies – Are specific ways of working within a paradigm. Scrum, RST, TDD, BDD are methodologies.
Verification and validation – I’ve never used that term in a professional capacity and I’ve heard several people claim the same thing. Still, it’s in all the classic textbooks. I still encounter it in testing philosophy though. In the debate video linked earlier, Rex Black made a note that Testing vs Checking (in the RST Namespace) is analogous to Verification and Validation for him.