S1. Understand Expectations
1. Understand the marketplace and target customers for the system
2. Identify the needs and expectations of end-users based on the features required to fulfill their tasks
3. Consider how the system will be deployed and the different types of end-users it will address
3. Hypothesis Based Testing (HBT) is a scientific personal test methodology that is unique in its approach to ensuring cleanliness of software. It is a
goal focused approach, commencing with setting up of cleanliness criteria, hypothesising potential defect types that can impede this, and then
performing activities to ensure that testing is purposeful and therefore effective and efficient. The central theme HBT is constructing a hypotheses
of potential defects that may be probable, and then scientifically proving that they do not indeed exist. The activities of testing like test strategy,
test design, tooling & automating become purposeful as these are focused on uncovering the hypothesised defect types ensuring that these
activities are done scientifically and in a disciplined manner.
HBT is based on sound engineering principles geared to deliver the promise of guaranteeing cleanliness. Its core value proposition is about
hypothesising potential defects that may be present in the software and then allow you to engineer a staged detection model to uncover the
defects faster and cheaper that other typical test methodologies.
HBT fits into any development methodology and weaves into your organisational test process. HBT is powered by STEMTM (STAG Test
Engineering Method) a collection of EIGHT disciplines of thinking. STEM provides the foundation for scientific thinking to perform the various
activities. It is personal scientific inquiry process that is assisted by techniques, principles and guidelines to decompose the problem, identify
cleanliness criteria, hypothesise potential defect types, formulate test strategy, design test cases, identify metrics and build appropriate automation.
3
4. Inspirations from nature
HBT has been inspired by certain ideas and these are discussed below. The inspirations have come from “Properties of matter”, “Fractional
distillation”, “Sherlock Holmes”, “Picture of baby growth”.
Properties of matter
Physical & Chemical properties of matter allow us to:
... classify
“affected by” ... understand behaviours, interactions
... enable checking purity
How can we use a similar train of thought to identify
“properties of cleanliness” and then “types of defects”?
“Properties of the system”
End user expectations Cleanliness criteria
Issues in specifications,
structure, environment Potential Defect Types (PDT)
and behaviour
4
5. Inspirations from nature
Fractional distillation
This is a technique to separate mixtures that have components of different
boiling points
In software systems, there exists a variety of defect types that may be present in
the system. How can we apply this thought process to optimally uncover the
defects, by “fractionally distilling” them?
Can we separate these types of defects on the basis of certain properties and
optimally uncover the defects?
From : http://withfriendship.com
5
6. Inspirations from nature
Picture of baby growth
The picture shows the health of the foetus/baby .
This picture shows size, shape, parts and types of issues not
present
Seeking inspiration, can we depict the health of software system in
a similar manner? Can we measure the ‘intrinsic quality’ at a
stage?
As we progressively evaluate in a staged manner, certain types of
defects detected & removed and therefore quality grows.
Can we chart this as “cleanliness index”?
Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg
6
7. Inspirations
Sherlock Holmes
Sherlock Holmes was person who applied deductive logic to solve
mysteries.
How can we see inspirations from Holmes to hypothesise the
types of defects that may be present and prove presence of these?
7
8. HBT - A Personal Scientific Test Methodology
Test methodologies focus on activities that are driven by a process which are powered by tools, yet successful outcomes still
depend a lot on experience.
Typically methodologies are at organisational level.
On the other hand HBT is a personal scientific methodology enabled by STEMTM , a defect detection technology to deliver
“Clean Software”
8
9. Scientific approach to detecting defects
Cleanliness criteria What is the end user expectation of “Good Quality”?
Potential Defect Types What types of issues can result in poor quality?
Evaluation Stage When should I uncover them?
Test Types How do I uncover them?
Test Techniques What techniques to generate test cases?
Scenarios/Cases What are the test cases? Are they enough?
Scripts How do I execute them?
Metrics & Management How good is it? How am I doing?
9
10. How is HBT different from other test methodologies?
The typical test methodologies in vogue have relied on strength of the process and the capability of the individual to ensure
high quality in the given cost and time constraints. They lack the scientific rigour to enable full cost optimisation and more
often rely on automation as the means to driving down cost and cycle time. For example, they do not provide a strong basis
for assessing the quality of test cases in terms of their defect finding potential and therefore improve effectiveness and
efficiency.
HBT on the other hand enables you to set a clear goal for cleanliness, derive potential types of defect and then devise a
“good net” to ensure that these are caught as soon as they get injected. It is intensely goal-oriented and provides you with a
clear set of milestones allowing you to manage the process quickly and effectively.
Goal
drives
la
T
ic
B
yp
H
Activities
T
Activities
defect detection
...................................... ......................................
Powered by
Powered by
technology
experience
(STEM)
...................................... ......................................
...................................... ......................................
...................................... ......................................
...................................... ......................................
...................................... ......................................
hopefully
results in
Goal
10
11. Hypothesis Based Testing - HBT 2.0
A Quick Introduction Personal, scientific test methodology.
SIX stage methodology powered by
EIGHT disciplines of thinking (STEMTM).
Setup Hypothesise
Cleanliness Criteria Potential Defect Types
SUT
Nine Stage
Cleanliness Assessment
Defect Removal Filter
11
12. A quick introduction to HBT
SIX stages of DOING powered EIGHT disciplines of THINKING
by
D8 D1
S6 S1
Analysis & Business value
Assess & Understand
management understanding
ANALYSE EXPECTATIONS
D7 D2
Execution & Defect
D8 D1
reporting STEM Core hypothesis
Tooling D7 D2
S5 STEM Understand 32 core
SUPPORT S2
D6 D3 CONTEXT concepts
D5 D4 Strategy &
Visibility
planning D3
D6
Devise Formulate
HYPOTHESIS Tooling Test design
PROOF
S4 S3 D5 D4
HBT powered STEM
Personal test methodology by Defect detection technology
12
14. Connecting HBT Stages to the
Scientific approach to detecting defects
S1
Cleanliness criteria Potential defect types S3
S2
Expectations Staged & purposeful detection
S4
Complete test cases
S6 Goal directed measures Sensible automation S5
14
15. Clear baseline
Set a clear goal for quality
Cleanliness criteria Potential defect types
Example: Clean Water implies
S1, S2 1.Colourless
Staged & purposeful 2.No suspended particles
3.No bacteria
detection
4.Odourless
Expectations
What information(properties) can be used to identify this?
Complete test cases
... Marketplace,Customers, End users
... Requirement(flows), Usage, Deployment
... Features, Attributes
Goal directed ... Stage of development, Interactions
Sensible automation
measures ... Environment, Architecture
... Behaviour, Structure
15
16. A goal focused approach to cleanliness
Identify potential defect types that can impede cleanliness
Cleanliness criteria Potential defect types S3
Example:
Data validation
Timeouts
Staged & purposeful
Resource leakage
detection
Calculation
Expectations Storage
Presentation
Complete test cases Transactional ...
Scientific approach to hypothesising defects is about looking at
Goal directed FIVE Aspects - Data, Logic, Structure, Environment & Usage
Sensible automation
measures from
THREE Views - Error injection, Fault proneness & Failure
Use STEM core concepts
> Negative thinking (Aspect)
> EFF Model (View)
“A Holmes-ian way of looking
at properties of elements”
16
17. Levels, Types & Techniques - STRATEGY
NINE levels to Cleanliness
Cleanliness criteria Potential defect types
L9 End user value
S4
Staged & purposeful L8 Clean Deployment
detection
Expectations L7 Attributes met
Complete test cases L6 Environment cleanliness
L5 Flow correctness
Goal directed Sensible automation
measures L4 Behaviour correctness
L3 Structural integrity
L3
Quality Levels Test Techniques (T1-T4)
PDT7 L2 Input interface cleanliness
TT5
PDT6
TT4 TT3 T3
L2 PDT5
L1 Input cleanliness
PDT4
TT3 TT2
L1 PDT3 T2
PDT2 TT2 TT1 T1
PDT1 TT1
PDT: Potential Defect Types 17
18. Countable test cases &
Fault coverage
Countable test cases & Fault coverage
Use STEM Core concepts
Cleanliness criteria Potential defect types > Box model
> Behaviour Stimuli approach
> Techniques landscape
> Coverage evaluation
Staged & purposeful
detection
to
Expectations - Model behaviour
S4 - Create behaviour scenarios
Complete test cases - Create stimuli (test cases)
Irrespective of who designs, #scenarios/cases shall be same -
COUNTABLE
Goal directed Sensible automation Test Scenarios/Cases
measures
R1 PDT1
TS TC1,2,3
R2 PDT2
TT
R3 TS TC4,5,6,7 PDT3
Requirements & Fault traceability
That test cases for a given requirement shall have the
ability to detect specific types of defects
FAULT COVERAGE
18
19. Focused scenarios + Good Automation Architecture
Level based test scenarios yield shorter scripts that are
more flexible for change and easily maintainable.
Cleanliness criteria Potential defect types
L9 End user value
Staged & purposeful
detection L8 Clean Deployment
Expectations
L7 Attributes met
Complete test cases
L6 Environment cleanliness
S5
L5 Flow correctness
Goal directed Sensible automation
measures
L4 Behaviour correctness
L3 Structural integrity
L2 Input interface cleanliness
L1 Input cleanliness
19
22. Six staged methodology to produce clean software
The act of validation in HBT consists of “SIX Stages of DOING”. It commences with first two stages focused on a scientific approach to
understanding of the customer expectations and the the context of the software. One of the key outcomes of the first two stages is
“Cleanliness Criteria” that gives a clear understanding of the expectation of quality. In the third stage, the Cleanliness Criteria and the
information acquired in the first two stages are used to hypothesise potential types of defects that are probable in the software. The fourth
stage consists of devising a proof to scientifically ensure that the hypothesised defects can be indeed be detected cost-efficiently. The fifth
stage focuses on building the tooling support needed to execute the proof. The last stage is about executing the proof and assessing if the
software does indeed meet the Cleanliness Criteria.
Who are the customers, end users, what do they need, and
S1
S6 S1 what do they expect?
Assess & Understand
ANALYSE What are the features of the system, what technologies are
EXPECTATIONS S2
used, architecture?
D8 D1 What types of defects may be present?
D2 S3
Tooling D7 Understand What types of fishes to catch?
S5 STEM
SUPPORT S2
D3 CONTEXT
D6
D5 D4 What is strategy, plan, test scenarios/cases?
S4
Sherlock Holmes
Devise Formulate
PROOF HYPOTHESIS What tools do I need to detect the defects?
S5
Boat in the fishing analogy
S4 S3
How am I doing? How is quality?
S6
Fisherman
22
23. Stage #1 : Understand EXPECTATIONS
The perception that end-users have of how well the product delivers the needs denotes the quality of the
Understand the software/system. "Needs" represent the various features that the software/system needs to have, to allow the
marketplace for end-user to fulfill his tasks effectively and efficiently. "Expectations" on the other hand represent how well the
the system needs are fulfilled.
The final software/system may be deployed in different marketplaces addressing the needs of various types of
Understand the customers. Hence it is imperative that we understand the various target markets (i.e marketplace) where the
technology(ies) used software or system will be deployed. There could be different types of customers in the marketplace. Hence it
is necessary to identify the various types of customers and then finally identify various types of end-users
present in the customer. What we have done now is to start from outward direction i.e marketplace and
adopt a customer/end-user centric view to understand the needs and expectations.
Understand deployment
environment Once we have identified the various types of customers and the corresponding end-users, we can move on to
understand the various technologies that make up the software or the system and also a deployment
environment. The intent is to get a good appreciation of the "construction components" and the target
environment of deployment. It is imperative that we should have a good understanding of the internal aspects
Identify end user types and not merely the external aspects of the system.
& #users for each type
Now we're ready to go into a detailed analysis of the various types of end-users and the typical number of
users for each of these end-users. Subsequent to this, we need to identify the various business requirements
Identify business i.e. "needs" for each end-user.
requirements for each
user type At the end of the stage, the objective is to have a good understanding of the various end-users and their
needs paving the way to understanding expectations clearly.
23
24. Needs & Expectations
NEEDS
Customers in End users Should write
Should have a eraser
Kids
Education EXPECTATIONS
Should be attractive
Seniors
Should be non-toxic
Lead should not break easily
Product Artists
Drawing
e.g Pencil NEEDS
Draftsmen Should write
Should not need sharpening
Management EXPECTATIONS
Corporate Thickness should be consistent
Variety of thickness should be
Engineering available
Variety of hardness should be
Admin available
Needs typically features that allow to get the job done.
Expectations are how well the need is satisfied.
Remember Functional & Non-functional requirements ?
24
25. What does “understanding” involve?
Good understanding of what is expected is key to effective testing. To accomplish this, it is imperative that we commence from understanding
who the various types of end users, their requirements and subsequently the expectations that they have from these. Having a deep domain
knowledge helps immensely. But what if I this is a domain that I am not very conversant with? Is there a scientific way to undertand?
Understanding is a non-linear activity, it is about identifying the various elements and establishing connections between these. In the process of
connecting the dots, missing information is identified, leading to intelligent questions. Seeking answers to these questions aids in deepening the
understanding.
These are some of the elements that need to be understood.
Some of the information elements are “external to the system” i.e.
marketplace, customer types, end users, business requirements
while some are “internal to the system” i.e. features, architecture,
technology etc.
Stage #1 (Understand EXPECTATIONS) focuses on “external
information while Stage #2 (Understand CONTEXT) focuses on
“internal information”.
“Good testing is about asking intelligent questions leading to deeper understanding.”
25
26. Information extracted & artefacts generated
Information At each stage, certain information is extracted, understood and transformed into artefacts useful to
perform effective & efficient testing.
Marketplace
Customers
Artefacts
The key outcomes as demonstrated by the
artefacts are:
User types
System overview ‣The big picture of the system
‣The various end users ascertained for
Requirements different classes of customers in different
HBT
marketplaces
Stage #1 User type list
Deployment ‣A list of business requirements for each
type of end user.
environment.
Requirement map
Technology
Lifecycle stage
In Stage #1, the focus is on external
information that relate to marketplace,
#Users/type customers, end users andHBT Stage 1
business “Good understanding is key to effective
requirements. This stage is useful to get the testing. Identifying who will use what is the
bigger picture of the system and its potential beginning to become customer-focused”
usage and the how it is deployed.
27. Deliverables from Stage #1
Should contain a a good overview of the marketplace, the various types of customers, end-users types,
System overview
deployment environment and technologies that will be used to build the system.
Should contain a list of the various types of users for different types of customers in various market
User type list
segments.
Should contain a list of the business requirements and high level technical features mapped to the various
Requirement map
individual types.
STEM Discipline applied in Stage #1
The STEM discipline “Business value understanding” of STEM is applied in this stage of HBT.
The two STEM core concepts of “Landscaping” and “Viewpoints” are useful in this stage to scientifically understand the expectations.
27
28. Stage #2 : Understand CONTEXT
In this stage the objective is to understand the technical context in terms of the various features, their relative
Identify technical business value, the profile of usage and ultimately arrive at the cleanliness criteria. Note that at this stage, we
features and baseline are moving inward to get a better understanding of technical features of the system.
them
Having identified the various business requirements mapped by each type of end-user, the next logical step is
to drill-down to the various technical features for each business requirement. It is important to understand
Understand the various technical features that constitute the entire system do not really work in isolation. Therefore it is
dependencies necessary to understand the interplay of the features i.e. understand the dependencies of a feature with other
features. Understanding this dependency is very useful at later stages of the life cycle, particularly to regress
Understand profile of optimally.
usage
We now have a list of requirements and the corresponding technical features mapped by each end-user. We
are ready to proceed logically to understand the profile of usage of each of the features by the various end-
Identify critical success users. To do this it is important to understand the typical in the maximum number of users for each user type
factors and then the volume of usage by each user for every technical feature. Since we already have a mapping
between the end-user type and the technical feature feature, all we have to do is to understand as to
approximately how many times this feature will be used by typical end-user of that end user type. The intent
Prioritize value of end of this is to gain a deeper understanding of the usage profile to enable an effective strategy formulation at the
users(s) and features later stage of HBT.
It is not only sufficient that the features work correctly, it is equally important that the various attributes of
Ensure attributes are the nonfunctional aspects of the various features are indeed met. Typically nonfunctional aspects of the
testable system are identified in the highest system level, and typically turn out to be fuzzy. Good testing demands that
each requirement is indeed testable. In HBT, attributes are identified for each key feature and then aggregated
to form the complete set of nonfunctional requirements. We will do this in two stages: firstly identifying the
Setup cleanliness critical success factors for the technical features and thereof the business requirement and then detailing the
criteria critical success factors to arrive at the nonfunctional requirements or attributes. Hence after figuring out the
usage profile, identify the success factors for each business requirement.
28
29. Stage #2 : Understand CONTEXT
Good testing is not about testing all features equally, it is about learning to focus more on those requirements/features that affect the customer
experience significantly. This does not imply that some requirements/features are less important than the others, it simply means that some
requirements/features are more important . Before we start detailing the various attributes, it is worthwhile to the prioritize the various
requirements /features and also various end-user types. To prioritize, start by prioritizing the various types of end users in terms of their
importance to the successful deployment of the final system. Subsequently rank the importance of each of the requirement/feature for each of
the end-user type. At the end of this exercise, we should have a very clear understanding of the business value of each requirement/feature. Note
that the understanding of usage profile comes in very handy here.
Now we are ready to derive the various attributes from the previously identified success factors and ensure that they are testable. A testable
requirement simply means that it is an unambiguously possible to state whether it failed all passed after executing it. In the context of attributes,
testability implies that each attribute does indeed have a clear measure/metric. Therefore it is necessary to identify the measures and the
expected value of the measures for each of the attribute.
Having identified the various technical features and the corresponding attributes, the usage profile in the ranking of the requirements/features,
we are now set to identify the various criteria that constitute the cleanliness of the intended software. Cleanliness criteria in HBT represents
testable expectations. Cleanliness criteria provides a very strong basis for ensuring a goal-focused testing. This allows one to identify potential
types of defects and then formulate an effective strategy in the complete set of test cases It is important that the cleanliness criteria is not vague
or fuzzy.
29
30. Information extracted & artefacts generated
At each stage, certain information is extracted,
Artefacts understood and transformed into artefacts useful
to perform effective & efficient testing.
Information
Feature list The key outcomes as demonstrated by the
Features artefacts are:
Value prioritization ‣A clear list of technical features
matrix ‣Ranking of features to focus on high risk
Usage areas
‣Profile of usage
HBT Usage profile ‣List of attributes
Focus areas Stage #2 ‣Feature interactions
Attributes list
‣Clarity of expectations outlined as
Attributes “Cleanliness criteria”
Interaction matrix
Interactions
Cleanliness criteria
In Stage #2, the focus is on internal
information that relate to technical features,
their interactions, focus areas, attributes,
architecture, technology.
30
31. Deliverables from Stage #2
Feature list Should contain the list of technical features, that forms the technical features baseline.
Value prioritization
Should contain a set of users, requirements and features ranked by importance.
matrix
Usage profile Should contain a the profile of various operations by various end users over time.
Should contain the key attributes stated objectively i.e. state expected value for all the measures
Attributes list
for each attribute.
Should contain the which feature affects what. Note that this should list the interactions and not the details
Interaction matrix
of interactions. The objective is to get a rapid understanding of the linkages.
Cleanliness criteria Should contain criteria that need to be met to ensure that the deployed system is indeed clean.
STEM Discipline applied in Stage #2
The STEM discipline “Business value understanding” of STEM is applied in this stage of HBT.
The STEM core concepts of “Interaction matrix”, “Operational profiling”, “Attribute analysis” and “GQM” are useful in this stage to
scientifically understand the context.
31
32. Cleanliness criteria
Cleanliness criteria is a mirror of expectations, The intention is to come up with criteria that if met will ensure that system meets the
expectations of the the various end users. This is not be confused with “Acceptance criteria”, as “Acceptance criteria” is typically at a higher
level. Acceptance criteria is typically “extrinsic” in nature i.e. it describes aspects like long duration running, migration of existing data, clean
installation and running in the final deployment environment, delivering stated performance under real-life load conditions.
Cleanliness criteria represents the “intrinsic quality” i.e. what properties should the final system have to ensure that it is deemed clean?
Use the properties of the FIVE aspects of Data, Business logic, Structure, Environment, Usage as applied to your application to arrive at these
criteria specific to your application.
Note that the cleanliness criteria should both the the functional and non-functional requirements.
The recommended style of writing Cleanliness criteria is:
“That the system shall meet ....”
Examples:
That the system is able to handle large data (need to qualify large)
That the system releases resources after use.
That the system displays meaningful progress for long duration activities.
That the system is able to detect inappropriate environment/configuration.
32
33. Stage #3 : Formulate HYPOTHESIS
Having understood the expectations and the context resulting in the formulation of cleanliness criteria, we are ready to hypothesize
the potential defects that could affect the cleanliness criteria. This is one of the important stages of HBT resulting in a clear
articulation of the various types of defects and forms the basis for the remaining stages of HBT.
The key idea is to use the external information like the feature’s behaviour, environment, attributes, usage and internal information like
construction material i.e technology, architecture to hypothesize the potential defects that may be present in the software under
construction. Also note that the history of the previous versions of the software or similar systems can also be used to construct and
strengthen the hypothesis. Having hypothesized the potential defects, it is possible to scientifically construct a validation strategy and
design adequate test cases, thereby ensuring that the final system to be deployed is indeed clean.
The FIVE key aspects useful for constructing hypotheses of defects are: data, business logic, structure, environment and usage. This
HBT stage allows us to follow a structured &scientific approach to the hypothesize the potential effects ensuring that we do not miss
any.
33
34. Stage #3 : Formulate HYPOTHESIS (continued)
Firstly use the external information like data specification and business logic
Identify potential faults for the five
specification to identify the potential defects. The information related to data that could
aspects - Data, Business logic, Structure,
help are: data type, boundaries, volumes, rate, format, data interrelationships. The intent
Environment, Usage
should be to get into a "negative mentality" and think of what can go wrong with
respect to all the information related to the data and then produce a list of potential
Identify potential failures of the five defects.
aspects - Data, Business logic, Structure,
Environment, Usage Now use the information related to the business logic to identify the potential effects.
Business logic or the intended behaviour primarily transforms the various inputs i.e.
input data to outputs that the user values. The intention is to identify potential
Identify potential errors that could be transformation losses. The information specific to business logic that is useful for
injected in the five aspects - Data, arriving at potential defects are : the various conditions and their linkages, values of
Business logic, Structure, Environment, conditions, exception handling conditions, access control and dependencies on the
Usage other parts of the software. Once again, the intent is to get into a "negative mentality",
and identify erroneous business flows of logic.
Now identify potential defects (PD) & Up to now the focus has been on using external information like the specification of
combine PDs, remove duplicate PDs data and business logic to identify the potential defects. Now focus on the internal
information like structure of the system and construction materials(i.e. language,
technology) used to build the system to hypothesize potential defects. Structure at the
Group similar PD to form Potential highest level represents the deployment architecture while structure at the lowest level
Defect Types (PDT) represents the structure of the code. Some of the structural information that could be
useful to hypothesize are: flow of control, resource usage, distributed architecture,
interfacing techniques, exception handling, timing information, threading, layering. As
Map PDTs to the elements-under-test i.e.
features/requirements explained above, continue with the similar train of thought of examining these
information with intent to identify potential problems in the structure.
34
35. Stage #3 : Formulate HYPOTHESIS (continued)
Having identified potential defects using the behavioural and structural information, examine information related to environment and how they
can affect the deployed system. By environment, we mean the associated hardware and software on which the system is deployed and the
hardware, software and application resources used by the system. The objective is to examine carefully how these can affect the finally deployed
system. Some of the key information related to the environment that could be useful are: hardware/software versions, system access control,
application configuration information, speed of hardware (CPU, memory, hard disk, communication links), environment configuration information
(e.g. #handles, cache size etc), system resources (hardware, OS and other applications).
Up till now we have taken a fault-centric approach of looking for potential faults (aka defects) by examining external or internal information. In
addition to a fault-centric approach, we can also view the system from potential failure points and then identify the potential defects.
Additionally, it is also possible to examine the system from an error injection point of view. That is, understand the kinds of potential errors that
could be injected into the system to irritate the potential defects if any. The objective is to ensure that we have examined the system from all
three views (error centric, fault centric & failure centric) and thereby ensure that we have not missed any potential defects.
A failure centric approach demands that we wear an end-user hat and identify the potential failures that could cause business loss. The cleanliness
criteria formulated earlier could come in very handy as this would force us to think like a customer/end-user. What we trying to do is to ensure
that we have considered all the potential failures and therefore hypothesized the potential defects.
Now move to a user centric view to examine the various ways that an end-user could abuse the system by identifying the various ways errors
could be injected into the system. Not that an end user does not always connote a physical person, it could be another system that interacts with
the system via some interface. so examine the various points of interaction and look at the possibilities of their injection and then hypothesize the
potential defects that could get irritated by these errors. The kinds of information that could be useful here are: workflows, data access, interesting
ways of using the system, accessibility, environmental constraints faced by the physical end-user and potential deviant ways of using the system.
Then consolidate the potential defects and group similar ones into potential defect types (PDT). Finally map the PDTs to the various elements-
under-test i.e. feature/requirements. Now we have a clear notion as to what types of defects that we should look forward to uncovering in what
parts of the system.
35
36. Information extracted & artefacts generated
At each stage, certain information is extracted,
understood and transformed into artefacts useful
to perform effective & efficient testing.
Information
Data
Structure Artefacts
The key outcomes as demonstrated by the
Environment artefacts are:
HBT PD catalog
Stage #3
‣List of potential defect types
Business logic ‣Mapping between PDTs & the elements-
Fault traceability under-test i.e. Feature/Requirement
matrix
Usage
Attributes
Past defects In Stage #2, the focus is on hypothesizing
PDTs using the FIVE aspects of Data, Business
logic, Structure, Environment & Usage from
THREE views - Error-centric, Fault-centric &
Failure-centric.
36
37. Deliverables from Stage #3
PD catalog Should contain the list of potential defects and the potential defect types
Fault traceability
Should contain the mapping between the potential defect types/potential defects and features/requirements.
matrix
STEM Discipline applied in Stage #3
The STEM discipline “Defect hypothesis” of STEM is applied in this stage of HBT.
The STEM core concepts of “Negative thinking”, “EFF model”, “Defect centricity principle” and “Orthogonality principle” are useful in this
stage to scientifically hypothesize defects.
37
38. Stage #4 : Devise PROOF (Part #1: Test Strategy & Planning)
HBT being a goal focused test methodology, the intent is about figuring out an optimal approach to detect the potential of defects in the
system. Therefore strategy in HBT is about staging the order of defect detection, identifying tests that are needed to uncover the specific
defect types and finally choosing test techniques best suited for each type of test.
Typically we have always looked at the levels of testing like unit, integration and system from the aspect of the “size” of entity-under test. Unit
test is typically understood as being done on the smallest component that can be independently tested. Integration test is typically understood
as being done once the various units have been integrated. System test is typically seen as the last stage of validation and is done on the whole
system.
What is not very necessarily very clear is the specific types of defects that are expected to be uncovered by each of these test levels. In HBT,
the focus shifts to specific types of defects to be detected, and therefore the act of detection is staged to ensure an efficient detection
approach.
In HBT, the notion is of quality levels, where each quality level represents a milestone towards meeting the final cleanliness criteria. In other
words each quality level represents a step in the staircase of quality. The notion is to ensure that the defects that can be caught earlier is
indeed caught. So the first step to formulation of strategy is to stage the potential defects and thereby formulating the various quality levels.
However in HBT, there are NINE pre-defined quality levels where the lowest quality level focuses on input correctness progressively going
onto the highest quality level to validate of the intended business value is indeed delivered.
38
39. Stage #4 : Devise PROOF (Part #1: Test Strategy & Planning)
Understand scope
Having identified the various types of potential defect types to be detected at various levels, it is now
necessary to understand the specific types of tests needed to uncover these potential defects. In HBT each
test shall be intensely goal focused. This means that a type of test shall only uncover specific type of defects.
Choose quality levels
The act of test type identification results in specific types of tests to be done at each of the quality levels.
Now that we know what types of defects need to be detected when and where what type of tests, we need
Identify test types
to know how to design sufficient yet adequate test cases for each type of test. In HBT, a test technique is one
that allows us to design test cases. Based on the types of defects i.e. types of tests, we have to identify the
test technique(s) that is best suited for uncovering these types of the defects.
Identify test techniques
Now we have a clearer idea of various types of defects, the levels of detection, types of tests and test
techniques., we are now ready to identify the optimal detection process best suited for design/execution of
Identify detection process
test cases. The the act of identifying detection process also allows us to understand whether we need
technology support to be able to execute test cases and therefore pave the way for automation strategy.
Identify tooling needs
At this point in time we have a strategy and are ready to develop the detailed test plan. Some of the key
elements of the test plan is the estimation of effort and time and formulating the various test cycles. In HBT
cycles are formulated first and then effort and time estimated.
Formulate cycles
Finally potential risks that could come in the way of executing the test plan are identified and the risk
management plan put in place.
Estimate effort
In summary, a strategy in HBT is a clear articulation of the quality levels, test types test techniques and
detection process model.
Identify risks
39
40. Information extracted & artefacts generated
At each stage, certain information is extracted,
understood and transformed into artefacts useful
Information to perform effective & efficient testing.
Cleanliness
criteria
PDT
Artefacts
Attributes
The key outcomes as demonstrated by the
HBT Test strategy artefacts are:
Techniques
Stage #4 ‣Test strategy
‣Test plan
Deployment env. Test plan
Scope of work
#Scenarios
Risks
In Stage #4 (Part 1) the focus is on 1
HBT Stage identifying
the quality levels, test types, test techniques
and the detection process.
40
41. Deliverables from Stage #4 (Part #1)
Test strategy Should contain the quality levels, test types, test techniques & detection process
Test plan Should contain the test effort estimate, cycle details and the potential risk & mitigation plan.
STEM Discipline applied in Stage #4 (Part #1)
The STEM discipline “Strategy & Planning” of STEM is applied in this stage of HBT.
The STEM core concepts of “Orthogonality principle”, “Quality growth principle”, “Defect centered activity breakdown” , “Cycle scoping” are
useful in this stage to scientifically developing the strategy & plan.
41
42. Stage #4 : Devise PROOF (Part #2: Test Design)
The act of designing test cases is a crucial activity in the test life cycle. Effective testing demands that the test cases possess the power to
uncover the hypothesized potential defects. It is necessary that the test cases are adequate and also optimal.
In HBT the design is done level-wise and within each level test-type wise. Based on the level & type, the test entity may be different. The test
design activity for an entity for a type of test at a quality level consists of two major steps, firstly to design test scenarios and then generate
these test cases for each scenario. Test scenarios are designed entity-wise and therefore there is a built-in notion of requirements
traceability. In addition to requirements traceability, it is expected that the test scenarios and corresponding test cases are traced to the
potential types of defects that they are expected to uncover. This is termed “Fault traceability”.
42
43. Stage #4 : Devise PROOF (Part #2: Test Design)
Identify test level to design The act of test design commences with the identification of the quality level and then the specific type
consider & identify entities of test for which the test cases are to be designed. This allows us to identify the various test entities for
which test cases have to be designed.
Identify conditions & data Having identified the test entities it is then required to identify the conditions that govern the business
logic and the data elements that drive these conditions. Subsequent to this, build the behavioral model.
Use the behavioral model to generate test scenarios. Then for every scenario, identify the data that
Model the intended behaviour varies and then generate values for each data element. Finally combine the data values to generate the
semi-formally test cases.
Since we have designed scenarios/cases entity-wise, requirements traceability is built-in i.e. the designed
Generate the test scenarios scenarios/cases automatically trace to the entity (or requirement). Now map the scenarios/cases to the
hypothesized PDTs to build the fault traceability matrix.
For each scenario, generate Finally assess the test adequacy of the designed scenarios/cases by checking test breadth, depth &
test cases porosity.
Trace scenarios to PDT &
entity-under -test
Assess the test adequacy by
fault coverage analysis
43
44. Information extracted & artefacts generated
At each stage, certain information is extracted,
understood and transformed into artefacts useful
Information to perform effective & efficient testing.
Conditions
Artefacts
Data
Test scenarios &
Logic cases The key outcomes as demonstrated by the
HBT artefacts are:
Stage #4 Requirements ‣Test scenarios & cases
Structure ‣Requirements traceability matrix
traceability matrix
‣Fault traceability matrix
PDT Fault traceability
matrix
Defect escapes
Attributes
In Stage #4 (Part 2), the focus is on designing
test scenarios/cases that can be proved to be
adequate and have the power to uncover the
hypothesized PDTs.
44
45. Deliverables from Stage #4 (Part #2)
Test scenarios &
Should contain the test scenarios/cases for each entity for all types of tests at various quality levels
cases
Requirements
Should contain the mapping between the scenarios/cases and the entity-under-test
traceability matrix
Fault traceability
Should contain the mapping between the scenarios/cases and the PDTs
matrix
STEM Discipline applied in Stage #4 (Part #2)
The STEM discipline “Test design” of STEM is applied in this stage of HBT.
The STEM core concepts of Reductionist principle, Input granularity principle, Box model , Behavior-Stimuli approach, Techniques landscape,
Complexity assessment,Operational profiling, Test coverage evaluation are useful in design test scenarios/cases scientifically.
45
46. Stage #4 : Devise PROOF (Part #3: Metrics Design)
In this stage, the objective is to design measurements to manage the process of validation in an
Identify progress aspects effective and efficient manner. Since HBT is a good focused test methodology, it is necessary to device
measurements that enable us to clearly show the progress towards this goal.
Identify adequacy(coverage) The measurements in HBT are categorized into progress related measures, test effectiveness
aspects measures and system risk measures. Therefore it is necessary to identity the various aspects related
to progress, effectiveness and the system health.
Identify progress aspects Once the aspects are identified, key goals related to these are identified and then the metrics
formulated. Finally it is necessary to understand when to measure and how to measure.
For each of the aspects identify
the intended goal to meet
For each of these goals, identify
questions to ask
To answer these questions,
identify metrics
Identify when you want to
measure and how to measure
46
47. Information extracted & artefacts generated
At each stage, certain information is extracted,
understood and transformed into artefacts useful
to perform effective & efficient testing.
Information
Quality aspects
Progress aspects Artefacts
HBT The key outcomes as demonstrated by the
Process aspects Stage #4 artefacts are:
Metrics chart
‣Chart of metrics that are goal-focused
Organization goals
When & how to
measure
In Stage #4 (Part 3), the focus is on designing
metrics that will ensure that we stay on
course towards the goal.
47
48. Deliverables from Stage #4 (Part #3)
Metrics chart Should contain the list of metrics, collection frequency and a how this meets the goal.
STEM Discipline applied in Stage #4 (Part #3)
The STEM discipline “Visibility” of STEM is applied in this stage of HBT.
The STEM core concepts of GQM, Quality quantification model are useful in design metrics that are goal-focused.
48
49. Stage #5 : TOOLING support
Perform tooling benefit In this stage, the objective is to analyze the support that we need from tooling/technology to
analysis perform the tests. Automation does always imply scripting that is typically automating the
designed scenarios. It could also involve development of test bench, custom tooling to enable the
system to be tested.
Identify automation scope
This stage of HBT allows you to identify the tooling needs, understand issues/complexity
involved, perform cost-benefit analysis, evaluate existing tools for suitability/fitment and finally
Assess automation complexity devising a good architecture that provides for flexibility/maintainability before embarking onto
automation.
Identify the order in which
scenarios need to be automated
Evaluate tools
Design automation architecture
Develop scripts
Debug and baseline scripts
49
50. Information extracted & artefacts generated
At each stage, certain information is extracted,
Information understood and transformed into artefacts useful
Artifacts
to perform effective & efficient testing.
Automation Needs & benefits
objectives
document
Scope Complexity
assessment report The key outcomes as demonstrated by the
Scenarios to artefacts are:
automate Tooling requirements
‣The reason for tooling & automation
HBT ‣Challenges involved
Stage #5 ‣Requirements of tooling
Scenario fitness
‣Scope of tooling & automation
Automation scope ‣Architecture of automation
Technologies used ‣Automated scripts
Automation
architecture
Tool info.
Tooling & Scripts
Complexity info.
In Stage #5, the focus is on identifying tooling
requirements and building automated scripts
that is delivers value & ROI.
50
51. Deliverables from Stage #5
Needs & benefits Should contain the technical & business need for automation.
document
Complexity
Should contain the technical challenges of automation
assessment report
Tooling requirements Should contain the requirements expected out of automation
Automation scope Should contain scope of automation
Automation
Should contain the architecture adopted to building tooling/scripts
architecture
Tooling & Scripts The actual tools/scripts for performing automated testing
STEM Discipline applied in Stage #5
The STEM discipline “Tooling” of STEM is applied in this stage of HBT.
The STEM core concepts of Automation complexity assessment, Minimal babysitting principle, Separation of concerns, Tooling needs analysis
are useful in adopting a disciplined approach to tooling & automation and deliver the ROI..
51
52. Stage #6 : Assess & ANALYZE
Identify test cases/scripts This stage is where you execute the test cases, record defects, report to the team and take
to be executed appropriate action to ensure that the system/application is delivered on time with the requisite
quality.
Execute test cases, record
outcomes
Record defects
Record learnings from the
activity and the context
Record status of execution
Analyze execution progress
Quantify quality and identify
risk to delivery
Update strategy, plan,
scenarios, cases/scripts
52
53. Information extracted & artefacts generated
At each stage, certain information is extracted,
Artifacts understood and transformed into artefacts useful
to perform effective & efficient testing.
Execution status
report
Information Defect report The key outcomes as demonstrated by the
artefacts are:
Execution ‣Report of test execution & progress
Progress report ‣Defect report
information
HBT ‣Report on cleanliness aka quality
Defect Stage #6 ‣Learnings from execution resulting in
information Cleanliness report improved strategy, scenarios & cases
‣Any other key learnings
Context Updated strategy,
plan, scenarios &
cases
Key learnings
In Stage #6, the focus is on ensuring a
disciplined execution, intelligent analysis and
continuous learning to ensure that the goal is
reached.
53
54. Deliverables from Stage #6
Execution status Should contain the status of test execution
report
Defect report Should contain defect information
Progress report Should contain progress of execution and thereof the cycle
Cleanliness report Should contain the cleanliness index and how well the cleanliness criteria has been met
Updated strategy,
Updated strategy, plan, scenarios, cases based on learnings from execution
plan, scenarios &
cases
Key learnings Key observations/learnings that could be useful in the future
STEM Discipline applied in Stage #6
The STEM disciplines of “Execution & reporting” and “Analysis and management” of STEM are applied in this stage of HBT.
The STEM core concepts of Contextual awareness, Defect rating principle, Gating principle, Cycle scoping enable a disciplined execution,
fosters continual learning and stay focused on the goal.
54
56. Discipline #1 : Business value understanding
How to This discipline enables one to understand the system, create a baseline of features, attributes and
Understand a system finally expectations. This discipline consists of SEVEN tools, each of which uses certain STEM core
concepts to ensure these are done in a scientific and disciplined manner.
Landscaping | Viewpoints
Good quality implies meeting expectations. This requires that we understand expectations in
How to
additions to the needs as delivered by the requirements. Understanding the intended business
Create a functional baseline
value to delivered is key to this.
Viewpoints | Reductionist principle
How to
Create an attribute baseline
Viewpoints | Reductionist principle
How to How to
Identify focus areas Understand interdependencies
Value prioritisation | Viewpoints Interaction matrix
How to How to
Understand usage Baseline expectations
Operational profiling | Viewpoints Goal-Question-Metric | Viewpoints
56
57. Baseline provides the basis for future work
What is to be tested needs to be clear.
Remember Functional & Non-functional requirements?
Functional Baseline Attribute Baseline
Consists of list of features to be tested. The non-functional aspects.
Essentially a agreed upon list of features. Agreed upon attributes & their values.
57
58. Tools in D1 -Business value understanding
STEM Core
Tools Description
Concepts
System is viewed as a collection of information elements that are
How to Landscaping
interconnected. This tool enables you to come up with intelligent questions to
Understand a system Viewpoints
understand the various information elements and their interconnections.
Commencing from an external view of end users, various use cases
How to Viewpoints (requirements) are identified and then technical features that constitute the
Create a functional baseline Reductionist principle use cases. This tool enables you to clearly setup a functional baseline that is
used as a basis for strategy, plan, design, tooling, reporting & management.
In addition to functional correctness, it is imperative that the attributes are
How to Attribute analysis
met,. This tool enables you to identify the attributes and ensure that these are
Create an attribute baseline Viewpoints
testable.
All requirements/features are not equally valued by the end users. This tool
How to Viewpoints
allows you to rank the end users, requirements, features thereby enabling
Identify focus areas Value prioritisation
prioritisation of testing based on the risk and perceived value.
Understanding the real life usage profile is about knowing what operations,
How to Viewpoints #concurrent operations, rate of arrival are in progress at a point in time. This
Understand usage Operational profiling tool allows to arrive at the closer to reality potential usage profile of the
system to ensure effective non-functional tests.
Understanding how a feature/requirement affects/is-dependent on other
How to
Interaction matrix feature/requirements is useful to understand impact & re-testing effort. This
Understand interdependencies
tool allows you to rapidly understand the interdependencies.
How to Viewpoints
This tool allows to derive cleanliness criteria that reflect the expectations.
Baseline expectations Goal-Question-Metric
58
59. Customers & End Users
Customers in End users
Kids
Education
Seniors
Product Artists
Drawing
e.g Pencil
Draftsmen
Management
Corporate
Engineering
Admin
A product or an application may be sold in different market places made up of different kinds of customers.
Each class of customer may have different types of end users who use the product.
It is important to understand that each end user may have different needs & expectations.
Testing is about ensuring that the product will indeed satisfy the variety of needs & expectations
59
60. Needs & Expectations
NEEDS
Customers in End users Should write
Should have a eraser
Kids
Education EXPECTATIONS
Should be attractive
Seniors
Should be non-toxic
Lead should not break easily
Product Artists
Drawing
e.g Pencil NEEDS
Draftsmen Should write
Should not need sharpening
Management EXPECTATIONS
Corporate Thickness should be consistent
Variety of thickness should be
Engineering available
Variety of hardness should be
Admin available
Needs typically features that allow to get the job done.
Expectations are how well the need is satisfied.
Remember Functional & Non-functional requirements ?
60
61. Customer Profile
Customer #1 Customer #2 Customer #3 Customer #4
Different customers have different types of end users, and differing number of users for type of end user.
61
62. Customer Profile & Usage
How many What does each one use?
What types
users What is order of importance?
of users
What is the usage frequency?
F1
F2
F3
F4
System
F5
F6
F7
F8
Different end users may use the system differently in terms of what they use, frequency of usage and how they value each each feature.
62
63. Business Value
Ultimately end users need the system to do their job
BETTER, FASTER, CHEAPER and deliver value to their customers.
Understand that it is about “business value” of system - how does the system
help my business to do BETTER, FASTER, CHEAPER.
63
65. Discipline #2 : Defect hypothesis
How to This discipline enables one to hypothesise potential defect types that may be present in the system
Hypothesise defects under test and setup a clear goal approach to detection/prevention. Goal focused approach implies
that we map the hypothesised potential defect types (PDT) to the elements-under-test i.e feature/
Negative thinking | EFF model | requirements.
Defect centricity principle
This discipline consists of TWO tools, each of which uses certain STEM core concepts to ensure
these are done in a scientific and disciplined manner.
How to
Setup goal-focus
Hypothesis is done by scientifically examining certain properties of the system and can be
complemented by ones experience.
Orthogonality principle
65