‘early Model Based testing’ (eMBT)
An early model based exploratory testing approach in practice
Silvio Cacace, TestCompass, www.compass-testservices.com
“That looks very nice and it works fine. But sorry, it’s actually not what we wanted”
Do you recognize this? Off course you want to prevent this, because it indicates that you have delivered a product that might look nice and works fine but is not in accordance with the customers wishes.
Still 35% of all bugs in production can be traced back to the requirements. And these bugs are almost always related to poorly defined requirements or a lack of understanding around those requirements. To make sure we meet the customer's needs (business value), we need to achieve a shared clear understanding of the complete set of requirements. In addition, requirements change continuously, and it is therefore important to manage these changes effectively. ‘early Model Based Testing’ (eMBT) is a solid approach to solving these problems and challenges.
‘early Model Based Testing' (eMBT)
‘early Model Based Testing' (eMBT) is a software testing approach to ensures the optimization of the test case design phase. But more importantly, eMBT stimulates communication and collaboration between all stakeholders (business and technical stakeholders) with the aim of getting early feedback and a shared understanding of the requirements in an early stage of the SDLC.
eMBT consists of the following 4 phases:
Exploring
Review
Coverage
Checking
In this article, I aim to give some insight into the 4 phases of eMBT and how an eMBT tool can support this eMBT approach to testing with a representative example of an app ‘price calculator hardware store’.
Example eMBT approach
The following requirements are stated.
|
App price calculator hardware store A hardware store sells hammers (5 euro) and screwdrivers (10 euro). If a client spends more than 200 euro, then the client gets a discount of 4%. If a client spends more than 100 euro, then the client gets a discount of 20%. If a client buys more than 30 screwdrivers, there is an additional discount. |
1. Exploring phase
The first phase is the exploratory phase. During this phase, the goal is to explore and examine the requirements and ensure that there are no open ends, ambiguities, inconsistencies, contradictions, etc. And there must be a shared understanding within the entire team, both business and IT.
We start by setting up a test model in the eMBT tool based on all the information collected relating to the requirements. Ideally this phase starts as soon as the business (product owner) defines the requirements and would involve the development team. However, in practice it often happens that this phase begins later, in the refinement sessions or when the requirement is picked up by the development team, as a user story.
The test model in eMBT is a graphical model, based on the principles of a flowchart but with some specific features. The test model has a high level of abstractions so it can be easily understood by all team members, both business and technical. This promotes clear communication and collaboration regarding the requirements in a very early stage.
Flowcharts are a general-purpose type of graph that can be used for testing. Graphs are used in many ways in software engineering, and they come in all kinds of flavours and styles. They may differ in style of nodes, way to draw and connecting them, and notational conventions. But in the end, they all consist of a collection of objects (called nodes and drawn as circles, ellipses, or boxes) and relations between them (called edges and drawn as lines or arrows).
To draw the test model, the following 5 nodes can be used.
The following comment node can be used to add questions or comments regarding open ends, ambiguities, inconsistencies, contradictions, etc.
The first part of the model, based on the given example, will look something like this.
(click on figure to enlarge it)
When drawing the test model, you will find that that it immediately facilitates discussions about the requirements. It helps to spot inconsistencies in the overall understanding of the problem domain as well as highlighting areas of ambiguity which need to be addressed. As the discussion progresses you will be able update the test model which may in turn lead to further questions and refinements.
When using the decision node with an outgoing yes and no test path, one is forced to think about the happy and unhappy paths, and the boundary values. The open ends will become apparent straight away if there any.
The first part of the test model containing all questions, comments, etc. should now look something like this.
(click on figure to enlarge it)
2. Review phase
Before being able to complete the test model, the questions raised should be answered within the team. This avoids any individual assumptions being made and ensures the whole team has the same clear understanding of the requirements.
The advantage of bringing the business and the test team together for this phase is that they can immediately answer many of the questions and ambiguities as soon as they are raised and we can update the model with these answers when the whole team is present.
eMBT is a very flexible approach, and therefore we can give and ask feedback during several moments or phases. Sometimes a special eMBT-meeting is organized, comparable with the 3 or 4-amigo's meeting in the BDD process, in which several models are discussed at the same time. But in other situations, the feedback/review of the test model takes place in the refinement session or in a less formal setting. It is very common that several e-MBT meetings or feedback/review moments are necessary for the same requirements, it is an iterative and exploratory process. An iterative approach is necessary to capture changes in the requirements during the development process. A change in the requirements means a change in the test model and so feedback/review is needed once again.
In the next test model, all the answers to our questions, comments, open ends, etc. have been integrated. Of course, this is optional and depends on how the feedback/review is performed. Sometimes it can be more convenient to just write down the answers and comments and include them in the test model later, and have it reviewed again.
During the review phase, the product owner also announced a change in a discount percentage. The discount for a buy of > 200 euros should not be 4% but 5%.
(click on figure to enlarge it)
After all our questions have been answered and any changes, comments, ambiguities, etc. have been discussed and incorporated in the requirements, we can update and finish the test model.
The updated requirements look like this now:
|
App price calculator A Hardware store sells hammers (5 euro) and screwdrivers (10 euro). If the total is more than 200 euro, then the client gets a discount of 5%. If the total is more than 1000 euro, then the client gets a discount of 20%. If the client buys more than 30 screwdrivers, there is an additional discount of 10
* There is no stack discount |
The complete test model should look like this.
(click on figure to enlarge it)
We can check the model to see if we have included all business rules and decisions from the requirements. In the requirements of the hardware store, there are 3 business rules and 3 decisions.
Business rules:
Buy > 200 euro, discount of 5%.
Buy > 1000 euro, discount of 20%.
If buy > 30 screw drives, client gets an additional discount of 10%.
Decisions:
Buy > 200 euro
Buy > 1000 euro
Buy > 30 screw drivers
Because we also want to check the different boundary values, there must be 3 different
conditions per decision in the test model.
Now that we have checked whether we have included all business rules and decisions in the test model, we can have the test model reviewed again by the team to be sure that the test model is correct and complete and to make sure the whole team has a shared understanding of the requirements.
3. Coverage phase
When the test model is made, discussed and assessed by the whole team, the test cases can then be automatically generated, based on the agreed test coverage. The agreed test coverage can be the result of a previously performed product risk analysis where the various functionalities are categorized according to priority. Or on an ad hoc risk analysis, performed in collaboration with the business.
In the eMBT tool it is possible to generate the test cases automatically, based on a pre-selected test coverage form, from weak to strong. The following test coverage forms are available.
Node coverage
All nodes in the model will be tested at least once. The Action/State, Decision and Result symbols are classified as nodes.
Edge coverage
All edges in the model will be tested at least once. The Yes- and No-path links out of the Decision node are classified as edges.
Multiple condition coverage
All multiple conditions will be tested at least once. A multiple condition is a combination of 2 consecutive test paths (Yes- and No-path out of a decision).
Path coverage
All possible paths from Start to End will be tested.
After the agreed test coverage is selected, we can easily generate the test cases, which will be shown at the left side of the test model.
(click on figure to enlarge it)
11 test cases were generated based on our test model. The used test coverage is ‘Path coverage’, which means that we need to execute 11 test cases to test all the possible paths in the test model. The test coverage is 100% and is shown (100% Path Coverage).
At this point it is possible to deselect one or more test cases. This can be useful if, despite the agreed test coverage, we decide to test with a lower test coverage. After deselection of one or more test cases the percentage of the test coverage will decrease automatically. Therefore, it is always clear what test coverage we have achieved for a certain test.
When the test cases are generated, we can export them to an Excel test design, our compass for test execution. The text in the different nodes and any details entered in the test model, such as the preconditions and test data, will be automatically included in the test design.
It is also possible to model in Gherkin-syntax or to enter the Gherkin syntax in de details of the test model (in this example entered as details), which results in a test design including the ‘given-when-then’ steps. An advantage of this is that you can now derive the various Gherkin feature files from here which you can then use, for example, in the Cucumber framework.
The Excel test design for the first 4 of the 11 test cases looks like this.
(click on figure to enlarge it)
Impact analysis
As we know requirements change continuously and it is therefore important to manage these changes effectively. Not least to avoid false positives and false negatives.
One of the advantages of eMBT is, that if there is a change in the requirements, we only need to update the model. After the test model has been updated and reviewed again within the team (review phase), we can perform the impact analysis within the eMBT tool. Every change (functional and text change) in the test model will automatically lead to the possibility to run the impact analysis, which will show the 'new' status of the test case(s); added, removed, updated and unchanged.
4. Checking phase
The last phase is the checking phase. In this phase we execute the test cases in the system under test (SUT) and check whether the expected result is equal to the result in the SUT. Depending upon the characteristics of the test case, for e.g. time-consuming test, repetitive test (regression), hard to do test, smoke test and risk-related test, we can execute the test cases manually or write automated test scripts to execute them automatically.
Conclusion
35% of all bugs in production can still be traced back to the requirements. And these bugs are almost always related to poorly defined requirements or a lack of understanding around those requirements. To make sure we meet the customer's needs (business value), we need clear and complete requirements and for all members of the team to have a shared understanding of those requirements. In addition, requirements change continuously and it is therefore important to manage these changes effectively. ‘early Model Based Testing’ (eMBT) is a solid approach to solving these problems and challenges.
'early Model Based Testing' (eMBT) is a software testing approach that optimizes the test case design phase. But more important, eMBT stimulates communication between all stakeholders (business and technical stakeholders) with the aim of obtaining a shared understanding of the requirements in an early stage of the SDLC process. To get the best possible result of eMBT, you need a well-organized test process and the right eMBT-tooling that supports that process.
This article was originally published in two parts on Software Testing Magazine: https://www.softwaretestingmagazine.com/knowledge/early-model-based-testing-embt-an-early-model-based-exploratory-testing-approach-in-practice/ and https://www.softwaretestingmagazine.com/knowledge/early-model-based-testing-embt-part-2/
Related Software Testing and Quality Assurance Resources
Exploratory Testing: Finding the Music of Software Investigation
Metrics for Implementing Automated Software Testing
Click here to view the complete list of Methods & Tools articles
This article was published in June 2022