What is Session-based test management?

ession-based test management (SBTM) is a structured approach to exploratory testing. Testers get a charter (a specific area to test) and a timebox (a session length), explore freely within that scope, then document what they tested and what they found. It sits between rigid, step-by-step test cases and completely unstructured exploration.

At a glance

• SBTM structures exploratory testing into time-boxed sessions, each guided by a charter
• The four components are sessions, charters, note-taking, and debriefing
• It keeps the bug-finding freedom of exploratory testing while giving you progress and coverage to report
• Reporting quality depends heavily on testers' notes, so results vary from tester to tester
• Many teams land on something more pragmatic: a lightweight checklist that guides testing without dictating it

The rest of this post covers where SBTM came from, its four components, and how to decide whether it fits your team. To explain where it fits in the testing world, we first need to define a few other types of testing.

Traditional Test Case Management vs. Exploratory Testing

Let’s start with traditional Test Case Management or TCM. TCM is built around the concept of – you guessed it – test cases.

Test Cases are detailed instructions for completing a test, along with expected outcomes and any other data a tester might need to know or capture. TCM is most commonly used by enterprise companies with large QA teams. They need a more rigid structure so they can report on specific KPIs on a specific cadence.

There is a major downside to Test Case Management, though: it only catches the bugs you can anticipate.

To get around that, you might use exploratory testing, a form of testing that gives testers the freedom to test as they see fit. We like to describe it as testing with your brain switched on: deciding what to try next based on what you just observed.

Unlike TCM, testers aren’t following a set of instructions. They are modifying their tests on the fly, based on product behavior and what they’ve already observed. This adaptive approach to testing tends to unearth bugs or enhancements that may not have been found if testers stuck to a rigid list of steps.

By the same token, exploratory testing without any framing around it can be too unstructured.

What is session-based test management?

Session-based test management (SBTM) is a formalization of the more general idea of exploratory testing. Rather than letting testers choose what they want to do next, you assign them areas of an application to test for a certain period of time. The approach was created by James Bach and Jonathan Bach in 2000, and their original Session-Based Test Management paper is still the canonical description.

Why do we need this?

Because – like we hinted at above – without it, exploratory testing has no built-in way to show that everything that needs testing got tested. Testers just pick up your app and see what works and what doesn’t.

Exploratory testing is also limitless. When you ask testers how long testing will take, they reply, “Well, how long have you got?”

That wishy-washy response isn’t something you can share with your boss in a status meeting or with cross-functional partners so they can plan downstream work.

SBTM, on the other hand, ensures testers are confined to testing a specific thing for a specific amount of time so you can share specific results and timelines. At the same time, SBTM isn’t as restrictive as Test Case Management – testers still have the freedom to decide how to test, which can reveal unknown unknowns.

There are four core components of SBTM:

1. Sessions

Sessions are the length of time testers are to spend testing a piece of software. They can differ from feature to feature.

The idea is to budget time according to the complexity of the topic and the likelihood that there will be bugs in it. For instance, you may want to allocate longer sessions for new features with a higher probability of issues. Shorter sessions can go to established, stable, old features that haven’t changed.

2. Charters

Charters are basically a test plan with a time box. Testers are to spend a certain amount of time in each area of functionality, testing as they see fit. For example, your charter might be:

• Set up and login - 20 minutes
• Social sharing - 10 minutes
• Account deletion - 15 minutes

…and so on. When testers are done, you know that each part of your app has been investigated at least once by a human for a given length of time.

3. Note-taking

Extensive notes should be the output of sessions and charters. Testers should journal what they tried during their sessions (in accordance with the charter) and what happened as a result. These logs should clearly show what testers tested and what they didn’t.

4. Debriefing

Debriefs are a chance to analyze and act on testers’ notes. During the conversation, pull out identified bugs and potential improvements to share with the development team. Then, plan subsequent sessions and charters based on what is left to test.

Here’s how SBTM compares to the approaches either side of it:

Does SBTM fit into your testing strategy?

Let’s just start by saying SBTM is much, much better than no manual testing at all. And it’s a good option for those who see the value in exploratory testing but want a bit more structure. SBTM makes sure you’ve:

• Tested everywhere you need to (and didn’t accidentally forget whole chunks)
• Spent adequate time on each of those sections
• Stayed focused – sessions are similar to the popular Pomodoro productivity technique
• Documented what’s been tested and what hasn’t

Plus, SBTM keeps humans involved, letting them brainstorm their own things to test and react to what they notice is happening – a much broader type of testing than traditional Test Case Management.

If testing were reduced entirely to step-by-step test cases, a robot could perform all the tasks all in the same manner. Human testers introduce subjectivity, using their unique background and abilities to pinpoint issues or potential enhancements you wouldn’t have found otherwise.

But SBTM isn’t perfectly project-manageable. It can come with:

• A lack of consistent note-taking - Some logs might be more or less precise than others, some more pithy or less pithy than others, making it hard to know what, exactly, was tested. This could:

  • Impact the extent to which certain features were tested across testers
  • Extend the time it takes to debrief post-session
  • Make it harder to standardize internal or external testing reports

• Less predictable coverage - If a charter is too broad, testers might miss important features. For example, if a charter says, “Explore login for 10 minutes,” testers may or may not have remembered to check forgotten password functionality.

• Subjectivity - Yes, it’s a pro and a con... Human testers also introduce variability. How well a login screen is checked, for instance, depends on the experience and skill of the tester.

To recap, SBTM is a good option if you like the idea of exploratory testing but need more structure. For some teams, though, that structure isn’t quite enough.

Why isn't SBTM used more?

Though we don’t have data to back it up, our perception is that SBTM isn’t used much. Perhaps it’s because the “test case management Process” – with a capital P – is just easier to track.

You can hand testers a thousand detailed test cases that trace back directly to requirements. If all those tests pass, you can say all your requirements have been tested. You’ve got the plans and audit trail to prove it.

But is that really true?

Sure, your app may have passed single-point tests, but did your testers find all the new bugs in your product? Maybe. It depends on how well those detailed test cases predicted the right bugs.

In all likelihood, someone writing a boring test case probably won’t have thought to include edge case tests in advance. A human, by contrast, invents their own tests, adapting to what they see. They notice things that look off and chase them down.

So, how do you get the reporting and traceability you need without restricting a tester’s natural ability to find bugs?

The pragmatic approach

There is no one single best way to test. But if test case management is too confining, unstructured exploration is too hard to track, and SBTM – though an improvement over pure exploratory testing – is still too uncontrolled, the best way to get things done might be to strike a balance between the two.

To standardize notes, you could create a templated grid in a spreadsheet or tool like Testpad. This keeps your testers on track while giving them the option to add in more tests as they think of them.

• But make sure that pass/fails have specific criteria, as in, pass = “Yes, we looked at this, and there are no problems,” and fail = “This particular thing failed, and I have comments in the next cell over explaining what went wrong.”

To ensure testing coverage, you could make your charters more detailed.

• But don’t go so far as to make each item in the checklist a “check.” Stick to high-level prompts using words like “look at” so a tester has autonomy over how to do their testing.

To make your testing more objective, you could reuse your charters.

• But remember to iterate on them from release to release so testers review areas that were updated and remember to do regression testing.

Testpad is designed with this pragmatic approach to testing in mind, giving testers enough control to uncover the hard-to-find bugs while maintaining enough control over the process to make your manager happy.

Sign up for our 30-day free trial to give Testpad a try. Or, if you’d like to learn more about our take on testing, give our editorials a read.

If you liked this article, consider sharing