In his webinar “How to achieve shorter release cycles for medical devices”, Paul Massey (Bluefruit Software’s Founder and Director) talked about how test automation for embedded systems can help with reducing release cycle times for medical device software.
So, what is automated testing in embedded software and how can it help your team?
Well, it’s more than just having a set of software and hardware solutions to help automate the testing of medical device software. So, first up, we need to understand where it has the best potential to flourish.
Automated testing in medical works best in Agile and Lean-Agile software teams
Embedded software teams who are actively working in an Agile or Lean-Agile way are usually engaged in practices that are ready for introducing test automation.
What do we mean by Agile or Lean-Agile? Here at Bluefruit, we see Agile as three layers with Lean software development being a part of it:
- Principles: these Lean-Agile principles put adaptation over prediction and have seven core ideas, such as eliminating waste, amplifying learning, empowering the team and more. (These principles are based on the principles discussed in Mary and Tom Poppendieck’s book: Lean Software Development.)
- Project management practices: these are things like sprints, retrospectives, points, user stories, and so on. Often other businesses who see themselves as Agile are already focused on these practices and are using them.
- Technical practices: these are things like frequent/continuous integration, Test-Driven Development (TDD), Behaviour-Driven Development (BDD), and so on. These processes are highly centred on achieving quality in software and are focused on ensuring effective software development. It is these technical practices that are often supportive of test automation.
But what does Agile or Lean-Agile look like in software development for medtech? Ideally it should look something like AAMI TIR45; if you’re planning on getting test automation working in a way that maintains compliance within IEC 62304 and reduces release times.
AAMI TIR45 in medical device software
TIR45 is a guidance document that teaches Agile product and software teams about compliance, but it also teaches compliance people about Agile.
At its core, it has a series of stages—project, release, iteration/sprint, and story. It basically looks like this:
TIR45 recommends that you decide which activities you’re going to do at which layer, allowing for a degree of flexibility. It asks that your software development plan records those decisions and spells out which activities are happening and when.
Working to TIR45, we can carry out test automation in a way that not only helps to decrease release cycles, but also maintain compliance. Here’s how.
Automated testing in embedded software for medical devices
As we’ve said, teams working in an Agile or Lean-Agile way will likely be using technical practices that are already ripe for automation.
For instance, at Bluefruit Software we use Behaviour-Driven Development (BDD). With BDD we can take advantage of:
- BDD feature files (also known as specifications by example).
- Specs and tests that are one artefact, which keeps tests synchronised.
- Traceability, showing how user stories/requirements have become part of software.
By having the above artefacts, your build server can automatically pull the correct versions of feature files into a test framework which is then pushed through to the target platform.
Then, you have a test rig which is either hardware or software focused, or both, that simulates the target platform. From all of this, out pops your test results. The process looks something like this:
What does this mean for release cycles? Being able to run constantly and automatically most integration and system testing means that testers are freed up, and have more time to do exploratory testing and usability testing—activities that add real value to the project.
The idea with test automation being used while following TIR45 is that your software team can move from this typical testing situation:
Interested in automated testing? Here are a few more considerations.
A few recommended script-based test automation frameworks
The test automation framework you choose depends on the software and hardware being used in a project.
At Bluefruit, our teams often use a Python framework called “Behave”. With Behave you can take a set of Gherkin feature files (Gherkin is a way of writing BDD acceptance tests) and then automate those tests using some Python code. Keep in mind that Behave is just the “runner” of those tests.
There are a set of tools that we use to test specific technologies end-to-end. For example: Selenium for the web, Squish and Funq for Qt, and Appium for WPF (C#).
What you decide on using will vary based on your project and the technologies involved.
The challenge of picking BDD test cases
How do you decide BDD test cases? It will depend on your starting point, whether it’s within a corporate environment with an R&D team or a greenfield project and how much Agile is already accepted and implemented.
If you’re already working in an Agile or Lean-Agile way with BDD then you will likely have the BDD test cases best suited to your project.
But what if you or organisation isn’t Agile?
R&D teams working in a corporate environment
Working in an Agile R&D team within a company that does not to have an Agile mindset? Look to the existing user needs document, or requirements document as a starting point. The product specification may also be a good place to start. In this instance BDD test cases will effectively be the software requirements.
Working in a greenfield project
Consider not having a specifications document and instead use the BDD documentation to cover all your user, product and software needs. When in this scenario, you’re starting much earlier in the V model.
Existing user needs or product requirements? It’s ATDD not BDD
In scenarios where you’re starting with the user needs or product requirements already laid out, this would be Acceptance Test-Driven Development (ATDD). Why the distinction? Behaviour-Driven Development is about using an Agile cycle of feedback to discover the requirements.
If that requirements document exists ahead of time, then that is a hindrance to a working, full BDD process and so it’s more accurate to describe that as ATDD.
The earlier the better
Regardless of the situation the team is starting from, the earlier, the better when it comes to deciding BDD test cases.
What level of testing automation is possible?
While it’s not possible to achieve 100% automated testing, we’ve found that for new products it’s possible to automate between 90-95%. The figure drops for legacy systems but between 30-40% test automation is possible, which would have a sizeable effect on release times.
The main challenge to succeeding in test automation is investment and the appetite for it. If you’re approaching it from scratch, in-house but want to see benefits sooner it would be worth outsourcing or bringing on people who have the experience and toolkit ready to go.
Watch the webinar
If you didn’t get the chance to watch “How to achieve shorter release cycles for medical devices”, you can watch the full webinar below:
Does your latest project need a AAMI TIR45 savvy embedded software team?
Bluefruit Software has over 20 years of experience providing high-quality software development, testing, and consulting to clients in a range of sectors including medtech. In the past five years we’ve worked on seven different medical devices, providing software development and testing to help our clients create products that meet IEC 62304. Want to talk?