Making living documentation and test automation work for medical device software: your questions answered

What proportion of systems tests is it possible or typical to automate?

The proportion of test automation depends on the appetite for investment. But you would rarely have 100% test automation in an embedded system. Instead, we typically strive for 90% to 95%.

Even customers who’ve got legacy projects or are not sure about how much to invest in this, even just automating 30-40% of your tests, can add a lot of value. It could be the push you need to get further stakeholder buy-in and invest more.

How much time does living documentation save on average?

Living documentation, roughly speaking, reduces the time it takes to produce documentation from what was weeks of activity for the engineering team down to just a few days in the release cycle.

And on testing and test automation, we’ve seen it taken months of regression testing down to weeks.

Have you had any quality audits on these processes? And how have they gone?

Bluefruit Software is an outsourcing company for embedded software development, so we are not directly regulated ourselves, but our customers are regulated. We don’t get audited by regulators, but we do get audited by our customers. And usually, they are delighted to see our engineering team actively engaged with our processes and authoring documents. Our customers give us positive feedback when they see that we’ve thought about this, and that we’re doing it for real.

How do you reassure customers that shorter release cycles will not introduce low quality to a product?

We start by settling on a quality agreement with our customers. In creating this agreement, we’ll discuss which quality activities Bluefruit Software will handle.

Overall, it’s best to take a quantitative and qualitative approach to monitor quality in a project.

From a quantitative perspective, customers and software teams decide together on what sort of metrics to track, ensuring that the team aren’t using metrics to make decisions; they’re just using them for monitoring. Metric types can include:

• Code quality metrics (such as lines of code, complexity scores, comments)
• Test coverage
• Bug tracking

The key is agreeing good, healthy, metrics and making them available to the client. Having reports for stakeholders at the end of every release, every sprint and including the quality metrics that they’re interested in as part of that document, with added commentary as needed.

From a qualitative approach, there are code reviews, and where customers have engineers on their team, it’s essential to welcome and encourage them to be involved in the code reviews throughout a project.

It’s good to get early feedback. So, if the client is not happy with the quality, or even if it’s just a style issue with the code, it’s good to know early and accommodate changes as needed.

How long does it take to introduce living documentation or test automation to an R&D project?

The time involved depends on whether it’s an existing project or a greenfield project with no current systems or infrastructure, or customers to be concerned with.

With a greenfield project, you’re looking at two to four weeks to get the infrastructure set up alongside other tasks done during this time.

Setting up for test automation and living documentation would happen during a “ramp-up period”, early on in the project, where the rest of the work isn’t running at full pace yet in terms of requirements, but there might be early trials taking place with usability testing, prototyping and so on.

With a legacy project, you would not rush it. Instead, it’s best to ease in living documentation and test automation gradually. Here it would take at least a few months for the process to mature.

It would start with refactoring the test cases into a Behaviour-Driven Development (BDD) style, and then running the test cases manually so that test automation is introduced over time.

Within a few months, a considerable number of the tests could be automated so that stakeholders can come to see the value of it. And then it can be ramped once there is buy-in.

What’s the investment like to set up automated testing and living documentation?

It depends on what appetite there is. Often, customers who have seen it work before or who have got experience with seeing regression testing, taking months of work, they can immediately see the value of it, and they invest heavily at the beginning.

There’s a kind of a choice: You can do it in-house, but if you’re not familiar with it, it can be quite time-consuming. So again, outsourcing or involving people who’ve got the experience and got the toolkit already can help reduce that kind of cost of that investment.

The time to register certain software versions in some countries is becoming a limiting factor, what recommendations or tools can simplify the submission or review process?

The main thing is to be well prepared. That means expecting to need from the outset, high-quality documents, high-quality plans, and being able to evidence all of those.

Through having clear, available documentation, it means auditors looking at what you’ve got can see that it’s all there. They then don’t feel the need to do that full audit.

That’s going to be the best tip at making sure that you avoid a real, drawn-out audit.

Living documentation helps with this because it shows that the documents are contemporary.

Sphinx looks like quite a developer orientated tool, quite techie, is it possible to work with Office documents for the documentation, which colleagues on my team are more comfortable working with?

Those source documents will be an entire range of file formats, that you can convert to Office documents. However, an aim here, with living documentation, is to ensure we’re reducing the manual load of bringing these documents together.

Setting up the infrastructure for living documentation is a techie activity. But if you can get good processes around that, then it’s possible to bring together this mix of documents, even if your software development plan is in Word and your feature files are in your development team’s preferred format.

Using a build server together with explicit version control, that knows to get the correct version of those Word documents, is essential here. You want to be able to combine the documentation and convert to the format required.

It’s great that customers can get an updated version earlier, but what about the learning cycles? The user, mostly in the medical field, must be familiar with the machine and software itself? Isn’t it a contradiction to the short release cycle? What are your thoughts?

There can be challenges ensuring users can work with the product. But the key to handling this is developing a product that is as intuitive to use as possible.

How do you make an intuitive product? By making sure that you get feedback from users, early and often. You do this by taking a User-Centred Design approach to development and using various user experience (UX) research methods so that you can gain feedback early and often.

A technique our UX experts make a point of using is putting a product (it can even be a prototype) in front of the user and videoing them interacting with it, but not interfering. We see how they get on and record those interactions and find what they struggle with. We observe whether they deviate from expected behaviour.

The earlier you find out things that they may struggle with or get wrong, the sooner you can adjust for it in either the software or the hardware. There is value in finding out and failing early.

In doing all of this, you’re aiming for such a good user experience that they don’t need to be exceptionally familiar with a machine to use it successfully.

How do you decide BDD test cases? Do these come from requirements documentation?

It depends on the starting point, whether it’s within a corporate environment with an R&D team or a greenfield project.

For R&D teams working in a corporate environment, we often find that the leading company tends not to have an Agile mindset, but the R&D team do. So, typically there will be an existing user needs document, or requirements document, and that will be our starting point. And maybe the product specification will look like that. Here the BDD test cases will effectively be the software requirements.

For a greenfield project, we might push the client not to have a user needs document and use the BDD documentation to cover all these user, product and software needs. We are starting much earlier in the V model.

Regardless though, the earlier, the better when it comes to deciding BDD test cases

A note on Acceptance Test-Driven Development v BDD: in those situations where we’re starting with the user needs or product requirements already laid out; it would be more appropriate to call this Acceptance Test-Driven Development (ATDD). Why? Because BDD is about using the Agile feedback loop to discover the requirements. If that document already exists, that’s a barrier to a working, full BDD process and it’s more accurate to describe that as ATDD.

Are there any commonly used test automation frameworks, script-based, that you would recommend?

Generally, the test automation framework you choose will depend on what technologies are being used on the project. What hardware and software are you using?

For our teams, we like using a Python framework called “Behave”. The idea around Behave is to take a set of Gherkin feature files (Gherkin is a way of writing BDD acceptance tests) and automate those tests using some Python code behind the scenes. Behave is just the “runner” of those tests and doesn’t do a lot more than that.

There are a set of tools that we use to test specific technologies end-to-end. For example: Selenium for the web, Squish and Funq for Qt, and Appium for WPF (C#). But it really varies based on the project and the technologies involved.

Watch “How to achieve shorter release cycles for medical devices”

If you missed my full webinar session which looked at how living documentation and test automation can help medical device product teams, you can watch it below. The session covered how to follow guidance from AAMI TIR45 for meeting IEC 62304 and ensure that processes and practices were truly Agile or Lean-Agile.