[TSTIL] QMS Integrity Checker
[ This is a part of "The Software That I Love", a series of posts about Software that I created or had a small part in ]
2020 - QMS Integrity Checker
When we set out to get external certification for our product, we were in a bit of a bind. There's a whole bunch of international standards that have to be followed, and we had almost no experience with these. For Software Medical Devices there's at least ISO 13485, ISO 14971 and IEC 62304. The ISO 13485 is interesting, as it's a set up of requirements for forming a "Quality Management System" (QMS).
Why is it interesting? Well, there's many ways of achieving high quality products, and I believe strongly in "move fast and break things". That's a good approach in general and many companies are successful with it (Facebook, SpaceX). It takes guts though. The failures will happen, and you need people with a high risk-tolerance to accept them. One of the reasons SpaceX can pull it off, but NASA can't, is that the tax-payers in the US would probably not accept many failures, and Elon does. It kind of makes sense to say that pacemakers should not be built this way, but then again, you could say the same about rockets.
It's safe to say that most medical devices don't take a "move fast and break things" approach. Rather the opposite. They try to prevent failure by over-engineering up-front as much as possible. For most people from the software world this is a bit of a shock. For me it certainly was. I had no idea what a QMS was, but we did have to build one.
You can not read the rules for a QMS for free, you have to buy the document online for a hefty fee. When we found the standard for relatively cheap at evs.ee we could finally read how to set up a QMS. Or so we thought. It's ironic that the 13485 standard exists to make you explain your processes and records in writing, but the standard itself does not contain any explanations. It just says "do X, do Y, do Z". Never "Rule X exists because manufacturers often have unclear descriptions of responsibilities within the organization, so it's unclear who can sign off on certain decisions". No, it just says "The organization shall document roles and responsibilities." It's a bit like reading a compiled program, stripped of all comments or useful variable names. This gives you a lot of freedom for interpretation, but no clear path forward.
Once we had more or less deciphered what we had to do, we figured out that a QMS is a bit like a computer program. There are users, roles, processes with inputs and outputs. It's super systematic. With our limited understanding we set out to build a computer program to contain our procedures, technical documentation and records. It would publish everything as a static website with a git-based database. This was fun and pretty cool to build, but we kind of got lost in the engineering details instead of the QMS details. We also realized that we'd need engineers to operate every part of our QMS because it was using git. That was not so good, because it excluded a large part of our company from contributing. We could also not download all html pages and submit them as PDFs to an auditor. This was very bad. A couple of weeks before we had to submit our QMS for external certification, we reverted back to the industry standard approach. That is A4 documents in Google Docs, then printed out and put signatures on them. Terrible, but it got the job done. We got certified with this system in 2019. We abandoned the QMS static page program we had built and loved.
The systematic approach was still there though, hidden within our Google Docs and Sheets. In the little spare time that I had I wanted to see if I could use Google Drive, Docs, and Sheets APIs to make a sort of "integrity checker" for our QMS. I had a lot of fun to talk to the APIs and parse the document structure in a Google Doc. At the end I even had a Google Docs to Markdown converter. What's more, I was able to parse all the roles, responsibilities, cross-links to other documents and much more. E.g. if you refer to a "Product Owner" in a procedure, but you haven't defined that role within your organization, it will throw an error. It scans hundreds of documents in a couple of minutes. Although it's still in its infancy, I run the integrity checker every now and then and point my colleagues to tiny mistakes in their documents. That's pretty fun and I look really smart. In theory this program combines best of both worlds: your regulatory people deal with Google Docs but the QMS can still be completely checked automatically using software. Might be a cool business idea actually. The plan was to incorporate this in some kind of CI/CD or monitoring system, but I never prioritize it and it just runs on my laptop. It's written in Python (what else) and the code is ugly but I love it.
I personally struggled a lot while getting certified, it was probably the most stressful period of my life. Afterwards I found out that we are not alone, many other software companies are struggling with the same things, and we could really help each other out. I got in touch with Oliver (hi!) and he set up OpenRegulatory.com with it's Slack group and I started a bi-monthly virtual SaMD meetup. Both of these communities are thriving! Now, if you have any questions, just ask on Slack and experts will point you in the right direction right away. I dearly wish I had access to this in 2019. It's not a software solution, but I'm extremely proud of starting the community.
Now that I'm a bit further in the regulatory field, I've also discovered that the rules do allow for a bit of "move fast and break things". The regulators realized that not everything can be checked beforehand, and rely on post-market surveillance to compensate. This catches problems that were not detected before. The feedback, CAPA and vigilance processes as described in ISO 13485 force you to learn from mistakes, and explain the serious problems to authorities. Realizing this made me feel very wise. Like a grown-up software rebel that realizes the entire world was thinking the same thing all along. Here's a quick shout-out to the people thinking about Chesterton right now.
previous: 2020 - easee TeleVisus
Comments
Post a Comment