Ensuring Regulatory Compliance in the HCLS Sector with Generative AI

Manual Review of FDA Form 483 Observations Is Too Slow, Inefficient, and Error-Prone

PSC Biotech has been helping companies in the Health Care & Life Sciences sector to provide the highest level of service, while complying with all applicable regulatory requirements for over two decades. The line of services offered by PSC to its clients is extremely diverse, and most of them involve a certain amount of document work — collecting, processing, analyzing, and managing various documents that are streaming through their client network.

PSC is an established business whose document processing operations have been set up for years.

However, most of the processes in the pipeline were manual, meaning that:

• More time and resources were needed to process documents
• Costs of processing were high, and perpetually rising over time
• Risks of errors due to human factor were ever-present
• The throughput rate depended entirely on the number of employees
• The accuracy of document processing remained stagnantly low, and fluctuated significantly over time

Given the sensitive nature of the HCLS business operations, any process that is slow, inefficient, and, most importantly, prone to errors poses a huge risk — the health and well-being of millions of people, as well as a company’s bottom line, could be impacted by a single mistake made by a document reviewer.

Consider FDA Form 483 observations. An FDA 483 observation is a notice sent by the FDA to highlight any potential regulatory violations (e.g. processes, controls, products, employee practices, etc.) found during a routine inspection. The cost of such an observation can be extremely high, but failure to introduce the required changes on time is even more costly. The resulting fines can cost any HCLS business millions or even billions of dollars.

PSC Biotech handles thousands of FDA Form 483 observations per year. Its document mappers and reviewers had been doing their best, but the need to streamline and scale the document processing pipeline was long overdue. This could be achieved by adopting AI/ML and implementing a Generative AI-powered solution for supporting document processing operations.

By augmenting the processing of documents with AI/ML & Generative AI, the leaders of PSC expected to decrease time spent on manual review of observations, decrease costs of form processing, mitigate risks of infractions made by mappers and reviewers, increase throughput rate, and increase the accuracy of document processing.

Provectus, an Artificial Intelligence (AI) consultancy, has developed and delivered a comprehensive suite of AI/GenAI solutions, including the Intelligent Document Processing solution. Given our expertise in the provision of AI/ML & Generative AI services, Provectus was chosen to design, build, and implement a Generative AI-powered solution for automated classification of FDA Form 483 observations for PSC Biotech.

Building a GenAI-powered Observation Classification Solution: From a Baseline Model to ML Infrastructure

Provectus approached the project via a series of engagements that involved data, model work (training, deployment to production, retraining), infrastructure and pipelines, logging and monitoring components, and a UI for document management.

To begin with, Provectus explored and prepared the data, set up all required environments (development, management, experimentation), established a baseline for text classification, and developed the observation classification model. A secure and reproducible, end-to-end machine learning infrastructure for experimentation and model training was also delivered as a part of the phase one engagement.

Provectus took advantage of various tools and services, including (but not limited to):

• Amazon CloudFront — Delivery of the user interface
• Amazon RDS — Storage of document- and model-related data
• Amazon Cognito — User authentication and resource access control management
• Amazon API Gateway, AWS Lambda, Amazon SQS, and Amazon DynamoDB — Backend implementation
• Amazon Textract — Optical Character Recognition (OCR) engine to process PDF documents
• AWS Step Functions — Orchestratration of services for building and updating the applications
• Amazon S3 Intermediate storage of texts, document observations, and model predictions
• Amazon SageMaker — ML/MLOps Infrastructure with pipelines for CI/CD, logging, monitoring, and model retraining
• Transformer-based encoder models, Deep Learning and Natural Language Processing (NLP) algorithms to extract and classify observations more accurately
• Experimentation frameworks: PyTorch, Tensorflow, NLTK, etc.

The Provectus team investigated several approaches to develop a Generative AI-powered solution that can accurately extract and classify information from observations. This exploration involved a comparative analysis of various transformer-based encoder models (such as BERT), alongside deep learning and natural language processing (NLP) algorithms. The final solution was designed as a multi-label classification model, capable of categorizing FDA Form 483 observations into over 100 labels, while maintaining precision and recall rates of 70% or higher. Observations were automatically labeled, simplifying their organization into different categories.

Next, the Provectus team developed the ML API service and enhanced the ML release cycle. The further stages of delivery of pipelines, including those for CI/CD, logging and monitoring, and model retraining, were prioritized. The ML infrastructure for the production environment was set up, building on the already existing, robust foundation.

Finally, Provectus set out to deliver:

• A user-friendly UI for document processing, to enable the employees of PSC Biotech to map and review forms more quickly
• A service for smooth integration of the machine learning component into the existing document processing pipeline
• A CI/CD pipeline that could retrain the existing model and push it into production when it achieved higher precision and recall.

The advancements in Generative AI, particularly through Amazon Bedrock, have the potential to further enhance PSC Biotech’s solution. Originally, the solution was designed for seamless integration with Amazon Bedrock’s model invocation API, ensuring that its feature extraction from FDA Form 483 observations is flexible, adaptable, and also ready to utilize the cutting-edge capabilities of any foundation model directly. The variety of foundation models accessible via Amazon Bedrock provides a comprehensive range of capabilities, positioning the solution to detect even subtler nuances in regulatory texts. The multi-label classification model can be tuned to achieve unmatched precision in categorizing observations. Utilizing the foundation models enhances the labeling automation process, enabling the solution to exceed the original precision and recall benchmarks — potentially increasing from around 70% to up to 95%. This strategy facilitates ongoing refinement of the categorization process and gives document mappers advanced search capabilities, positioning PSC Biotech’s Generative AI solution at the cutting edge of regulatory document processing technology.

Transforming Document Processing with Generative AI Leads to Fewer Errors, Faster Processing, and Cost Savings

PSC Biotech is a trusted provider of services in Health Care & Life Sciences. It is critical for PSC to be able to process FDA Form 483 observations as quickly and as accurately as possible, to ensure their clients stay up to date in introducing necessary changes to their products, services, and operations.

Accelerating and scaling the document processing pipeline with Generative AI is considered by PSC Biotech to be a key step towards meeting their goals of providing the highest standard of HCLS services. Provectus helped PSC on their path to excellence by contributing our expertise in AI/ML, Generative AI, and MLOpse.

The Provectus team designed and built a Generative AI-powered observation classification solution from scratch. We ensured that the model returns precision and recall of no less than 70%, helping PSC to automate a significant portion of the labeling, mapping, and review of observation forms. We also delivered an entire ecosystem around the model, with ML infrastructure, CI/CD pipelines, monitoring and UI components, and more.

The Generative AI solution has enabled PSC Biotech to accelerate document-centric operations by 90%, reduce document processing costs by 44%, and increase document throughput tenfold. It has led to a substantial acceleration in time-to-compliance, saving 5,000 man-hours per year, and enabling PSC Biotech to achieve an estimated return on investment (ROI) of 93% over 12 months..

PSC Biotech is now ready to pursue other opportunities in the AI/ML space, and Provectus will remain its trusted advisor. We look forward to future collaboration with PSC, to further empower their delivery of high-quality consulting services in the HCLS sector.

Moving Forward

1. Learn more about the Provectus Intelligent Document Processing (IDP) solution
2. Find out if your organization is ready to adopt Generative AI with Generative AI Readiness Assessment with Provectus
3. Building your first Generative AI Proof of Concept with Generative AI by Provectus