Essential Daifuku Technologies for Social InfrastructureReducing Pharmacy Workloads with AI-Powered Prescription Auditing Systems

Reducing the risk of dispensing errors for drugs with similar names and packaging

At pharmacies, when pharmacists provide prescription medications to patients they will commonly pick the drugs from the shelves and rely on visual inspection to ensure that they have selected the correct medication.

The “audit” series of prescription audit systems identify the type and quantity of a drug using images and weight, cross-referencing this data against prescription information to check for errors. The system also automatically creates records of these audits. By simply placing the drugs on a tray, the system captures an image with its internal camera and pulls up the receipt data. The system accurately identifies the types and quantities even for drugs that are bundled together.

Development of the audit series began more than 10 years ago. Kenji Takahira, Project Coordinator of the New Business Development Project Group, received a product planning proposal from Daifuku and began conducting interviews with pharmacies. Through these interviews, he learned that the dispensing error rate was about 0.25% when conducted by visual inspection. Meanwhile, in the logistics industry, new technologies and equipment were reducing errors to practically zero, leading Kenji to recognize that there was ample room for improvement in pharmacy operations.

Kenji Takahira, Project Coordinator for Contec’s New Business Development Project Group

“Even though pharmacies and logistics hubs are different in terms of industry and scale, there are many workflow similarities, such as picking, inspection, and distribution. We felt that Daifuku’s expertise could help reduce the risk of prescription errors. At first, we tried to perform audits by cross-referencing images of the drugs and barcodes against master data, but the unique packaging of drugs presented various challenges, such as reflections and transparent bags,” recalls Kenji.

To deal with these challenges, the team added a space inside the device for capturing images of the medicine with uniform lighting, increasing the accuracy of image recognition of both the medicine itself and of the barcodes printed on the packaging. “By combining the image data with information from the built-in scale, we were able to ensure the type and quantity of a drug prescription could be identified at the same time. Although other companies also offer image-based recognition systems, only the audit series simultaneously identifies multiple drug types and quantities,” says Kenji.

Some drugs have similar names or packaging, and changes in packaging design and the addition of generic drugs have increased the risk of errors, resulting in increasing mental burdens on pharmacists, who must remain ever vigilant to avoid mix-ups.

According to Heinrich’s Law, every serious accident is preceded by 29 minor accidents and 300 non-injury accidents (near-misses). As such, various industry associations are actively collecting and analyzing near-miss accidents to help promote medical safety. Still, despite careful attention, human errors can never be entirely eliminated, resulting in cases of drug mix-ups and missing prescriptions. The audit series reduces the risk of errors and eases the mental burden on pharmacists while also providing audit records for when patients report discrepancies such as incorrect medicines or quantities.

The Japan Ministry of Health, Labour and Welfare has also recently been encouraging pharmacies to enhance patient-oriented services over task-oriented services, including explaining prescription details, providing guidance on longer prescriptions, and confirming prescription status. Streamlining the auditing process gives pharmacists more time to focus on communicating with patients.

Using difference extraction technology to improve AI image recognition

Some 2,000 units of the previous audit series model, released in 2012, have been installed in pharmacies across Japan. As the variety of drugs handled by pharmacies has increased year by year, there has been a growing demand for smaller devices. However, the technology at the time meant dedicated space was necessary for imaging, limiting how small devices could be designed. Kenji saw the potential of AI as a solution to this size reduction issue.

Yusuke Yokoyama, Supervisor of Contec’s Applied Development Group, Technology Division

At around the same time, Yusuke Yokoyama, a supervisor in the Applied Development Group in Contec’s Technology Division, began researching AI for a completely different reason. “The company was abuzz with discussions about AI applications as we started looking into using AI-based image recognition technology for detecting defects and assembly errors in electronic circuit boards. In the spring of 2020, Kenji approached me about the possibility of using AI to identify pharmaceuticals. I first learned about an open-source AI that could distinguish between 9,000 different image types, but it could not differentiate between drugs with similar packaging, only clearly distinct objects like people and dogs,” recalls Yusuke.

In search for a solution, Yusuke researched various image search technologies. “While searching for a way to distinguish between drugs with similar packaging, we found there was a high possibility that technology for extracting differences in images could be applied, and this change in thinking ultimately led to the breakthrough we were looking for,” says Yusuke.

Shota Kawase, member of Contec’s Applied Development Group, Technology Division

Shota Kawase of the Technology Division’s Applied Development Group joined the project in the spring of 2021.

By this stage, the foundational AI technology for identifying the correct drug from among tens of thousands had already been established. However, there were still some recognition issues with the images captured by the system’s camera. “During image processing, sometimes the lighting conditions made it impossible to recognize the drugs correctly, or if the tray used for imaging was the same color as the drug’s packaging, the two would blend together and make recognition too difficult. We used AI to resolve this by identifying the visual characteristics of the drug, separating the drug from the background, and extracting only the image of the drug itself. This significantly improved recognition accuracy, enabling identification of targets regardless of lighting or background color,” explains Shota.

The COVID-19 pandemic meant there was a lot of working from home during the system’s development, but this led to surprising benefits. “I think that the fact that we could test the image recognition not only at the office but also at home, in different and diverse environments, was one of the reasons for our success this time,” says Shota.

Whenever packaging designs change, or new or generic drugs are introduced, the AI must be updated with the new information. This additional learning for a standard AI that directly replaced the image processing function was found to take between 8 and 12 hours for each learning session. “Pharmacies don’t have that kind of time to wait. So, we developed a new process wherein we limited the AI processing to the main strengths—capturing external features such as the shape and packaging design. During the additional learning, only the extracted features and drug name are registered, speeding up the process significantly. Drug identification now takes no more than 1 second, and additional learning takes no more than 3 seconds. We have patented this new method,” says Kenji.

With the previous model, pattern matching was used to compare the captured image with the master images. This means that the imaging device and environment must be the same as the master image. With AI-based feature extraction, however, the imaging environment does not have to be the same and learned data sets can also be more easily shared online.

With the adoption of AI, the audit-i system offers significantly improved performance with relaxed imaging condition requirements and a 60% reduction in the overall size of the system.

The ability to connect the system to the internet is another significant change, enabling the use of various cloud services. Cloud-based centralization of audit records can be incredibly meaningful for companies managing multiple locations, with the data being useful for pharmacy management, planning, and staff training. “Being able to analyze trends for drugs that are prone to near-misses and the relationship between a pharmacy’s location and their best-selling products can benefit pharmaceutical manufacturers in various ways. By expanding our services to more locations, we hope to create a better working environment for pharmacists and a more convenient system for patients across the country,” says Kenji.