Machine learning for pharmacy triage
Predictive models with SHAP explainability to identify high-risk patients in more than 500 community pharmacies across Spain
Developing a decision-support system for pharmacy triage to identify which patients need referral to primary care
The Indica+PRO project held a very large volume of data on pharmacy consultations for minor ailments, but lacked analytical tools to help pharmacists identify the patients who needed a medical assessment.
As part of an international academic collaboration between the Universidad de Granada and the University of Technology Sydney, more than 500 community pharmacies in Spain were taking part in a national, protocol-based minor ailment service programme. The project had accumulated clinical data from more than 14,000 consultations across 39 categories of minor ailments, together with patient demographics, pharmacist characteristics and the referral criteria applied.
The technical challenge had two dimensions. The first was to build predictive models able to classify the need for referral to primary care correctly, in a setting with strong class imbalance and high clinical heterogeneity (39 symptoms with very different referral profiles). The second was to ensure the model was interpretable, so that the results were understandable and useful to pharmacists and clinical researchers, not just to statisticians.
14,000+
consultations included in the predictive analysis
39
categories of minor ailments assessed
500+
community pharmacies across Spain
An iterative predictive pipeline with triple variable selection, class balancing and SHAP explainability
Building the DataMart and triple variable selection
Integration of three databases (patients, pharmacists, pharmacies) into a unified DataMart with 61 candidate variables. A variable selection pipeline with three complementary methods — Boruta (random forest-based importance), LVQ (Learning Vector Quantization) and RFE (Recursive Feature Elimination) — retaining the variables identified as relevant by at least one of the three. Combining the three criteria maximised the capture of predictive signal without relying on a single selection criterion.
Training 9 algorithm families to predict referral to a doctor
Systematic evaluation of kNN, Naive Bayes, logistic regression, artificial neural networks, linear SVM, radial SVM, C5.0, Random Forest and XGBoost. Class balancing with SMOTE to counter the skewed distribution of referrals. All nine algorithms were trained to predict referral to a doctor, defined according to previously agreed clinical protocols, with stratified cross-validation and multiple metrics (Accuracy, Kappa, Sensitivity, Specificity, AUC). The radial SVM model achieved an accuracy of 0.934 with an AUC of 0.897.
Model-agnostic explainability and evaluation by clinical profile
Incorporation of SHAP (SHapley Additive exPlanations) to identify the specific weight of each variable in predicting referral. Complementary evaluation using 25 common clinical profiles (combinations of gender × age range × symptom × duration), calculating subpopulation-specific predicted probabilities. The key methodological finding was that including variables describing the pharmacist as decision-maker (experience, training, work setting) added relevant predictive capacity, showing that the professional's profile carries weight in the prediction, not just the patient's.
Evidence to support pharmacy triage with explainable artificial intelligence
93.4% accuracy in a test cohort of 4,218 consultations
The model was trained on 70% of the 14,083 records and evaluated on the remaining 30%, a test cohort of 4,218 consultations. The radial SVM model correctly identified 95.2% of actual negatives and 74.8% of actual positives, with a Cohen's kappa of 0.630 and an AUC of 0.897, using 20 predictor variables selected from a total of 61 assessed. In the stratified evaluation, performance was considered adequate in 14 of the 25 most common patient profiles, which reinforces its role as support for the professional rather than a substitute for their clinical judgement.
Pharmacist variables identified as relevant predictive factors
The SHAP analysis revealed that pharmacist characteristics (professional experience, type of training, pharmacy setting) carried considerable weight in predicting referral, a finding that is not obvious from a conventional clinical perspective. Identifying these professional and pharmacy-setting variables is directly applicable to any setting where the clinical decision depends on both the patient and the professional.
High-impact scientific output and international recognition
Publication in Frontiers in Pharmacology covering the radial SVM model, the variable selection and the evaluation by clinical profile. First prize for best presentation at the International Pharmaceutical Federation (FIP) Congress. SHAP explainability belongs to a later internal iteration of the pipeline, not to the published content. The model was iterated and improved over three methodological cycles during 4 years of collaboration, demonstrating the ability to support longitudinal research programmes.
What this case shows about the way we work
A full ML pipeline iterated over three methodological cycles
A demonstrable progression from caret with evaluation by clinical profile to tidymodels with SHAP, by way of incorporating NLP (topic modelling). The ability to improve an analytical system iteratively over time while keeping full traceability.
Triple variable selection to maximise predictive signal
A combination of three complementary methods (Boruta, LVQ, RFE) rather than relying on a single criterion. An approach that can be replicated in any predictive modelling project where feature selection is critical.
Interpretability from the model's first version
Evaluation by clinical profile from the model's first version, and SHAP explainability in the iterations that followed. The results are understandable to clinical researchers and pharmacists, not just to statisticians.
Identifying professional and setting variables
Empirical evidence that the pharmacist's profile, not just the patient's, adds predictive capacity for referral. A methodology applicable to any setting of assisted clinical decision-making.
Queríamos contaros una muy buena noticia: nuestro abstract que enviamos al Congreso de la Federación Internacional de Farmacéuticos fue seleccionado para una presentación oral de 15 minutos y ganamos el primer premio.
Emma L. Graham
Grupo de Investigación en Atención Farmacéutica · Universidad de Granada
Project resources
Resulting scientific output
Identification of high-risk patients for referral through machine learning assisting the decision making to manage minor ailments in community pharmacies
Emma L. Graham
Noelia Amador-Fernández
Capabilities deployed in this project
Shall we talk?
Tell us what you are working on and we will give you a straight answer on how we can help.