VESS19 - Generative-AI can accurately populate Vascular Quality Initiative procedural databases using narrative operative reports

Objectives: Participation in the Vascular Quality Initiative (VQI) provides important resources to surgeons, but the ability to do so is often limited by time and data entry personnel. Large language models (LLMs) such as ChatGPT (OpenAI) represent generative artificial intelligence (AI) products that may help bridge this gap. Trained on large volumes of data, the models are used for natural language processing (NLP) and text generation. We evaluated the ability of LLMs to accurately populate VQI procedural databases using operative reports.

Methods: A single-center retrospective study was performed using institutional VQI data from 2021-2023. The 50 most recent procedures for carotid endarterectomy (CEA), endovascular aneurysm repair (EVAR) and infrainguinal bypass (BYP) were analyzed using VersaChat, a HIPAA-compliant institutional version of ChatGPT. We created an automated function to analyze operative reports and generate a shareable VQI file using both gpt-35-turbo and the more powerful gpt-4. Application of the LLMs was accomplished with a cloud-based application programming interface. The outputs of this model were compared to VQI data for accuracy. We defined a metric as ‘missing’ if it was discussed by surgeons in < 20% of operative reports.

Results: 150 operative notes were analyzed, including 50 CEA, 50 EVAR, and 50 BYP. These procedural VQI databases included 25, 179, and 51 metrics, respectively. For all fields, gpt-35-turbo had a median accuracy of 84.0% for CEA (IQR 80.0-88.0%), 92.2% for EVAR (IQR 87.2-94.0%), and 84.3% for BYP (IQR 80.2-88.1%). There were 3 of 25, 6 of 179, and 7 of 51 VQI metrics missing from the operative reports, respectively. Some examples of missing metrics were patch manufacture in CEA (though patch type was always discussed, and discerned with 96% accuracy by the models), dressing type in BYP, and whether an imaging assistance technology was used during EVAR. Excluding these, the median accuracy rate was 95.5% for each CEA procedure (IQR 90.9-100.0%), 93.2% for BYP (IQR 90.2-96.4%) and 94.8% (IQR 92.2-98.5) for EVAR (Table). Across procedures, gpt-4 did not meaningfully improve performance compared to gpt-35 (student t-test p=0.97, 0.95, 0.85 for CEA, BYP, EVAR overall performance). The cost for analyzing 150 operative reports with gpt-35-turbo and gpt-4 were $0.12 and $3.39, respectively.

Conclusions: LLMs can accurately populate VQI procedural databases with both structured and unstructured data, while incurring minor processing costs. Increased workflow efficiency may improve center ability to successfully participate in the VQI. Further work examining other VQI databases and methods to increase accuracy are needed.