Integrating AI Predictions With Clinician Expertise

Conditions

• Diagnostic Decision Making

Eligibility Criteria

Sex

ALL

Age

18 Years - N/A

Healthy Volunteers

Accepted

Interventions

• Bayesian-Updated Post-Test Probability — BEHAVIORAL
  Rather than presenting the AI model's raw predicted probability, the system takes the clinician's pre-test probability (entered before seeing AI output) and applies a continuous likelihood ratio (CLR) derived from the AI model to calculate a Bayesian-updated post-test probability. The output is displayed as a shift from the clinician's own assessment (e.g., "Your assessment: 45% -\> Updated assessment: 72%"). The raw AI prediction is not shown. This approach mirrors how clinicians use diagnostic test results such as D-dimer to update pre-test probability of pulmonary embolism.
• Standard AI Predicted Probability — BEHAVIORAL
  AI model prediction is presented as a simple predicted probability (0-100%) for each of the possible diagnoses, together with the top 3 clinical features driving the prediction (e.g., "Acute Myocardial Infarction: 68% - Key factors: elevated troponin, ST-segment changes on ECG, chest pain radiation to left arm"). This represents the most common current approach to presenting AI-based diagnostic predictions in clinical settings.
• Uncertainty Quantification (95% Confidence Interval) — BEHAVIORAL
  The AI output (whether Bayesian-updated post-test probability or standard predicted probability) is presented together with a 95% confidence band displayed as error bars on probability bars. For accurate AI predictions, confidence interval width is approximately +/-12-15 percentage points. For misleading AI predictions, confidence intervals are widened by a factor of 1.5x (approximately +/-18-23 percentage points) to simulate reduced model confidence in unfamiliar or edge-case scenarios. Confidence intervals are constrained to the 0-100% range.

Study Details

Optimizing the interaction between the human and the machine is a major topic when deploying artificial intelligence (AI) at the bedside. The goal of this randomized clinical vignette study is to learn if presenting AI model outputs via continuous Bayesian updates and/or uncertainty quantification can improve diagnostic accuracy and clinician trust in healthcare professionals (physicians, residents, fellows, physician assistants (PAs), and nurse practitioners (NPs)) from US academic institutions evaluating patients with chest pain or dyspnea. The main questions it aims to answer are: * Does presenting AI predictions as Bayesian-updated post-test probabilities improve diagnostic accuracy compared to standard predicted probabilities? * Does the addition of uncertainty quantification (95% confidence intervals) to AI predictions improve diagnostic accuracy? * Do these interventions (Bayesian updating and/or uncertainty quantification) help clinicians recover from the negative effects of intentionally misleading AI predictions? Comparison: Researchers will compare standard AI predicted probabilities (presented without uncertainty) to Bayesian-updated post-test probabilities and/or outputs containing 95% confidence intervals to see if the interventions improve diagnostic accuracy, clinician confidence, and resilience against misleading AI. Participants will: * Review 8 clinical vignettes (simulated patient cases) focusing on chest pain or dyspnea. * Provide an initial "pre-test" diagnostic probability for 5 possible diagnoses based on the clinical history alone. * View AI model outputs that vary by experimental condition (standard probability vs. Bayesian update, with or without uncertainty intervals, and accurate vs. misleading). * Provide an updated "post-test" diagnostic probability for the diagnoses after viewing the AI output. * Select and rank diagnostic tests and therapeutic steps for each vignette. Complete a post-survey regarding their trust in the AI, comfort with the data presentation, and demographics.

Key Dates

First listed

Mar 9, 2026

Start date

Feb 15, 2026

Status verified

Jul 2026

Primary completion

Dec 31, 2026

Completion

Dec 31, 2026

Study Design

Enrollment

100 participants (estimated)

Allocation

RANDOMIZED

Intervention model

FACTORIAL

Primary purpose

OTHER

Arms

• Active Comparator: Standard Probability + No Uncertainty (Control)
  AI model prediction is presented as a standard predicted probability for each possible diagnosis (point estimate only), together with the top 3 clinical features driving the prediction. No confidence interval is shown. This is the control condition representing the most common current approach to presenting AI predictions in clinical settings. Within this arm, half of vignettes contain accurate AI predictions and half contain intentionally misleading predictions (balanced by design).
• Experimental: Bayesian Updating (CLR) + No Uncertainty
  AI model prediction is used to perform Bayesian updating of the clinician's pre-test probability into a post-test probability using continuous likelihood ratios (CLR). The post-test probability is presented as a point estimate only, without a confidence interval. The raw AI predicted probability is not shown to the participant. Within this arm, half of vignettes contain accurate AI predictions and half contain intentionally misleading predictions (balanced by design).
• Experimental: Standard Probability + Uncertainty (95% CI)
  AI model prediction is presented as a standard predicted probability for each possible diagnosis, together with the top 3 clinical features driving the prediction. The predicted probability is accompanied by a 95% confidence interval. Within this arm, half of vignettes contain accurate AI predictions and half contain intentionally misleading predictions (balanced by design). When AI predictions are misleading, confidence intervals are widened by a factor of 1.5x to simulate greater model uncertainty.
• Experimental: Bayesian Updating (CLR) + Uncertainty (95% CI)
  AI model prediction is used to perform Bayesian updating of the clinician's pre-test probability into a post-test probability using continuous likelihood ratios (CLR). The post-test probability is presented with a 95% confidence interval. The raw AI predicted probability is not shown to the participant. This arm represents the full intervention combining both candidate approaches. Within this arm, half of vignettes contain accurate AI predictions and half contain intentionally misleading predictions (balanced by design). When AI predictions are misleading, confidence intervals are widened by a factor of 1.5x to simulate greater model uncertainty.

Primary Outcome Measure

Clinician Diagnostic Accuracy [ Time Frame: Day 1 during survey completion ]

Locations (1)

Find similar trials in San Francisco, CA