Clinical reasoning with machines: evaluating the interpretive depth of AI in urological case assessments

dc.contributor.authorTaskiran, Arda Taskin
dc.contributor.authorBalik, Ahmet Yildirim
dc.contributor.authorBasaran, Ekrem
dc.contributor.authorBaba, Dursun
dc.contributor.authorKayikci, Muhammet Ali
dc.date.accessioned2026-07-01T11:39:38Z
dc.date.available2026-07-01T11:39:38Z
dc.date.issued2026
dc.departmentDüzce Üniversitesi
dc.description.abstractBackground Large language models (LLMs) are increasingly utilized as decision-support tools in medicine. However, their clinical reliability and applicability remain uncertain. This study compared ChatGPT-3.5, ChatGPT-4o, and Gemini 1.0 Pro in responding to standardized urological clinical scenarios evaluated by blinded experts. Methods This observational cross-sectional study included 75 urology specialists categorized by experience (< 10 years vs. >= 10 years). Participants independently and blindly rated anonymized AI-generated responses for 10 common urological cases using a 5-point Likert scale across four predefined domains: accuracy, reliability, clinical applicability, and interpretive depth. Normality was assessed with the Shapiro-Wilk test, and ANOVA or Kruskal-Wallis tests were used as appropriate, followed by post-hoc pairwise analyses. Inter-rater reliability was calculated using Cronbach's alpha and Fleiss' kappa. Spearman correlation coefficients were computed to examine associations among rating domains. Results ChatGPT-4o achieved the highest mean scores across all domains, followed by Gemini 1.0 Pro and ChatGPT-3.5. Performance differences were statistically significant for all parameters (p < 0.05), with the largest gaps observed in accuracy (4.4 +/- 0.48 vs. 4.0 +/- 0.52 vs. 3.7 +/- 0.56) and clinical applicability (4.2 +/- 0.49 vs. 3.8 +/- 0.51 vs. 3.5 +/- 0.55). A moderate positive correlation was observed between accuracy and reliability (r = 0.50), while the previously reported negative correlation between reliability and interpretive depth was corrected to r = - 0.18, indicating only a weak inverse relationship. Inter-rater agreement was high (Cronbach's alpha = 0.84; Fleiss' kappa = 0.72). Conclusion Newer-generation large language models, particularly ChatGPT-4o, showed higher performance scores in terms of accuracy and clinical applicability in standardized urological decision-support scenarios. However, these findings should be interpreted with caution and require confirmation through repeated-measures or mixed-model analyses as well as validation in real-world clinical settings. Ongoing benchmarking of evolving AI systems remains important to monitor longitudinal improvements while ensuring safety, reliability, and appropriate clinical use.
dc.identifier.doi10.1186/s12894-026-02048-x
dc.identifier.issn1471-2490
dc.identifier.issue1
dc.identifier.orcid0000-0001-8051-5802
dc.identifier.orcid0000-0001-8319-512X
dc.identifier.pmid41507902
dc.identifier.scopus2-s2.0-105029455478
dc.identifier.scopusqualityQ2
dc.identifier.urihttps://doi.org/10.1186/s12894-026-02048-x
dc.identifier.urihttps://hdl.handle.net/20.500.12684/23392
dc.identifier.volume26
dc.identifier.wosWOS:001682979700001
dc.identifier.wosqualityQ3
dc.indekslendigikaynakWeb of Science
dc.indekslendigikaynakScopus
dc.indekslendigikaynakPubMed
dc.language.isoen
dc.publisherBmc
dc.relation.ispartofBmc Urology
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanı
dc.rightsinfo:eu-repo/semantics/openAccess
dc.snmzKA_WOS_20260623
dc.subject[Keyword Not Available]
dc.titleClinical reasoning with machines: evaluating the interpretive depth of AI in urological case assessments
dc.typeArticle

Dosyalar