Clinical reasoning with machines: evaluating the interpretive depth of AI in urological case assessments
| dc.contributor.author | Taskiran, Arda Taskin | |
| dc.contributor.author | Balik, Ahmet Yildirim | |
| dc.contributor.author | Basaran, Ekrem | |
| dc.contributor.author | Baba, Dursun | |
| dc.contributor.author | Kayikci, Muhammet Ali | |
| dc.date.accessioned | 2026-07-01T11:39:38Z | |
| dc.date.available | 2026-07-01T11:39:38Z | |
| dc.date.issued | 2026 | |
| dc.department | Düzce Üniversitesi | |
| dc.description.abstract | Background Large language models (LLMs) are increasingly utilized as decision-support tools in medicine. However, their clinical reliability and applicability remain uncertain. This study compared ChatGPT-3.5, ChatGPT-4o, and Gemini 1.0 Pro in responding to standardized urological clinical scenarios evaluated by blinded experts. Methods This observational cross-sectional study included 75 urology specialists categorized by experience (< 10 years vs. >= 10 years). Participants independently and blindly rated anonymized AI-generated responses for 10 common urological cases using a 5-point Likert scale across four predefined domains: accuracy, reliability, clinical applicability, and interpretive depth. Normality was assessed with the Shapiro-Wilk test, and ANOVA or Kruskal-Wallis tests were used as appropriate, followed by post-hoc pairwise analyses. Inter-rater reliability was calculated using Cronbach's alpha and Fleiss' kappa. Spearman correlation coefficients were computed to examine associations among rating domains. Results ChatGPT-4o achieved the highest mean scores across all domains, followed by Gemini 1.0 Pro and ChatGPT-3.5. Performance differences were statistically significant for all parameters (p < 0.05), with the largest gaps observed in accuracy (4.4 +/- 0.48 vs. 4.0 +/- 0.52 vs. 3.7 +/- 0.56) and clinical applicability (4.2 +/- 0.49 vs. 3.8 +/- 0.51 vs. 3.5 +/- 0.55). A moderate positive correlation was observed between accuracy and reliability (r = 0.50), while the previously reported negative correlation between reliability and interpretive depth was corrected to r = - 0.18, indicating only a weak inverse relationship. Inter-rater agreement was high (Cronbach's alpha = 0.84; Fleiss' kappa = 0.72). Conclusion Newer-generation large language models, particularly ChatGPT-4o, showed higher performance scores in terms of accuracy and clinical applicability in standardized urological decision-support scenarios. However, these findings should be interpreted with caution and require confirmation through repeated-measures or mixed-model analyses as well as validation in real-world clinical settings. Ongoing benchmarking of evolving AI systems remains important to monitor longitudinal improvements while ensuring safety, reliability, and appropriate clinical use. | |
| dc.identifier.doi | 10.1186/s12894-026-02048-x | |
| dc.identifier.issn | 1471-2490 | |
| dc.identifier.issue | 1 | |
| dc.identifier.orcid | 0000-0001-8051-5802 | |
| dc.identifier.orcid | 0000-0001-8319-512X | |
| dc.identifier.pmid | 41507902 | |
| dc.identifier.scopus | 2-s2.0-105029455478 | |
| dc.identifier.scopusquality | Q2 | |
| dc.identifier.uri | https://doi.org/10.1186/s12894-026-02048-x | |
| dc.identifier.uri | https://hdl.handle.net/20.500.12684/23392 | |
| dc.identifier.volume | 26 | |
| dc.identifier.wos | WOS:001682979700001 | |
| dc.identifier.wosquality | Q3 | |
| dc.indekslendigikaynak | Web of Science | |
| dc.indekslendigikaynak | Scopus | |
| dc.indekslendigikaynak | PubMed | |
| dc.language.iso | en | |
| dc.publisher | Bmc | |
| dc.relation.ispartof | Bmc Urology | |
| dc.relation.publicationcategory | Makale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanı | |
| dc.rights | info:eu-repo/semantics/openAccess | |
| dc.snmz | KA_WOS_20260623 | |
| dc.subject | [Keyword Not Available] | |
| dc.title | Clinical reasoning with machines: evaluating the interpretive depth of AI in urological case assessments | |
| dc.type | Article |












