Evaluating the Impact of Temperature and Instruction Strategies on Hallucination in Large Language Models

dc.contributor.authorKabakuş, Abdullah Talha
dc.date.accessioned2026-07-01T11:30:17Z
dc.date.available2026-07-01T11:30:17Z
dc.date.issued2026
dc.departmentDüzce Üniversitesi
dc.description.abstractLarge Language Models (LLMs) have demonstrated impressive generative and reasoning abilities, yet their tendency to produce factually incorrect or fabricated information—so-called hallucinations—remains a key limitation. This study systematically examines how temperature and system instruction strategies affect hallucination behavior in open-source LLMs executed through the Ollama framework. Three representative models—Gemma 2B, Mistral 7B Instruct, and Phi-3 Mini—were evaluated on the TruthfulQA benchmark using zero-shot, few-shot, and “say-I-don’t-know” prompting paradigms. Performance was measured through exact match, token-level F1, semantic similarity, and embedding-based similarity metrics. Two-way ANOVA and3 Tukey post-hoc analyses revealed that system instruction significantly influenced factual accuracy across all models, while temperature effects were comparatively minor. Few-shot prompting achieved the highest mean F1 score (0.1889), indicating that example conditioning effectively constrained hallucinations. Conversely, “say-I-don’t-know” prompts increased semantic alignment but reduced precision, suggesting a conservative refusal bias. Embedding-based similarity analyses confirmed higher semantic consistency for zero-shot responses. The results highlight that prompt design exerts a stronger and more interpretable influence on hallucination than sampling stochasticity, offering practical guidance for improving the factual reliability of open-source LLMs.
dc.identifier.dergipark1819131
dc.identifier.doi10.54287/gujsa.1819131
dc.identifier.endpage32
dc.identifier.issn2147-9542
dc.identifier.issue1
dc.identifier.orcid0000-0003-2181-4292
dc.identifier.startpage13
dc.identifier.urihttps://doi.org/10.54287/gujsa.1819131
dc.identifier.urihttps://hdl.handle.net/20.500.12684/22504
dc.identifier.volume13
dc.institutionauthorKabakuş, Abdullah Talha
dc.language.isoen
dc.publisherGazi Üniversitesi
dc.relation.ispartofGazi University Journal of Science Part A: Engineering and Innovation
dc.relation.publicationcategoryMakale - Ulusal Hakemli Dergi - Kurum Öğretim Elemanı
dc.rightsinfo:eu-repo/semantics/openAccess
dc.snmzKA_DergiPark_20260623
dc.subjectLarge Language Model
dc.subjectHallucination
dc.subjectPrompt Engineering
dc.subjectNatural Language Processing
dc.subjectArtificial Intelligence
dc.titleEvaluating the Impact of Temperature and Instruction Strategies on Hallucination in Large Language Models
dc.typeArticle

Dosyalar