Evaluating the Impact of Temperature and Instruction Strategies on Hallucination in Large Language Models
| dc.contributor.author | Kabakuş, Abdullah Talha | |
| dc.date.accessioned | 2026-07-01T11:30:17Z | |
| dc.date.available | 2026-07-01T11:30:17Z | |
| dc.date.issued | 2026 | |
| dc.department | Düzce Üniversitesi | |
| dc.description.abstract | Large Language Models (LLMs) have demonstrated impressive generative and reasoning abilities, yet their tendency to produce factually incorrect or fabricated information—so-called hallucinations—remains a key limitation. This study systematically examines how temperature and system instruction strategies affect hallucination behavior in open-source LLMs executed through the Ollama framework. Three representative models—Gemma 2B, Mistral 7B Instruct, and Phi-3 Mini—were evaluated on the TruthfulQA benchmark using zero-shot, few-shot, and “say-I-don’t-know” prompting paradigms. Performance was measured through exact match, token-level F1, semantic similarity, and embedding-based similarity metrics. Two-way ANOVA and3 Tukey post-hoc analyses revealed that system instruction significantly influenced factual accuracy across all models, while temperature effects were comparatively minor. Few-shot prompting achieved the highest mean F1 score (0.1889), indicating that example conditioning effectively constrained hallucinations. Conversely, “say-I-don’t-know” prompts increased semantic alignment but reduced precision, suggesting a conservative refusal bias. Embedding-based similarity analyses confirmed higher semantic consistency for zero-shot responses. The results highlight that prompt design exerts a stronger and more interpretable influence on hallucination than sampling stochasticity, offering practical guidance for improving the factual reliability of open-source LLMs. | |
| dc.identifier.dergipark | 1819131 | |
| dc.identifier.doi | 10.54287/gujsa.1819131 | |
| dc.identifier.endpage | 32 | |
| dc.identifier.issn | 2147-9542 | |
| dc.identifier.issue | 1 | |
| dc.identifier.orcid | 0000-0003-2181-4292 | |
| dc.identifier.startpage | 13 | |
| dc.identifier.uri | https://doi.org/10.54287/gujsa.1819131 | |
| dc.identifier.uri | https://hdl.handle.net/20.500.12684/22504 | |
| dc.identifier.volume | 13 | |
| dc.institutionauthor | Kabakuş, Abdullah Talha | |
| dc.language.iso | en | |
| dc.publisher | Gazi Üniversitesi | |
| dc.relation.ispartof | Gazi University Journal of Science Part A: Engineering and Innovation | |
| dc.relation.publicationcategory | Makale - Ulusal Hakemli Dergi - Kurum Öğretim Elemanı | |
| dc.rights | info:eu-repo/semantics/openAccess | |
| dc.snmz | KA_DergiPark_20260623 | |
| dc.subject | Large Language Model | |
| dc.subject | Hallucination | |
| dc.subject | Prompt Engineering | |
| dc.subject | Natural Language Processing | |
| dc.subject | Artificial Intelligence | |
| dc.title | Evaluating the Impact of Temperature and Instruction Strategies on Hallucination in Large Language Models | |
| dc.type | Article |












