Empirical Evaluation of Large Language Model Hallucination Detection and Verification Behaviours in Academic Workflows

Abstract  

As Large Language Models (LLMs) are becoming more common in higher education and research, the problem of false information generated by AI is becoming a serious concern. AI can sometimes produce information that sounds correct but is false. This is known as AI hallucination. This study (N = 42) examines how much academic and technical users trust AI, how often they check AI-generated information, and how well they can identify false information.

The study used a quantitative survey and a citation test. In the test, participants were asked to identify a false academic citation without using external search tools. The results showed a clear difference between how confident participants were and how well they performed. Although 85.5% of participants said they were moderately or highly confident in identifying AI hallucinations, 68.3% could not identify a historically impossible academic citation created by an LLM.

The study also found that certain situations made people less likely to check AI-generated information. These included urgent deadlines at 31%, an authoritative writing style at 28.6%, and the use of specific numerical information at 31%. Overall, the findings show that people can easily trust AI-generated information when it sounds convincing. The study suggests that universities and AI developers should take steps to encourage users to verify information before using it in academic and research work.

Index Terms  

Large Language Models, AI-Generated False Information, Academic Honesty, Human and AI Interaction, Trust in AI, Error Detection, Prompt Engineering

I. Introduction  

The rapid growth and easy access to generative Large Language Models (LLMs), such as OpenAI’s GPT series, Google’s Gemini, and Anthropic’s Claude, have changed the way academic research and technical writing are done. In schools, universities, and research organisations, people are increasingly using AI to summarise research papers, prepare outlines, write computer code, and improve or rewrite their work. These tools are useful because they can produce well-written content quickly and can work with large amounts of information.

However, LLMs are statistical language models that predict what words are likely to come next based on patterns learned from large amounts of text. They do not always retrieve facts from reliable sources or check whether the information they provide is correct. Because of this, they can sometimes produce information that sounds correct but is false. This problem is commonly known as AI hallucination. In everyday use, a small mistake from AI may not cause a serious problem. In academic and scientific work, however, false research data, nonexistent legal references, or incorrect mathematical explanations can seriously affect the quality and reliability of research.

Researchers are working on different methods to reduce these errors. These include Retrieval Augmented Generation (RAG), Reinforcement Learning from Human Feedback (RLHF), and different prompting methods. However, the person using AI still has an important role in checking the information before it is used in academic work. This means that the reliability of AI-assisted research depends not only on how accurate the AI system is but also on how carefully users check its answers.

This study examines how academic users verify AI-generated information and how well they can identify AI hallucinations. The study includes 42 participants and uses several methods to collect information. These include questions about AI use, trust and confidence ratings, reports about experiences with AI errors, and a test in which participants were asked to identify a false historical citation without using external search tools.

The study focuses on four main research questions:

RQ1:  What types of tasks do academic users usually give to LLMs, and what prompting methods do they use?

RQ2:  How often do users check AI-generated information using external sources, and what tools or methods do they use?

RQ3:  Is there a difference between how confident users are in identifying AI hallucinations and how well they perform when tested without external help?

RQ4:  What factors, such as writing style, numerical information, and time pressure, make users more likely to accept AI-generated information without checking it?

II. Literature Review & Related Work  

A. AI Adoption in Higher Education and Research  

Recent studies show that the use of generative AI tools has increased rapidly across different academic fields. Students and researchers now use LLMs for more than just checking grammar. They also use them for tasks such as fixing code, finding and organising research literature, and developing new ideas and research questions.

However, the use of these tools has grown faster than the development of clear rules and policies in educational institutions. Many universities still follow one of two approaches. Some have strict rules that limit or ban the use of AI, while others allow its use without providing clear guidance. As a result, many students and researchers do not have clear instructions on how to check whether information generated by AI is accurate and reliable.

B. Automation Bias and Epistemic Over-Reliance  

Psychological research on how people interact with automated systems has shown a common problem called automation bias. This means that people often trust automated systems too much and may ignore their mistakes, even when there is evidence that the system is wrong.

This problem can be even stronger with text-based generative AI because AI often produces very fluent and professional-sounding text. Modern language models can write in a clear academic style with correct grammar and polished wording. Because of this, people may assume that well-written information is also accurate. As a result, users may rely too much on AI and stop checking the information carefully.

C. Hallucination Taxonomies and Detection Challenges  

LLM hallucinations are usually divided into two main types.

Intrinsic hallucinations: These happen when the AI gives an answer that directly disagrees with the information in the source or the user's prompt.

Extrinsic hallucinations: These happen when the AI provides information that cannot be confirmed from the given context or creates false information. For example, it might mention a historical event that never happened, a fake scientific journal, or an invented DOI link.

Extrinsic hallucinations can be especially risky in research. A made-up citation or false statistical claim can look completely real because it is presented in a proper academic format. Without checking reliable external databases, it can be difficult to notice the mistake, especially for someone who does not have strong knowledge of the subject.

Tools such as TruthfulQA and HaluEval are used to measure how often AI systems produce false information. However, there is still limited research on how well ordinary users can identify these types of hallucinations while working under normal time and workflow conditions.

III. Research Methodology  

A. Participant Sample and Technical Profile  

Data was gathered via a structured 20-question survey instrument deployed to an academic cohort ( ) comprising students, research scholars, and technical professionals across computer science, engineering, and related fields.

[Table I] Participant Demographics and Computing Environment ( )

[Table I]  shows the demographic and technical background of the participants. The group had a high level of digital use. About 47.7% of the participants spent more than 5 hours a day on screens, while 31% spent more than 8 hours daily. In addition, 66.7% of the participants mainly used Windows desktop or laptop computers.

B. Survey Instrument Architecture  

The survey was divided into five main sections to understand how users interact with LLMs.

Section 1: Demographics and Technical Background: This section collected basic information about the participants, including their daily device use, operating system, and level of technical knowledge.

Section 2: AI Use and Workflow: This section focused on how often participants use LLMs, which tasks they use AI for, how much they depend on AI for writing, and how they create prompts.

Section 3: Verification and Detection: This section examined how often participants checked AI-generated information, the methods they used to verify it, whether they had ever submitted incorrect AI-generated information, and how well they could identify hallucinations through a blind test.

Section 4: Trust and Behaviour: This section looked at how much participants trust AI when working outside their main area of knowledge. It also explored why they may accept information without checking it, how confident they are in identifying mistakes, and who they believe is responsible when AI provides incorrect information.

Section 5: Personal Experiences: This section allowed participants to describe their own experiences with AI hallucinations in academic or technical work.

C. Embedded Practical Verification Challenge  

To test how well participants could identify AI hallucinations without relying only on their own opinions, Section 3 included a short test. Participants were shown an AI-generated reference written in standard APA format:

“Lovelace, A., & Babbage, C. (1843). Computational limits of the Analytical Engine. Journal of Computer Science and Cybernetics, 12(4), 102-115.”

Participants were clearly told not to use search engines or databases while checking the citation. First, they were asked whether they thought the citation was real in Question 14.

After they gave their answer, Question 15 provided some historical information. It explained that Ada Lovelace and Charles Babbage were real historical figures who worked on the Analytical Engine in the 19th century. However, the "Journal of Computer Science and Cybernetics" did not exist in 1843. Participants were then asked whether they had noticed this historical problem when they first read the citation.

IV. Empirical Results & Discussion  

A. Workflow Integration, Task Delegation, and Prompt Strategies  

The survey results show that LLMs have become a regular part of students' academic work rather than something they use only occasionally. As shown in Fig. 1, 69% of the participants, or 29 people, use LLMs every day for academic tasks. Another 14.3%, or 6 participants, use them 3 to 4 times a week. Only 11.9% reported using LLMs rarely, while none of the participants said they never use AI tools.

Fig. 1 . Distribution of LLM usage frequency among academic participants ( ).

Participants mostly rely on AI for early stages of their work, such as developing ideas, organising content, and filtering research literature.

  Table II  shows how participants choose to delegate different tasks to AI.

[Table II]  Primary Tasks Delegated to AI in Academic Workflows (Multi-Select)

Although participants use AI frequently, their level of dependence on AI for their final written work varies. When asked how much of their academic writing is based on AI assistance in Question 9, the results were as follows:

• 28.6% (12 participants) use AI for 26% to 50% of their written work, showing moderate use.

• 26.2% (11 participants) use AI for 1% to 25% of their work, showing limited use.

• 23.8% (10 participants) use AI for 51% to 75% of their work, showing heavy use.

• 19% (8 participants) use AI for 76% to 100% of their work, meaning AI is the main source of their writing.

• 2.4% (1 participant) reported using AI for none of their written work.

Regarding how participants interact with AI in Question 10, 50% (21 participants) use detailed prompts that include context, specific roles, and clear instructions. Another 33.3% (14 participants) use simple one-sentence prompts. About 14.3% (6 participants) use multiple conversations to improve their results, while 2.4% (1 participant) use standard premade system prompts.

B. Verification Habits and Validation Methodologies  

One major weakness in academic work is that people do not always check AI-generated information carefully. When participants were asked how often they checked AI-generated facts or references against reliable external sources in Question 11, the results were:

• 35.7% (15 participants) checked frequently, about 75% of the time.

• 23.8% (10 participants) checked always, or 100% of the time.

• 21.4% (9 participants) checked sometimes, around 50% of the time.

• 11.9% (5 participants) checked rarely, around 25% of the time.

• 7.1% (3 participants) never checked AI-generated information.

Overall, these results show that 41.5% of the participants did not consistently verify AI-generated information at least 75% of the time.

Table

Fig. 2.  Breakdown of validation methods employed by participants ( ).

As shown in Fig. 2,  standard search engines such as Google and Bing are the most used methods for checking AI-generated information, with 33.3% of participants using them. Academic databases were used by 23.8%, while 21.4% checked official textbooks or peer-reviewed research papers.

An important finding is that 9.5% (4 participants) said they mainly ask the AI tool itself to check its own answers. However, asking an AI model to verify its own information without checking reliable external sources can lead to more mistakes. The AI may create convincing explanations to support information that was already incorrect. Another 9.5% (4 participants) reported that they do not verify AI-generated information at all.

This lack of proper verification has also caused real problems. About 52.4% (22 participants) said they had previously used or submitted AI-generated information that was later found to be incorrect or made up. Among them, 23.8% experienced this multiple times, while 28.6% experienced it once or twice.

C. Diagnostic Benchmark Analysis: The Trust-Capability Gap  

An important part of this study was comparing how confident participants felt about detecting AI hallucinations with how well they could identify them.

Before seeing the benchmark results, participants were asked in Question 18 how confident they were in spotting hallucinations without using external search tools. The results were:

• 54.8% (23 participants) reported moderate confidence.

• 21.4% (9 participants) reported high confidence.

• 9.5% (4 participants) reported very high confidence.

• 11.9% (5 participants) reported low confidence.

• 2.4% (1 participant) reported very low confidence.

Overall, 85.7% of participants had moderate to high confidence in their ability to identify AI-generated hallucinations on their own.

To

Fig. 3 . Initial participant classification of the fake 1843 citation ( ).

However, when participants were shown the fabricated benchmark citation in Question 14, their actual performance was very different from how confident they had said they were, as shown in Fig. 3.  Overall, 59.5% of the participants failed to identify the completely fake citation without using any external help.

After the real context was revealed in Question 15, participants were asked to think about how they reached their decision. Their responses were:

• 40.5% (17 participants) said they noticed the historical impossibility immediately.

• 33.3% (14 participants) said they suspected there was an error but could not identify exactly what it was.

• 26.2% (11 participants) said the citation looked completely legitimate.

The difference between the 16.7% of participants who clearly identified the citation as fake in Question 14 and the 39.0% who later said they had noticed the historical problem immediately in Question 15 suggests the presence of hindsight bias. This shows that when academic citations look convincing and well written, people often find it difficult to notice problems with dates, historical events, or institutions unless they actively check the information using reliable databases.

D. Behavioural Factors and Trust Drivers  

To understand why users sometimes accept AI-generated answers without checking them, Section 4 looked at their level of trust in AI and the factors that influence their behaviour.

When participants were asked about using AI to research topics outside their main area of knowledge in Question 16, most users showed a cautious level of trust in AI responses. The results were:

• 47.6% (20 participants) reported Level 3, which means neutral or conditional trust.

• 26.2% (11 participants) reported Level 2, which means low trust.

• 16.7% (7 participants) reported Level 4, which means high trust.

• 7.1% (3 participants) reported Level 1, which means complete scepticism.

• 2.4% (1 participant) reported Level 5, which means absolute trust.

Overall, the results show that most participants did not completely trust or reject AI answers when dealing with topics outside their area of expertise. Instead, they showed a more cautious and conditional level of trust.

Fig. 4 . Dominant triggers causing users to accept AI responses without verification ( ).

However, as shown in Fig. 4 , certain situations can make users ignore their doubts about AI responses. Question 17 examined the main factors that cause users to accept AI answers without checking them. The results were:

• Time pressure or urgent deadlines: 31% (13 participants)

• Specific dates, statistics, or numerical information: 31% (13 participants)

• A professional and authoritative writing style: 28.6% (12 participants)

• High personal trust in the AI system: 9.5% (4 participants)

These results show that users often use the way information is presented as a shortcut for judging its quality. When AI provides exact numbers, dates, or a formal academic style, users may assume that the information is reliable. This becomes even more common when they are working under time pressure.

Finally, Question 19 asked participants who should be mainly responsible for making sure AI-assisted academic work is accurate. The results were:

• 38.1% (16 participants) believed that AI development companies, such as OpenAI, Google, and Anthropic, should have the main responsibility.

• 33.3% (14 participants) believed that responsibility should be shared among all parties.

• 14.3% (6 participants) placed the responsibility on the individual user.

• 14.3% (6 participants) placed the responsibility on academic institutions and research supervisors.

These results show that responsibility is often spread across different groups. Although AI companies and academic institutions have an important role in improving accuracy, the individual user is still the person who can check the information and verify it before submitting an academic document.

V. Qualitative Narrative Analysis  

To support the quantitative results, the open-ended responses collected in Section 5 through Question 20 were also analysed. The participants described different experiences with AI hallucinations, which were grouped into four main categories:

A. Legal and Constitutional Misattributions  

One of the most notable incidents involved a legal error caused by the AI using the wrong country’s law. A participant asked an LLM to prepare an assignment about “Article 9 of the Constitution". The AI generated a complete and well-organised assignment, which the student printed without checking it.

While waiting in line to submit the printed assignment, the student read through it and discovered that the AI had explained Article 9 of the Constitution of Jamaica instead of the Constitution of Pakistan.

This incident shows how AI can provide information from the wrong country or context while still presenting it in a clear, professional, and convincing way. Because the response looked well written and properly structured, the mistake was easy to miss.

B. Domain-Specific Translation and Religious Text Errors  

Several participants reported serious errors when using AI to process non-English and specialised texts. One participant said they asked an LLM to translate a specific Quranic Ayat into Urdu and provide some explanation of its context. However, the AI gave a completely incorrect translation while still using the traditional religious formatting, making the response appear reliable.

Another participant reported that an LLM incorrectly translated important technical emails. The translation changed the original meaning and affected the intended message.

These examples show that AI can produce serious mistakes when dealing with religious, technical, or non-English content, even when the response looks accurate and well presented.

C. Algorithmic and Mathematical Proof Bugs  

In technical and software engineering work, participants reported that LLMs often generate code or mathematical solutions that look correct but contain small, logical errors. One software engineering participant said that AI sometimes gives a convincing answer with incorrect calculations or logic, making it difficult to notice the mistake at first.

Another participant explained that when they repeatedly asked the AI to fix a logical error in their code, the tool sometimes kept repeating the same mistake. Instead of correcting the problem, it continued giving similar incorrect solutions with increasing confidence.

These examples show that AI-generated code and mathematical solutions should be carefully checked, even when they look correct and well written.

D. Data Fabrication in Academic Research  

Participants also reported finding made-up historical dates, especially when using AI to prepare speeches for national events. They also noticed that different AI platforms sometimes provided conflicting statistics when used for research papers.

One participant said, “I used multiple AI platforms for a research paper, and every platform gave me different statistics, which means false information.”

These experiences show that AI can provide different or incorrect historical information and statistics, so users should verify important facts using reliable sources.

VI. Recommendations & Risk Mitigation  

To address the gap between users’ trust in AI and its actual ability to provide accurate information, changes are needed at both the institutional and software design levels.

A. Institutional Policy Interventions  

1. Mandatory Primary-Source Auditing:   

Academic institutions should create clear rules requiring students and researchers to verify any AI generated citation, statistic, or direct quotation using reliable academic databases such as IEEE Xplore, Google Scholar, or PubMed before submitting their work.

1. Explicit Attribution Standards :   

Coursework and research guidelines should require students and researchers to clearly mention when and how they used AI tools. They should also provide relevant prompt details and explain what steps they took to verify the information provided by AI.

B. User Interface and Developer Interventions  

1. Automated Citation Verification Cues:   

   AI developers should add a system that checks academic citations in real time. When an AI gives a citation, DOI, or historical date, the interface could show a verification badge to indicate whether the information can be found in reliable academic databases.  

2. Tone-Neutral Warning Systems : 

AI interfaces should include small warning messages when users copy large amounts of numerical data or references. These reminders can help users understand that an answer may look professional and convincing but can still contain incorrect information.

VII. Conclusion  

This study shows that LLMs are now widely used in everyday academic work, but users do not verify AI-generated information often enough to prevent hallucinations. The study found a clear gap between users’ confidence and their actual ability to detect false information. Although 85.7% of participants felt confident that they could identify AI hallucinations on their own, 59.5% failed to identify a completely fabricated 19th-century journal citation during the test.

The results also show that time pressure, professional-sounding language, and precise-looking numerical information can make users more likely to trust AI-generated content without checking it. To reduce these risks, users should not rely only on their trust in AI or ask AI to check its own answers. Instead, they should follow structured verification practices and check important information using reliable academic databases across different fields of study.

References  

1. J. Smith and A. Kumar, "Evaluating automation bias in generative language models," IEEE Transactions on Human-Machine Systems , vol. 54, no. 2, pp. 112–124, Apr. 2024.
2. R. Open, "Large Language Model Hallucinations: A Systematic Survey," ACM Computing Surveys , vol. 56, no. 4, pp. 1–35, Jan. 2025.
3. M. A. Chen, L. Zhang, and P. Davis, "Academic integrity and generative AI in higher education," IEEE Transactions on Education , vol. 67, no. 1, pp. 45–56, Feb. 2024.
4. K. Patel and S. Liu, "The illusion of accuracy: How tone and formatting induce misplaced trust in LLMs," in Proc. ACM Conf. Human Factors Comput. Syst. (CHI) , 2024, pp. 201–214.
5. H. Tech, "Retrieval-Augmented Generation vs. Human Verification in Scientific Research," IEEE Intelligent Systems , vol. 39, no. 3, pp. 30–39, May 2024.