Exploring the Behavior and Performance of Large Language Models: Can LLMs Infer Answers to Questions Involving Restricted Information?

Ángel Cadena-Bautista; Francisco López-Ponce; Sergio Ojeda-Trueba; Gerardo Sierra; Gemma Bel-Enguix

doi:10.20944/preprints202411.0502.v1

Preprint Article Version 1 This version is not peer-reviewed

Version 1 : Received: 6 November 2024 / Approved: 7 November 2024 / Online: 7 November 2024 (08:48:07 CET)

In this paper various LLMs are tested in a specific domain using a Retrieval-Augmented Generation (RAG) system. The study focuses on the performance and behaviour of the models and was conducted in Spanish. A questionnaire based on The Bible, which consist of questions that vary in complexity of reasoning, was created in order to evaluate the reasoning capabilities of each model. The RAG system matches a question with the most similar passage from The Bible and feeds the pair to each LLM. The evaluation aims to determine whether each model can reason solely with the provided information or if it disregards the instructions given and makes use on its pretrained knowledge.

RAG; Large Language Models; Information Retrieval; Bible corpus

Computer Science and Mathematics, Artificial Intelligence and Machine Learning

Not displayed online.

Please leave your comment to this article below.

Mathematical equations can be typed in either LaTeX formats \\[ ... \\] or $$ ... $$, or MathML format <math> ... </math>. Try the LaTeX or MathML example.

Type equation:

Preview:

Copy equation into message below

Close menu

Please click a symbol to insert it into the message box below:

Please enter the link here:

Optionally, you can enter text that should appear as linked text:

Copy link into message below

Close menu

Please enter or paste the URL to the image here (please only use links to jpg/jpeg, png and gif images):

Copy image into message below

Close menu

Type author name or keywords to filter the list of references in this group (you can add a new citation under Bibliography):

No existing citations in Discussion Group

Wikify editor is a simple editor for wiki-style mark-up. It was written by MDPI for Sciforum in 2014. The rendering of the mark-up is based on Wiky.php with some tweaks. Rendering of mathematical equations is done with MathJax. Please send us a message for support or for reporting bugs.

Close menu

Please declare any conflicts of interest you may have with this paper, financial or otherwise.

I do not have a conflict of interest

I would like to declare a conflict of interest

Display my name on the website

Captcha

Renew

I accept the following conditions:

Comments must follow the standards of professional discourse and should focus on the scientific content of the article. Insulting or offensive language, personal attacks and off-topic remarks will not be permitted. Comments must be written in English. Preprints reserves the right to remove comments without notice. Readers who post comments are obliged to declare any competing interests, financial or otherwise.

* All users must log in before leaving a comment

Notify me about updates to this article or when a peer-reviewed version is published.

Exploring the Behavior and Performance of Large Language Models: Can LLMs Infer Answers to Questions Involving Restricted Information?

Abstract

Keywords

Subject

Comments (0)