Preprint Communication Version 1 Preserved in Portico This version is not peer-reviewed

How Trustworthy is ChatGPT? The Case of Bibliometric Analyses

Version 1 : Received: 24 March 2023 / Approved: 28 March 2023 / Online: 28 March 2023 (09:34:13 CEST)

A peer-reviewed article of this Preprint also exists.

Farhat, F.; Sohail, S.S.; Madsen, D.Ø. How Trustworthy Is ChatGPT? The Case of Bibliometric Analyses. Cogent Engineering 2023, 10, doi:10.1080/23311916.2023.2222988. Farhat, F.; Sohail, S.S.; Madsen, D.Ø. How Trustworthy Is ChatGPT? The Case of Bibliometric Analyses. Cogent Engineering 2023, 10, doi:10.1080/23311916.2023.2222988.

Abstract

The introduction of the AI-powered chatbot ChatGPT by OpenAI has sparked much interest and debate among academic researchers. Commentators from different scientific disciplines have raised many concerns and issues, especially related to the ethics of using these tools in scientific writing and publications. In addition, there has been discussions about whether ChatGPT is trustworthy, effective, and useful in increasing researchers’ productivity. Therefore, in this paper, we evaluate ChatGPT’s performance on tasks related to bibliometric analysis, by comparing the output provided by the chatbot with a recently conducted bibliometric study on the same topic. The findings show that there are large discrepancies and ChatGPT’s trustworthiness is low in this particular area. Therefore, researchers should exercise caution when using ChatGPT as a tool in bibliometric studies.

Keywords

ChatGPT; bibliometrics; trustworthiness; artificial intelligence; chatbots

Subject

Computer Science and Mathematics, Artificial Intelligence and Machine Learning

Comments (4)

Comment 1
Received: 31 March 2023
Commenter: Deep Kumar Kirtania
The commenter has declared there is no conflict of interests.
Comment: Very well done, congratulation to all of you for this excellent research paper.I have also done two similar paper about this topic. If you are interested, you can view these two papers. Thank youOpenAI ChatGPT Generated Content and Similarity Index: A study of selected terms from the Library & Information Science (LIS) disciplinehttps://www.qeios.com/read/FO1CP6 ChatGPT as a Tool for Bibliometrics Analysis: Interview with ChatGPThttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391794
+ Respond to this comment
Response 1 to Comment 1
Received: 4 April 2023
Commenter:
The commenter has declared there is no conflict of interests.
Comment: Thank you @Deep.
Comment 2
Received: 16 April 2023
Commenter:
The commenter has declared there is no conflict of interests.
Comment: As far as I know, GPT is not currently connected to internet resources (ChatGPT and GPT 3.X). Therefore, a task focused on bibliometric analysis using web resources such as WoS or Scopus is actually pointless, as GPT cannot accomplish this in the version used.
+ Respond to this comment
Response 1 to Comment 2
Received: 18 April 2023
Commenter:
The commenter has declared there is no conflict of interests.
Comment: Ofcourse. ChatGPT can access these internet resources like WOS and SCOPUS databases upto 2021 and we have tasked ChatGPT to fetch the data upto 2021 only. Indeed it can fetch very relevant data from these databases but when inquire for bibliometric analysis it glitched somehow or couldn't sum up the data for bibliometric analysis. Moreover, it never declines the query even it couldn't able to process right information and repetitively fetched the same incorrect data.

We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.

Leave a public comment
Send a private comment to the author(s)
* All users must log in before leaving a comment
Views 0
Downloads 0
Comments 4


×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.