A Systematic Review of Large Language Models (LLMs) in Scientific Literature: Scientometric Analysis and Tree of Science Application
Keywords:
Large Language Models, ChatGPT, Tree of Science, Systematic Review, Scientometrics, Output AnalysisAbstract
The analysis of scientific literature has had a significant impact on multiple areas of knowledge. Its
importance lies in the characterization and extraction of information related to scientific production,
guaranteeing the homogeneity of references. Although the use of Large Language Models (LLMs)
and algorithmic tools such as ChatGPT has positive aspects, the application of ChatGPT in the
review of scientific literature presents challenges due to its lack of precision in the construction of
bibliometric analyses. To address this limitation, the Tree of Science (ToS) tool, which provides a
more reliable characterization of the articles, has been used. This scientometric review includes an
analysis of the research output in the main databases, Scopus and Web of Science on MLLs and
their different applications. International collaborations, the main journals of publication and the
dynamics of collaboration between authors have been considered as study variables, using the
algorithm employed by ToS to identify and explain the main contributions, articles, authors,
journals, keywords, and countries highlighted. The findings reveal the novelty of the topic, with
academic production concentrated in 2023 and 2024, and a notable influence of researchers from
the United States and China. Medical fields lead in this area, with prominent journals and prolific
authors in research.
Downloads
References
C. A. Martínez-Pinzón, «Mapeando el Panorama Cienciométrico de los Activos Intangibles: Tendencias, Mediciones y Dinámicas Colaborativas», Interfaces , vol. 6, no. 2, Dec. 2023, Accessed: May 22, 2024. [Online]. Available: https://revistas.unilibre.edu.co/index.php/interfaces/article/view/11190
I. A. Blank, «What are large language models suppodatosed to model?», Trends in Cognitive Sciences, vol. 27, n.o 11, pp. 987-989, nov. 2023, doi: 10.1016/j.tics.2023.08.006.
J. De Winter, «Can ChatGPT be used to predict citation counts, readership, and social
media interaction? An exploration among 2222 scientific abstracts», Scientometrics, vol. 129, n.o 4, pp. 2469-2487, abr. 2024, doi: 10.1007/s11192-024-04939-y.
F. E. Sandnes, «Can we identify prominent scholars using ChatGPT?», Scientometrics,
vol. 129, n.o 1, pp. 713-718, ene. 2024, doi: 10.1007/s11192-023-04882-4.
W. H. Walters y E. I. Wilder, «Fabrication and errors in the bibliographic citations generated by ChatGPT», Sci Rep, vol. 13, n.o 1, p. 14045, sep. 2023, doi: 10.1038/s41598-023-41032-5.
A. E. Rubio, G. Y. F. Yepes, and L. A. V. Marín, «Gobernanza para el desarrollo y la sostenibilidad de los destinos turísticos: una revisión de la literatura con ToS», Interfaces , vol. 5, no. 1, Sep. 2022, Accessed: May 22, 2024. [Online]. Available: https://revistas.unilibre.edu.co/index.php/interfaces/article/view/9459
P. Duque, M. Cárdenas, y S. Robledo, «Marketing social: evolución y tendencias», entramado, vol. 20, n.o 1, oct. 2023, doi: 10.18041/1900-3803/entramado.1.10378.
A. M. G. A., S. Robledo, y M. Zuluaga, «Topic Modeling: Perspectives From a Literature Review», IEEE Access, vol. 11, pp. 4066-4078, 2023, doi: 10.1109/ACCESS.2022.3232939.
A. Echeverri Rubio, G. Yaneth Flórez Yepes, y L. A. Vargas Marín, «Gobernanza para el desarrollo y la sostenibilidad de los destinos turísticos: una revisión de la literatura con ToS», Revista de Ingenierías Interfaces, vol. 5, n.o 1, pp. 1-22, 2022, Accedido: 23 de mayo de 2024. [Online]. Available: https://dialnet.unirioja.es/servlet/articulo?codigo=8661443
C.-A. Gonzalez-Correa, L.-O. Tapasco-Tapasco, y P.-A. Gomez-Buitrago, «A Method
for a Literature Search on Microbiota and Obesity for PhD Biomedical Research Using the Web of Science (WoS) and the Tree of Science (ToS)», ISTL, n.o 99, feb. 2022, doi: 10.29173/istl2679.
D. S. Valencia-Hernandez, S. Robledo, R. Pinilla, N. D. Duque-Méndez, y G. Olivar-Tost, «SAP Algorithm for Citation Analysis: An improvement to Tree of Science», Ing. Inv., vol. 40, n.o 1, pp. 45-49, ene. 2020, doi:10.15446/ing.investig.v40n1.77718.
S. Robledo, J. E. Vasquez, N. D. Duque3-Méndez, y V. Duque-Uribe, «Networking as
an entrepreneurial marketing tool: the link between effectuation and word of mouth», JRME, vol. 25, n.o 2, pp. 270-285, mar. 2023, doi: 10.1108/JRME-08-2020-0112.
C. M. Botero, C. B. Milanes, y S. Robledo, «50 years of the Coastal Zone Management Act: The bibliometric influence of the first coastal management law on the world», Marine Policy, vol. 150, p. 105548, abr. 2023, doi: 10.1016/j.marpol.2023.105548.
G. A. Ruiz, S. Robledo Giraldo, y H. Hernando Morales, «Static Code Analysis: A Tree of Science Review», Entre cienc. ing., vol. 17, n.o 34, pp. 9-14, 2023, doi: 10.31908/19098367.2846.
S. Robledo-Giraldo, J. G. Figueroa-Camargo, M. V. Zuluaga-Rojas, S. B. Vélez-Escobar, y P. L. Duque- Hurtado, «Mapping, evolution, and application trends in co-citation analysis: a scientometric approach», rev. investig. desarro. innov, vol. 13, n.o 1, pp. 201-214, feb. 2023, doi: 10.19053/20278306.v13.n1.2023.16070.
D. Díez, J. Díaz-Ospina, S. Robledo, y M. D. P. Rodríguez-Córdoba, «Tendencias teóricas y desafíos en la comunicación de la responsabilidad social corporativa», anagramas rumbos sentidos comun., vol. 20, n.o 40, pp. 146-176, abr. 2022, doi:10.22395/angr.v20n40a7.
M. Sallam, «ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns», Healthcare, vol. 11, n.o 6, p. 887, mar. 2023, doi: 10.3390/healthcare11060887.
R. Gupta et al., «Expanding Cosmetic Plastic Surgery Research With ChatGPT», Aesthetic Surgery Journal, vol. 43, n.o 8, pp. 930-937, jul. 2023, doi: 10.1093/asj/sjad069.
K. Cheng et al., «Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example», Ann Biomed Eng, vol. 51, n.o 6, pp. 1130-1135, jun. 2023, doi: 10.1007/s10439-023-03203-3.
V. Sorin et al., «Utilizing large language models in breast cancer management: systematic review», J Cancer Res Clin Oncol, vol. 150, n.o 3, p. 140, mar. 2024, doi: 10.1007/s00432-024-05678-6.
S. S. Sohail et al., «Decoding ChatGPT: A taxonomy of existing research, current challenges, and possible future directions», Journal of King Saud University – Computer and Information Sciences, vol. 35, n.o 8, p. 101675, sep. 2023, doi: 10.1016/j.jksuci.2023.101675.
O. Temsah et al., «Overview of Early ChatGPT’s Presence in Medical Literature: Insights From a Hybrid Literature Review by ChatGPT and Human Experts», Cureus, abr. 2023, doi: 10.7759/cureus.37281.
B. Sikander, J. J. Baker, C. D. Deveci, L. Lund, y J. Rosenberg, «ChatGPT-4 and Human Researchers Are Equal in Writing Scientific Introduction Sections: A Blinded, Randomized, Non-inferiority Controlled Study», Cureus, nov. 2023, doi: 10.7759/cureus.49019.
Z. N. Khlaif et al., «The Potential and Concerns of Using AI in Scientific Research:ChatGPT Performance Evaluation», JMIR Med Educ, vol. 9, p. e47049, sep. 2023, doi: 10.2196/47049.
H. Raja et al., «Automated Category and Trend Analysis of Scientific Articles on Ophthalmology Using Large Language Models: Development and Usability Study», JMIR Form Res, vol. 8, p. e52462, mar. 2024, doi: 10.2196/52462.
H. Bagde, A. Dhopte, M. K. Alam, y R. Basri, «A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research», Heliyon, vol.9, n.o 12, p. e23050, dic. 2023, doi: 10.1016/j.heliyon.2023.e23050.
N. Schopow, G. Osterhoff, y D. Baur, «Applications of the Natural Language Processing Tool ChatGPT in Clinical Practice: Comparative Study and Augmented Systematic Review», JMIR Med Inform, vol. 11, p. e48933, nov. 2023, doi: 10.2196/48933.
R. Gupta, K. Bagdady, y B. A. Mailey, «Response to: Truth or Lies? The Pitfalls and Limitations of ChatGPT in Systematic Review Creation», Aesthetic Surgery Journal, vol. 43, n.o 8, pp. NP656-NP657, jul. 2023, doi: 10.1093/asj/sjad108.
B. Hasan et al., «Integrating large language models in systematic reviews: a ramework and case study using ROBINS-I for risk of bias assessment», BMJ EBM, p. bmjebm-2023-112597, feb. 2024, doi: 10.1136/bmjebm-2023-112597.
S. Ruksakulpiwat et al., «Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis», JMIR Mhealth Uhealth, vol. 12, p. e51526, may 2024, doi: 10.2196/51526.
Y. Artsi, V. Sorin, E. Konen, B. S. Glicksberg, G. Nadkarni, y E. Klang, «Large language models for generating medical examinations: systematic review», BMC Med Educ, vol. 24, n.o 1, p. 354, mar. 2024, doi: 10.1186/s12909-024-05239-y.
T. Karakose, M. Demirkol, R. Yirci, H. Polat, T. Y. Ozdemir, y T. Tülübaş, «A Conversation with ChatGPT about Digital Leadership and Technology Integration: Comparative Analysis Based on Human–AI Collaboration», Administrative Sciences, vol. 13, n.o 7, p. 157, jun. 2023, doi: 10.3390/admsci13070157.
S. Biswas, «ChatGPT and the Future of Medical Writing», Radiology, vol. 307, n.o 2,
p. e223312, abr. 2023, doi: 10.1148/radiol.223312.
H. H. Thorp, «ChatGPT is fun, but not an author», Science, vol. 379, n.o 6630, pp.313-313, ene. 2023, doi: 10.1126/science.adg7879.
M. Salvagno, F. S. Taccone, y A. G. Gerli, «Can artificial intelligence help for scientific writing?», Crit Care, vol. 27, n.o 1, p. 75, feb. 2023, doi: 10.1186/s13054-023-04380-2.
I. Dergaa, K. Chamari, P. Zmijewski, y H. Ben Saad, «From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing», bs, vol. 40, n.o 2, pp. 615-622, 2023, doi: 10.5114/biolsport.2023.125623.
Alexis-Tahta, Joaquin. 2023. «Inteligencia Artificial en la Administración: una revisión de la literatura utilizando Tree of Science.» Interfaces 6 (1). https://revistas.unilibre.edu.co/index.php/interfaces/article/view/10731.
K. I. Roumeliotis y N. D. Tselikas, «ChatGPT and Open-AI Models: A Preliminary Review», Future Internet, vol. 15, n.o 6, p. 192, may 2023, doi: 10.3390/fi15060192.
M. Imran y N. Almusharraf, «Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature», CONT ED TECHNOLOGY, vol. 15, n.o 4, p. ep464, oct. 2023, doi: 10.30935/cedtech/13605.
V. D. Blondel, J.-L. Guillaume, R. Lambiotte, y E. Lefebvre, «Fast unfolding of communities in large networks», J. Stat. Mech., vol. 2008, n.o 10, p. P10008, oct. 2008, doi: 10.1088/1742-5468/2008/10/P10008.
S. Ruksakulpiwat, A. Kumar, y A. Ajibade, «Using ChatGPT in Medical Research: Current Status and Future Directions», Journal Of Multidisciplinary Healthcare, vol. Volume 16, pp. 1513-1520, may 2023, doi: 10.2147/jmdh.s413470.
A. S. Treviño-Juarez, «Assessing Risk of Bias Using ChatGPT-4 and Cochrane ROB2 Tool», Med.Sci.Educ., abr. 2024, doi: 10.1007/s40670-024-02034-8.
I. Maita, S. Saide, A. M. Putri, y D. Muwardi, «ProsCons of Artificial Intelligence-ChatGPT Adoption in Education Settings: A Literature Review and Future Research Agendas», IEEE Eng. Manag. Rev., pp. 1-12, 2024, doi:10.1109/EMR.2024.3394540.
L. De Angelis et al., «ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health», Front. Public Health, vol. 11, p. 1166120, abr. 2023, doi: 10.3389/fpubh.2023.1166120.
J. Rudolph, S. Tan, and S. Tan, «ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?», JALT, vol. 6, no. 1, pp. 342–363, Jan. 2023.
M. Montenegro-Rueda, J. Fernández-Cerero, J. M. Fernández-Batanero, y E. López-Meneses, «Impact of the Implementation of ChatGPT in Education: A Systematic Review», Computers, vol. 12, n.o 8, p. 153, jul. 2023, doi:10.3390/computers12080153.
H. Bečulić, E. Begagić, R. Skomorac, A. Mašović, E. Selimović, y M. Pojskić, «ChatGPT’s contributions to the evolution of neurosurgical practice and education: a systematic review of benefits, concerns and limitations», Medicinski Glasnik, n.o 1, 2024, doi: 10.17392/1661-23.
G. Eysenbach, «The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers», JMIR Med Educ, vol. 9, p. e46885, mar. 2023, doi: 10.2196/46885.
U. T. Sevgi, G. Erol, Y. Doğruel, O. F. Sönmez, R. S. Tubbs, y A. Güngor, «The role
of an open artificial intelligence platform in modern neurosurgical education: a preliminary study», Neurosurg Rev, vol. 46, n.o 1, p. 86, abr. 2023, doi: 10.1007/s10143-023-01998-2.
Y. Yanagita, D. Yokokawa, F. Fukuzawa, S. Uchida, T. Uehara, y M. Ikusaka, «Expert
assessment of ChatGPT’s ability to generate illness scripts: an evaluative study», BMC Med Educ, vol. 24, n.o 1, p. 536, may 2024, doi: 10.1186/s12909-024-05534-8.
Z. Han, F. Battaglia, A. Udaiyar, A. Fooks, and S. R. Terlecky, «An explorative assessment of ChatGPT as an aid in medical education: Use it with caution», Med. Teach., vol. 46, no. 5, pp. 657–664, May 2024.
