This is a Preprint and has not been peer reviewed. The published version of this Preprint is available: https://doi.org/10.1145/3615886.3627745. This is version 2 of this Preprint.
Downloads
Supplementary Files
Authors
Abstract
This paper examines the performance of ChatGPT, a large language model (LLM), in a geographic information systems (GIS) exam. As LLMs like ChatGPT become increasingly prevalent in various domains, including education, it is important to understand their capabilities and limitations in specialized subject areas such as GIS. Human learning of spatial concepts significantly differs from LLM training methodologies. Therefore, this study aims to assess ChatGPT's performance and spatial literacy by challenging it with a real GIS exam. By analyzing ChatGPT's responses and evaluating its understanding of GIS principles, we gain insights into the potential applications and challenges of LLMs in spatially-oriented fields. We conduct our evaluation with two models, GPT-3.5 and GPT-4, to understand whether general improvements of an LLM translate to improvements in answering questions related to the spatial domain. We find that both GPT variants can pass a balanced, introductory GIS exam, scoring 63.3\% (GPT-3.5) and 88.3\% (GPT-4), which correspond to grades D and B+ respectively in standard US letter grading scale. In addition, we also identify specific questions and topics where the LLMs struggle to grasp spatial concepts, highlighting the challenges in teaching such topics to these models. Finally, we assess ChatGPT's performance in specific aspects of GIS, including spatial analysis, basic concepts of mapping, and data management. This granular analysis provides further insights into the strengths and weaknesses of ChatGPT's GIS literacy. This research contributes to the ongoing dialogue on the integration of AI models in education and can provide guidance for educators, researchers, and practitioners seeking to leverage LLMs in GIS. By focusing on specific questions or concepts that pose difficulties for the LLM, this study addresses the nuances of teaching spatial concepts to AI models and offers potential avenues for improvement in spatial literacy within future iterations of LLMs.
DOI
https://doi.org/10.31223/X5P38P
Subjects
Earth Sciences, Education, Environmental Studies, Geographic Information Sciences, Geography, Higher Education, Other Geography, Science and Mathematics Education, Spatial Science
Keywords
GIS, education, ChatGPT, large language models, Generative AI, spatial literacy}, Spatial Literacy, Examination, Assessment
Dates
Published: 2023-06-20 15:52
Last Updated: 2023-10-16 22:21
Older Versions
License
CC BY Attribution 4.0 International
Additional Metadata
Conflict of interest statement:
None.
Data Availability (Reason not available):
Replication data and analysis scripts are available from: https://doi.org/10.17605/OSF.IO/RU6MF
There are no comments or no comments have been made public for this article.