Towards Understanding the Geospatial Skills of ChatGPT: Taking a Geographic Information Systems (GIS) Exam

This is a Preprint and has not been peer reviewed. The published version of this Preprint is available: https://doi.org/10.1145/3615886.3627745. This is version 2 of this Preprint.

Add a Comment

You must log in to post a comment.


Comments

There are no comments or no comments have been made public for this article.

Downloads

Download Preprint

Supplementary Files
Authors

Peter Mooney, Wencong Cui, Boyuan Guan, Levente Juhász 

Abstract

This paper examines the performance of ChatGPT, a large language model (LLM), in a geographic information systems (GIS) exam. As LLMs like ChatGPT become increasingly prevalent in various domains, including education, it is important to understand their capabilities and limitations in specialized subject areas such as GIS. Human learning of spatial concepts significantly differs from LLM training methodologies. Therefore, this study aims to assess ChatGPT's performance and spatial literacy by challenging it with a real GIS exam. By analyzing ChatGPT's responses and evaluating its understanding of GIS principles, we gain insights into the potential applications and challenges of LLMs in spatially-oriented fields. We conduct our evaluation with two models, GPT-3.5 and GPT-4, to understand whether general improvements of an LLM translate to improvements in answering questions related to the spatial domain. We find that both GPT variants can pass a balanced, introductory GIS exam, scoring 63.3\% (GPT-3.5) and 88.3\% (GPT-4), which correspond to grades D and B+ respectively in standard US letter grading scale. In addition, we also identify specific questions and topics where the LLMs struggle to grasp spatial concepts, highlighting the challenges in teaching such topics to these models. Finally, we assess ChatGPT's performance in specific aspects of GIS, including spatial analysis, basic concepts of mapping, and data management. This granular analysis provides further insights into the strengths and weaknesses of ChatGPT's GIS literacy. This research contributes to the ongoing dialogue on the integration of AI models in education and can provide guidance for educators, researchers, and practitioners seeking to leverage LLMs in GIS. By focusing on specific questions or concepts that pose difficulties for the LLM, this study addresses the nuances of teaching spatial concepts to AI models and offers potential avenues for improvement in spatial literacy within future iterations of LLMs.

DOI

https://doi.org/10.31223/X5P38P

Subjects

Earth Sciences, Education, Environmental Studies, Geographic Information Sciences, Geography, Higher Education, Other Geography, Science and Mathematics Education, Spatial Science

Keywords

GIS, education, ChatGPT, large language models, Generative AI, spatial literacy}, Spatial Literacy, Examination, Assessment

Dates

Published: 2023-06-20 15:52

Last Updated: 2023-10-16 22:21

Older Versions
License

CC BY Attribution 4.0 International

Additional Metadata

Conflict of interest statement:
None.

Data Availability (Reason not available):
Replication data and analysis scripts are available from: https://doi.org/10.17605/OSF.IO/RU6MF