{"pk":21348,"title":"Comparing Abstraction in Humans and Machines Using Multimodal Serial Reproduction","subtitle":null,"abstract":"Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"Artificial Intelligence; Psychology; Language and thought; Bayesian modeling; Large Language Models"}],"section":"Papers with Oral Presentation","is_remote":true,"remote_url":"https://escholarship.org/uc/item/0k26s3b7","frozenauthors":[{"first_name":"Sreejan","middle_name":"","last_name":"Kumar","name_suffix":"","institution":"Princeton University","department":""},{"first_name":"Raja","middle_name":"","last_name":"Marjieh","name_suffix":"","institution":"Princeton University","department":""},{"first_name":"Byron","middle_name":"","last_name":"Zhang","name_suffix":"","institution":"Princeton University","department":""},{"first_name":"Declan","middle_name":"","last_name":"Campbell","name_suffix":"","institution":"Princeton University","department":""},{"first_name":"MICHAEL","middle_name":"Y","last_name":"HU","name_suffix":"","institution":"New York University","department":""},{"first_name":"Umang","middle_name":"","last_name":"Bhatt","name_suffix":"","institution":"New York University","department":""},{"first_name":"Brenden","middle_name":"","last_name":"Lake","name_suffix":"","institution":"NYU","department":""},{"first_name":"Tom","middle_name":"","last_name":"Griffiths","name_suffix":"","institution":"Princeton University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2024-01-01T10:00:00-08:00","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/21348/galley/10947/download/"},{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/21348/galley/21793/download/"}]}