{"pk":33230,"title":"Not Channels But Composite Signals: Speech Gesture, Diagrams and Object Demonstrations Are Integrated in Multimodal Explanations","subtitle":null,"abstract":"This paper provides empirical evidence that multimodal signals are produced and understood as integrated units of communication called composite signals, rather than being independently interpretable \"channels\" of communication. I propose that using composite signals relies on two communicative norms, co-expressivity and consistency: - co-expressivity: each element of a composite signal refers to the same underlying referent • consistency: elements of the same composite do not contradict each other. This paper will show that these norms are consistent with data comprising a set of explanations of how locks work in which participants spoke while gesturing, drawing diagrams, and manipulating a sample lock. Co-expressivity is supported by the fact that co-expressive speech segments can be found in nearby speech for communicative nonverbal behaviors but not for non-communicative nonverbal behaviors. Consistency is evidenced in inferences that maintain number and modality consistency in cases of apparent contradiction.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[],"section":"Long Papers","is_remote":true,"remote_url":"https://escholarship.org/uc/item/2vb8p9hg","frozenauthors":[{"first_name":"Randi","middle_name":"A.","last_name":"Engle","name_suffix":"","institution":"Stanford University School of Education","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"1998-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/33230/galley/24290/download/"}]}