{"pk":27974,"title":"Are emoji a poor substitute for words? Sentence processing with emoji substitutions","subtitle":null,"abstract":"With the integration of emoji into digital keyboards, people areincreasingly using multimodal interactions between text andimage in real-time interactions. One technique of using emojiis to substitute them into sentences. We here investigate theonline processing of these interactions, by modulating eitherthe grammatical category of those substitutions (Experiment 1:nouns vs. verbs) or the type and location of substitutions(Experiment 2: emoji vs. logos, within sentences vs. at theirend). We found a processing cost for self-paced reading timesof images compared to words, which indeed extended past theemoji itself, but no difference in comprehensibility ratingsbetween word and congruent-image substitutions. Overall,these results suggest that, despite costs of switching modalities,text and images can be integrated into holistic multimodalexpressions.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"multimodality; sentence processing; emoji; visuallanguage"}],"section":"Publication-based-Talks","is_remote":true,"remote_url":"https://escholarship.org/uc/item/33d802h8","frozenauthors":[{"first_name":"Neil","middle_name":"","last_name":"Cohn","name_suffix":"","institution":"Tilburg university","department":""},{"first_name":"Tim","middle_name":"","last_name":"Roijackers","name_suffix":"","institution":"Tilburg university","department":""},{"first_name":"Robin","middle_name":"","last_name":"Schaap","name_suffix":"","institution":"Tilburg university","department":""},{"first_name":"Jan","middle_name":"","last_name":"Engelen","name_suffix":"","institution":"Tilburg university","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2018-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/27974/galley/17612/download/"}]}