{"pk":29365,"title":"Systematicity in a Recurrent Neural Network by Factorizing Syntax andSemantics","subtitle":null,"abstract":"Standard methods in deep learning fail to capture composi-tional or systematic structure in their training data, as shownby their inability to generalize outside of the training distribu-tion. However, human learners readily generalize in this way,e.g. by applying known grammatical rules to novel words. Theinductive biases that might underlie this powerful cognitive ca-pacity remain unclear. Inspired by work in cognitive sciencesuggesting a functional distinction between systems for syn-tactic and semantic processing, we implement a modificationto an existing deep learning architecture, imposing an analo-gous separation. The resulting architecture substantially out-performs standard recurrent networks on the SCAN dataset, acompositional generalization task, without any additional su-pervision. Our work suggests that separating syntactic fromsemantic learning may be a useful heuristic for capturing com-positional structure, and highlights the potential of using cog-nitive principles to inform inductive biases in deep learning.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"compositional generalization; systematicity; deeplearning; inductive bias; SCAN dataset"}],"section":"Semantics","is_remote":true,"remote_url":"https://escholarship.org/uc/item/3c0462ph","frozenauthors":[{"first_name":"Jacob","middle_name":"","last_name":"Russin","name_suffix":"","institution":"UC Davis","department":""},{"first_name":"Randall","middle_name":"C.","last_name":"O’Reilly","name_suffix":"","institution":"UC Davis","department":""},{"first_name":"Jason","middle_name":"","last_name":"Jo","name_suffix":"","institution":"Universite de Montreal","department":""},{"first_name":"Yoshua","middle_name":"","last_name":"Bengio","name_suffix":"","institution":"Universite de Montreal","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2020-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/29365/galley/19226/download/"}]}