{"pk":29716,"title":"Can neural networks acquire a structural bias from raw linguistic data?","subtitle":null,"abstract":"We evaluate whether BERT, a widely used neural network forsentence processing, acquires an inductive bias towards form-ing structural generalizations through pretraining on raw data.We conduct four experiments testing its preference for struc-tural vs. linear generalizations in different structure-dependentphenomena. We find that BERT makes a structural general-ization in 3 out of 4 empirical domains—subject-auxiliary in-version, reflexive binding, and verb tense detection in embed-ded clauses—but makes a linear generalization when tested onNPI licensing. We argue that these results are the strongest ev-idence so far from artificial learners supporting the propositionthat a structural bias can be acquired from raw data. If this con-clusion is correct, it is tentative evidence that some linguisticuniversals can be acquired by learners without innate biases.However, the precise implications for human language acqui-sition are unclear, as humans learn language from significantlyless data than BERT.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"inductive bias; structure dependence; BERT;learnability of grammar; poverty of the stimulus; neural net-work; self-supervised learning"}],"section":"Poster Session 1","is_remote":true,"remote_url":"https://escholarship.org/uc/item/3rx3h34n","frozenauthors":[{"first_name":"Alex","middle_name":"","last_name":"Warstadt","name_suffix":"","institution":"New York University","department":""},{"first_name":"Samuel","middle_name":"R.","last_name":"Bowman","name_suffix":"","institution":"New York University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2020-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/29716/galley/19573/download/"}]}