{"pk":29894,"title":"Adding biological constraints to deep neural networks reduces their capacity tolearn unstructured data","subtitle":null,"abstract":"Deep neural networks (DNNs) are becoming increasingly pop-ular as a model of the human visual system. However, theyshow behaviours that are uncharacteristic of humans, includingthe ability to learn arbitrary data, such as images with pixel val-ues drawn randomly from a Gaussian distribution. We investi-gated whether this behaviour is due to the learning and memorycapacity of DNNs being too high for the training task. We re-duced the capacity of DNNs by incorporating biologically mo-tivated constraints – an information bottleneck, internal noiseand sigmoid activations – in order to diminish the learning ofarbitrary data, without significantly degrading performance onnatural images. Internal noise reliably produced the desiredbehaviour, while a bottleneck had limited impact. Combiningall three constraints yielded an even greater reduction in learn-ing capacity. Furthermore, we tested whether these constraintscontribute to a network’s ability to generalize by helping it de-velop more robust internal representations. However, none ofthe methods could consistently improve generalization.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"deep learning; biological details; memorization;generalization; internal noise; bottleneck"}],"section":"Poster Session 2","is_remote":true,"remote_url":"https://escholarship.org/uc/item/5pr8s6b7","frozenauthors":[{"first_name":"Christian","middle_name":"","last_name":"Tsvetkov","name_suffix":"","institution":"University of Bristol","department":""},{"first_name":"Gaurav","middle_name":"","last_name":"Malhotra","name_suffix":"","institution":"University of Bristol","department":""},{"first_name":"Benjamin","middle_name":"D.","last_name":"Evans","name_suffix":"","institution":"University of Bristol","department":""},{"first_name":"Jeffrey","middle_name":"S.","last_name":"Bowers","name_suffix":"","institution":"University of Bristol","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2020-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/29894/galley/19748/download/"}]}