November 2, 2018
Jump to better conclusions: SCAN both left and right
Empirical Methods in Natural Language Processing (EMNLP)
Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for.
By: Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, Douwe Kiela