Out-of-Distribution Generalization Challenge in Dialog State Tracking

Review Report

MetaReview

This is an interesting paper that attempts to link out-of-distribution generalization to dialog state tracking. I am not completely convinced that unseen semantics are necessarily “out-of-distribution”; they may be in-distribution but on the long tail. For example, we might imagine that our training data follows the simple distribution P(U_1, U_2) = P_1(U_1) P_2(U_2). Given a finite training set, we might still encounter unseen tuples at test time, especially for rare values of U_1 and U_2. This is a different situation from, say, P(U_1, U_2) = P_1(U_1) P(U_2 | U_1), in which both U_1 and U_2 are frequent at training time but never appear together, implying P(U_2 | U_1) ~= 0. So I’m not convinced that compositional generalization and OOD generalization are the same phenomenon, as the paper seems to imply. Nonetheless, the paper offers interesting empirical results and is clearly written, so I think it would make a good contribution to the workshop.

Overall Recommendation

Review #1: 7/10; Review #2: 7/10