Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog HistoryView Publication
Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context. Each utterance in spoken conversation can be represented by many semantic and speaker attributes, and there has been an interest in building Spoken Language Understanding (SLU) systems for automatically predicting these attributes. Recent work has shown that incorporating dialogue history can help advance SLU performance. However, separate models are used for each SLU task, leading to an increase in inference time and computation cost. Motivated by this, we aim to ask: can we jointly model all the SLU tasks while incorporating context to facilitate low-latency and lightweight inference? To answer this, we propose a novel model architecture that learns dialog context to jointly predict the intent, dialog act, speaker role, and emotion for the spoken utterance. Note that our joint prediction is based on an autoregressive model and we need to decide the prediction order of dialog attributes, which is not trivial. To mitigate the issue, we also propose an order agnostic training method. Our experiments show that our joint model achieves similar results to task-specific classifiers and can effectively integrate dialog context to further improve the SLU performance.
Related PublicationsView All
The Pipeline System of ASR and NLU with MLM-based Data Augmentation toward STOP Low-resource Challenge
Hayato Futami, Jessica Huynh*, Siddhant Arora*, Shih-Lun Wu*, Yosuke Kashiwagi, Yifan Peng*, Brian Yan*, Emiru Tsunoo, Shinji Watanabe*This paper describes our system for the low-resource domain adaptation track (Track 3) in Spoken Language Unde […]
E-Branchformer-Based E2E SLU Toward Stop on-Device Challenge
Yosuke Kashiwagi, Siddhant Arora*, Hayato Futami, Jessica Huynh*, Shih-Lun Wu*, Yifan Peng*, Brian Yan*, Emiru Tsunoo, Shinji Watanabe*In this paper, we report our team’s study on track 2 of the Spoken Language Understanding Grand Challenge, whi […]