Fundamental Exploration of Evaluation Metrics for Persona Characteristics of Text UtterancesView Publication
To maintain utterance quality of a personaaware dialog system, inappropriate utterances for the persona should be thoroughly filtered. When evaluating the appropriateness of a large number of arbitrary utterances to be registered in the utterance database of a retrieval-based dialog system, evaluation metrics that require a reference (or a “correct” utterance) for each evaluation target cannot be used. In addition, practical utterance filtering requires the ability to select utterances based on the intensity of persona characteristics. Therefore, we are developing metrics that can be used to capture the intensity of persona characteristics and can be computed without references tailored to the evaluation targets. To this end, we explore existing metrics and propose two new metrics: persona speaker probability and persona term salience. Experimental results show that our proposed metrics show weak to moderate correlations between scores of persona characteristics based on human judgments and outperform other metrics overall in filtering inappropriate utterances for particular personas.
Related PublicationsView All
Dialogue Generation Conditional on Predefined Stories: Preliminary Results
Chiaki MiyazakiThis paper introduces dialogue generation conditional on predefined stories, which will be called Story2Dialog […]
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives
Silin Gao*, Beatriz Borges*, Soyoung Oh*, Deniz Bayazit*, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, Antoine Bosselut*Sustaining coherent and engaging narratives requires dialogue or storytelling agents to understand how the per […]