【作者】 李宾； 朱敏郡； 翁诣轩； 夏飞； 李树涛；

【Author】 Bin Li;Minjun Zhu;Yixuan Weng;Fei Xia;Shutao Li;College of Electrical and Information Engineering, Hunan University;National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy Sciences;

【机构】 湖南大学电气信息与工程学院； 中国科学院自动化所模式识别国家重点实验室；

【摘要】 文本生成任务作为自然语言处理中重要的组成部分,近年来受到学者们的广泛关注。特别地,在基于大纲的条件故事生成任务上,由于预训练模型的预训练-微调范式具有灾难性遗忘的缺点,现有的方法难以基于给定的大纲词汇生成完整、逻辑通顺的文本。为了解决这个问题,我们建立了结合迁移降噪学习的大纲条件故事生成方法框架。首先,我们提出了Rake迁移训练方法,通过构造故事大纲实现预训练任务到下游任务的迁移学习。其次,我们引入Child微调的方法来减轻由于训练数据不足而引起的参数过拟合,从而减轻了预训练模型的灾难性遗忘。此外,为了在语言理解和语言生成之间取得平衡,我们基于预训练模型引入In-trust损失函数进行降噪学习。大量的实验表明,我们的方法在基于大纲的条件故事生成任务的六个评价指标均超过了最先进的方法,这证明我们所提出方法的有效性。更多还原

【Abstract】 As an important part of natural language processing, text generation tasks have received extensive attention from scholars in recent years. Specifically, on the task of outline-based conditional story generation, due to the disadvantage of catastrophic forgetting in the pre-training-then-fine-tuning paradigm of pre-trained models, it is difficult for existing methods to generate com-plete and logical texts based on the given outline. To address this issue, we establish a framework for an outline-story generation method incorporating transfer denoising learning. First, we propose the Rake transfer training method, which realizes the transfer learning from pre-training tasks to downstream tasks by constructing a story outline. Second, we introduce the method of Child fine-tuning to alleviate parameter overfitting due to insufficient training data, thereby mitigating the catastrophic forgetting of pre-trained models. Furthermore, to strike a balance between language understanding and language generation, we introduce an In-trust loss function for denoising learning based on the pretrained model. Extensive experiments show that our method outperforms state-of-the-art methods on all six metrics of the outline-based story generation task, which demonstrates the effectiveness of our proposed method.更多还原