【作者】 刘飞凡； 李斯源； 刘鹏；

【Author】 Feifan Liu;Siyuan Li;Peng Liu;Harbin Institute of Technology;

【机构】 哈尔滨工业大学；

【摘要】 多模态大模型在机器翻译、摘要生成、人机交互、情感分析等NLP领域广泛应用。其丰富的常识和强大的逻辑推理能力,使得它们有潜力成为具身智能任务的自动化代理,完成任务分解、任务规划等工作。然而,大模型幻觉显著影响了其任务规划的可靠性;多模态大模型因其缺乏精确定位物体以及准确理解物体间位置关系的问题,无法将其所具有的物理规则常识运用于任务规划之中。为解决这些问题,本文提出了基于多模态大模型的具身智能体任务分解与安全规划方法,综合考虑了大语言模型和多模态大模型的性质,使用PDDL语法描述具身任务中的状态以消除大模型幻觉,结合思维链技术,并基于机器人执行具身任务的轨迹图像和碰撞信息,训练出用于预测具身任务子任务执行风险的神经网络,作为安全模块,辅助大模型进行任务规划。在Habitat2.0模拟环境中进行了实验,在物体重排任务设定下,达到了93.3%的规划成功率,并使物体与场景的平均碰撞数下降了41.2%,显著提高了规划的安全性。更多还原

【Abstract】 Large Multimodal Models are extensively used in various NLP domains such as machine translation,text summarization,human-computer interaction,and sentiment analysis.Their vast knowledge base and robust logical reasoning capabilities endow them with the potential to serve as automated agents for embodied intelligence tasks,handling task decomposition and planning.However,the phenomenon of "model hallucination" considerably undermines the reliability of their task planning.Additionally,multimodal large models struggle with precise object localization and accurate understanding of spatial relationships,limiting their ability to apply physical rule knowledge in task planning.To address these challenges,this paper proposes a method for task decomposition and safety planning for embodied agents based on multimodal large models.This approach integrates the properties of both large language and multimodal models,using PDDL syntax to describe states in embodied tasks and mitigate model hallucination.We incorporate chain-of-thought techniques and train a neural network based on trajectory images and collision data from robots performing embodied tasks.This neural network predicts the risks associated with sub-task execution,serving as a safety module to assist large models in task planning.Experiments conducted in the Habitat2.0 simulation environment for object rearrangement tasks demonstrated a planning success rate of 93.3% and a 41.2% reduction in average collisions between objects and the environment,significantly enhancing the safety of the planning process.更多还原