【作者】 黄蕾；

【导师】 骆斌；

【作者基本信息】 南京大学 ， 应用软件工程， 2018， 博士

【摘要】 随着互联网技术和移动设备的飞速发展,视频的采集、存储和传播变得越来越普及和便捷.由于集成了视觉、听觉、文本等模态,视频具有丰富生动的表现能力,逐步成为了社会媒体中分享信息的重要媒体形态.微视频是一种时长受限的新型视频形态,其时长通常被限定为几秒至十几秒.这大大降低了视频制作、传播和观看的要求,因而微视频在社交媒体网站上呈现出爆炸式增长.构建面向微视频的有效管理和智能化服务的需求也应运而生.作为视频有效管理和消费的基础,视频内容分析技术近年来取得了长足进展.但与传统长视频相比,微视频具有来源广泛、多样性强、信息密度高、标签稀疏等特点.这对面向微视频的内容分析技术提出了新的挑战.本文从微视频的自身特点入手,围绕面向微视频的内容分析关键技术展开研究,主要关注面向微视频的似物性采样、显著物体检测、标签精化和个性化推荐,从而为有效管理和消费微视频提供有力支撑.本文的创新性工作主要包括:(1)提出了一种基于多特征融合的微视频似物性采样方法.相比于面向图像的似物性采样,视频似物性采样难以直接获取候选区域,需要分析不同模态的似物性特征,并且要求提供具有准确位置和持续时间的边界框轨迹作为结果,因而具有更高的难度.根据视频中物体的特性,本文提出了一种融合微视频中多种特征的似物性采样方法,可以有效地检测出具有不同特性的物体.此外,本文充分利用了视频内容的时空一致性和冗余性,对视频似物性采样中关键帧似物性采样和短时长边界框轨迹的跟踪和匹配等关键环节进行改进,有效地控制了计算开销.该方法能够有效地处理包含复杂场景的视频,在综合考虑效果和效率的情况下,取得比已有代表性视频似物性采样方法更好的效果.(2)提出了一种基于时空差异度和一致性的微视频显著物体检测方法.相比于面向图像的显著物体检测,视频显著物体检测除了需要考虑视频帧内的静态显著性特征,还需要考虑帧间运动显著性特征、显著度一致性以及处理效率等问题,因而面临更多的挑战.根据视频中显著物体具有的特性,本文提出了一种融合微视频内容的空间差异度和时间差异度的显著物体检测方法,可以同时有效地检测出静止和运动的显著物体.同时,本文充分利用了相邻视频帧在内容上的冗余性,仅在关键帧上直接进行显著物体检测,并根据时空一致性传播显著性来生成其它各帧的显著性图,从而提高了方法效率.该方法能够有效处理包含复杂物体运动和场景的视频,在综合考虑效果和效率的情况下优于已有代表性视频显著物体检测方法.(3)提出了一种基于多源数据辅助的微视频标签精化方法.微视频的哈希标签存在严重的分布不均、质量低下等问题,很难直接采用哈希标签来描述微视频内容;同时,由于难以构建大规模的、人工标注的微视频数据集,直接训练有效的微视频自动标注模型也十分困难.考虑到许多已有人工标注的数据集在内容上与微视频具有较强的相关性,本文采用多个用于物体检测、行为检测和场景检测的公开数据集,提出了基于多源数据辅助的微视频标签精化方法,从而克服了微视频标签精化中哈希标签质量过低、训练数据严重不足等困难.该方法对拥有不同质量哈希标签甚至缺乏哈希标签的微视频均能取得较好的标签精化结果.同时,本文构建了一个面向微视频标签精化的测试数据集,可以用作微视频标签精化任务的评测基准.(4)提出了一种基于层次式用户兴趣建模的微视频个性化推荐方法.由于微视频的时长很短,用户在观看微视频时不可能逐个人工选择,而需要依赖于社会媒体网站的自动推荐.微视频推荐除了要考虑流行度等微视频的自身属性外,还需要根据用户观看历史和点赞、评论等反馈来对用户兴趣建模.考虑到用户的个性化兴趣及其随时间的变化,本文提出了一种面向微视频的层次式用户兴趣模型,可以同时有效地描述用户职业爱好引起的长期兴趣和观看微视频过程中的临时兴趣.以此为基础,本文提出了一种微视频个性化推荐方法,其推荐结果会满足不同用户需要并随时间推移而变化.该方法能够有效地描述用户观看微视频时的兴趣,从而取得比已有代表性方法更好的微视频推荐效果.更多还原

【Abstract】 With the rapid development of Internet technology and mobile devices,video acquisition,storage and propagation have been increasingly prevalent and convenient.Because of its integration of visual,audio and textural modalities,video has prominent ability in information expression.Hence,video becomes an important media form in sharing information on social network.As a new type of video,micro-video has very limited duration,which is usually several seconds.It greatly reduces the requirements in video production,propagation and viewing.Thus,the number of micro-video grows explosively on social network.The effective management and intelligent service for micro-videos emerge as the times require.Served as the foundation of effective video management and consumption,video content analysis techniques have made significant progresses in recent years.However,compared to conventional video,micro-video has some characteristics,such as various sources,high diversity,high information density,and sparse hashtag.These characteristics bring in new challenges to micro-video content analysis.In this dissertation,we investigate the inherent characteristics of micro-video and dedicate to studying several key techniques of micro-video content analysis,i.e.,object proposal,salient object detection,tag refinement and personalized recommendation.These techniques can provide strong support to effective management and consumption of micro-video on social network.The main contributions of the dissertation include:(1)We propose a novel object proposal method on micro-videos based on multi-feature fusion.Compared to object proposal on images,video object proposal is more difficult,because it cannot simply generate the candidates for objectness estimation,needs to analyze the objectness under different features,and requires to provide the bounding box trajectories with accurate locations and durations as the results.According to the characteristics of the objects in videos,we propose a video object proposal method based on multi-feature fusion,which can effectively propose the objects with different characteristics.Moreover,we improve the key procedures in video object proposal,such as keyframe object proposal and the generation and matching of short bounding box trajectories,by exploring the potential of spatio-temporal coherence and redundancy in micro-video content,in order to limit the computational cost.The experimental results show that the proposed method can effectively handle the micro-videos with complex content,and it obtains the better performance than the state-of-the-art video object proposal methods by considering both effectiveness and efficiency.(2)We propose a novel salient object detection method on micro-videos using spatial-temporal difference and coherence.Compared to salient object detection on images,video salient object detection should handle more challenges,such as detecting motion saliency cue,improving saliency coherence inter frames,and pursuing high efficiency,besides extracting static saliency cues intra each video frame.Based on the characteristics of salient objects in videos,we propose a salient object detection method by fusing spatial difference and temporal difference of video content,which can effectively detect both static and moving salient objects.Meanwhile,by making full use of the content redundancy of adjacent video frames,we only directly detect salient objects on keyframes and generate the saliency maps of non-keyframes by spatial-temporal coherence based saliency propagation,which can obviously improve the efficiency of the proposed method.The experimental results show that the proposed method can effectively handle the micro-videos contacting complex object motion and scenes,and it outperforms the state-of-the-art methods when taking account of both effectiveness and efficiency.(3)We propose a novel tag refinement method for micro-videos by learning from multiple data sources.Because the user-generated hashtags of micro-videos have seriously unbalanced distribution and low quality,it is challenging to directly describe micro-video content via hashtags.Meanwhile,considering the huge workload of constructing a large-scale,manually labelled micro-video dataset,it is also difficult to train an effective model for effective micro-video labeling.Considering the existence of many manually labeled datasets,which have relevant content to micro-videos,we propose a tag refinement method for micro-videos by learning from multiple public data sources with manually labelled tags,which are provided for object detection,activity detection and scene detection.The proposed method can overcome the difficulty of directly refining the imprecise hashtags and address the problem of lacking manually labelled micro-video datasets for training.It can obtain good performance in tag refinement of micro-videos with high quality or low quality hashtags even without hashtag.Moreover,we construct a micro-video dataset and manually label the microvideos with target tags.It can be used as a benchmark for micro-video tag refinement.(4)We propose a novel micro-video recommendation method based on hierarchical user interest modeling.Because micro-video has very limited duration,users prefer to obtain interesting micro-videos by automatic recommendation instead of selecting them manually.Effective micro-video recommendation technique should model user interest based on his/her viewing history and feedback,such as like and comment.Considering personal user interest and its changeability,we propose a hierarchical model to describe user interests in viewing micro-videos,which can describe both the long-term interests related to users’ hobby and temporal interests in a certain viewing.Based on it,we present a micro-video recommendation method based on hierarchical user interest modeling,which can recommend micro-videos for different users according to their personalized and current interests.The proposed method can effectively model user interest in viewing micro-videos and it outperforms the state-ofthe-art methods in micro-video recommendation.更多还原