【作者】 张博；

【导师】 张建；

【作者基本信息】 东南大学 ， 结构工程， 2021， 博士

【摘要】 桥梁作为基础设施的重要组成部分,其健康状况直接关系到社会公共安全。在当前的人工智能时代背景下如何利用前沿技术来为桥梁的监测、运营与维护服务,成为研究人员关注的热点。本文以计算机视觉技术为基础并结合深度学习算法,针对当前基于视频图像的桥梁车流信息识别及防船撞预警方法中存在的问题进行研究。在桥梁车流信息识别中现有方法存在识别目标信息单一,鲁棒性不强等问题。针对上述问题,本文提出基于图像实例分割的车辆全面信息识别方法。并结合该识别方法与结构有限元计算,实现桥梁运营状态在线监测。而在船撞预警方面,目前缺乏有效的船舶实时定位与船桥及船间共同防撞预警方法。为此,本文提出基于单应性变换的船舶实时定位方法与基于数据驱动的船桥及船间防撞预警方法。提出的防撞预警方法主要包括用于船舶轨迹数据增强的基于多评价模型的轨迹生成式对抗网络与用于船桥及船间碰撞预警的双任务编码解码网络。本文主要研究内容及创新点包括:(1)提出基于图像实例分割的车辆全面信息识别方法,可同时实现车型识别、车辆轴数识别、车辆3D边框构建、车速测定、车辆长度计算、行驶车道识别与车辆计数,其中图像实例分割基于Mask R-CNN。在车辆信息识别中,提出了基于车辆与车轮掩膜交并比的车辆轴数识别方法。同时利用高质量的车辆分割掩膜生成车辆3D边界框,其较基于传统帧间差分或背景差分方法生成的车辆3D边界框更精确。为了计算车辆长度与速度,利用一场景消失点与车道线确定路面控制点进而求解世界坐标系与像素坐标系间的单应性变换。该单应性变换结合车辆3D边界框即可计算得到车辆长度与速度。车辆3D边界框同时也用于车辆行驶车道的识别。为了获得可靠的车辆参数,利用目标跟踪方法SORT跨帧关联车辆目标进而对多帧中识别的车辆参数进行后处理,从而获得较单帧识别结果更可靠的车辆参数。基于提出的方法,开发了车辆全面信息识别系统,并应用于安庆长江公路大桥。(2)提出一种融合视频车辆信息识别、车辆荷载分布、有限元建模与影响线计算的桥梁运营状态在线监测方法。该方法首先基于提出的车辆全面信息识别方法反演得到车辆在桥面的实时空间位置。然后基于识别得到的车辆轴数与桥梁所在地的车辆荷载分布,确定桥面每辆车的荷载区间。为了模拟桥梁的响应,建立桥梁有限元模型,并计算桥梁关键部位的响应影响线。再结合影响线正负区域与车辆位置及荷载区间,确定针对桥梁各关键部位响应上下界的车辆荷载组合。最后将每种荷载组合施加至有限元模型上,便可得计算时刻桥梁每个关键位置的响应区间。该方法无需在桥梁上布置其他传感器即可实现桥梁运营状态在线监测。(3)提出基于单应性变换的船舶实时定位方法,解决了水面船舶难以实时定位的难题。该定位方法将水面航标作为控制点来求解视频图像中像素坐标系与水面世界坐标系间的单应性变换。结合该单应性变换与船舶识别及跟踪,即可实现船舶的实时定位。在求解单应性变换中关键是获取控制点的世界坐标。本文基于几何相似关系,并借助无人机航拍与图像倾斜矫正手段一次性获得了全部控制点的瞬时世界坐标。由于水面航标即控制点位置不稳定,在计算单应性变换时,控制点的像素坐标与对应的世界坐标需保持同步。本文所提出的船舶实时定位方法,除固定摄像机外只需借助一张包含桥梁及水面航标的航拍图像,方法简便,易于实现。(4)提出基于多评价模型的轨迹生成式对抗网络用于增强船舶轨迹数据,特别是增强具有较高撞桥风险的少量异常船舶轨迹数据,进而为基于数据驱动的船桥碰撞预警提供充足的数据支持。提出的生成式对抗网络采用基于Wasserstein距离并带梯度惩罚的目标函数。网络的生成模型以门控循环单元作为基本模块,生成轨迹伪样本。在网络的评价模型中使用一维卷积与一维自适应池化运算来构建变长度序列特征提取模块。为了生成具有多样性的轨迹样本,组合不同的序列特征提取模块构成具有不同结构的多个评价模型来指导生成模型的更新。由于不同评价模型对生成模型的约束程度不同,因此基于多评价模型的网络可生成具有不同模式的轨迹样本。在训练提出的轨迹生成式对抗网络中,采用了课程式学习的策略来减轻训练早期生成模型与评价模型处理整条轨迹序列的负担。(5)提出双任务编码解码网络用于船桥及船间碰撞预警,其特点是将针对船桥碰撞预警的轨迹异常检测与针对船间碰撞预警的轨迹预测两个任务融合至同一网络中。该网络的编码器将观测轨迹编码为一固定长度的轨迹信息向量,该向量在解码器中被分别解码用于船舶轨迹异常检测与预测。在解码器的异常检测分支中,将轨迹信息向量映射为船舶航行风险度,然后基于该风险度判断是否发出撞桥预警。在解码器的预测分支中,加入了注意力机制来缓解编码器的压力,进而增加预测结果的准确性。若不同船舶在当前位置间与每个预测位置间存在小于安全阈值的距离,则说明船间有碰撞趋势进而发出预警。在双任务编码解码网络训练过程中采用三步训练方式。先单独训练网络两个分支,再联合训练。本文中开发了一个船撞在线预警系统,应用于广东九江大桥,并在监测过程中成功识别到有高撞桥风险的船舶。更多还原

【Abstract】 Bridge is an important part of the infrastructure,and its health condition is directly related to the social public security.Under the background of the current artificial intelligent age,how to use the advanced technologies to service for monitoring,operation and maintenance of the bridges has become a hot topic for the researchers.In this article,the existing problems in current video/image based methods about identification of vehicle information on the bridge and warning for avoiding vessel collision are researched based on computer vision technologies and deep learning algorithms.In the existing methods about identifying the vehicle information,there are problems that the identified vehicle information is simple and the robustness of the methods is weak.For solving the above problems,a method about identifying the comprehensive information of the vehicles is proposed based on image instance segmentation.And the proposed identification method is combined with the finite element computation of the structure for online monitoring the operation condition of the bridge.In the aspect of vessel collision warning,there are lack of effective methods about realtime vessel positioning and jointly warning vessel-bridge and vessel-vessel collisions.For these reasons,a homography based real-time vessel positioning method and a data-driven warning method about avoiding vessel-bridge and vessel-vessel collisions are proposed.The proposed warning method mainly includes the trajectory generative adversarial networks with multiple critics(TGANsMC)which is used for vessel trajectory data augmentation and a dual-task encoder-decoder network which is used for jointly warning vessel-bridge and vessel-vessel collisions.In this paper,the main research contents and innovations are as follows.(1)A method for identifying vehicle comprehensive information,including type,number of axles,3D bounding box,speed,length,current driving lane,and traffic volume,is proposed based on image instance segmentation which is realized by Mask R-CNN.In the identification of the vehicle information,a method for identifying the number of axles based on the Mask Io U between the vehicle and wheel is proposed.And the vehicle 3D bounding box is generated based on the high-quality segmented vehicle mask,which is more accurate than that generated by the traditional frame difference or background subtraction method.For calculating the vehicle speed and length,the control points on the road are determined by the lane lines and one scene vanishing point to calculate the homography.The length and speed of the vehicle can be calculated by combining the homography and the vehicle 3D bounding box.The 3D bounding box is also used for identifying the current driving lane.In order to obtain the reliable vehicle parameters,the tracking method SORT is used to associate the vehicle objects across frames,and then the vehicle parameters identified from multi-frames are post processed,so as to obtain more reliable vehicle parameters than the results identified from a single frame.Based on the proposed method,the system about identifying vehicle comprehensive information is developed and it is applied to Anqing Yangtze River highway bridge.(2)An online monitoring method about bridge operation condition is proposed,which integrates vehicle information identification,vehicle load distribution,finite element modeling and influence line calculation.Based on the proposed method about vehicle comprehensive information identification,the real-time spatial locations of the vehicles on the bridge are deduced firstly.Then,based on the identified number of axles of vehicles and the local vehicle load distribution,the load range of each vehicle on the bridge is determined.In order to simulate the response of the bridge,the finite element model of the bridge is established,and the response influence lines of the key positions of the bridge are calculated.After that,the combinations of the vehicle loads corresponding to the upper and lower bounds of the response of each key position of the bridge are determined according to the positive and negative regions of the influence line and the position and load range of the vehicles.Finally,each load combination is applied to the finite element model,and the response range of each key position of bridge can be calculated at arbitrary moment.This method can realize the online monitoring about operation condition of the bridge without installing other sensors on the bridge.(3)A homography based real-time vessel positioning method is proposed for solving the problem about positioning the vessels on the water in real time.In this method,the buoys on the water are used as the control points for solving the homography between the pixel coordinate system in the video images and the world coordinate system on the water.Combining the obtained homography with the recognition and tracking of the vessel,the vessel real-time positioning can be realized.The key in solving the homography is to obtain the world coordinates of the control points.In this paper,on the basis of geometrical similarity relation,the instantaneous world coordinates of all control points are obtained at one time with the aids of aerial photography by UAV and rectification of the oblique aerial image.Because the positions of buoys(i.e.,control points)are unstable on the water,the pixel coordinates of the buoys should be synchronized with the corresponding world coordinates in calculating the homography.In addition to the fixed camera,the proposed real-time vessel positioning method only needs an aerial image that contains the bridge and some buoys on the water,which is simple and easy to implement.(4)The trajectory generative adversarial networks with multiple critics(TGANs-MC)is proposed for augmenting the vessel trajectory data,especially the few abnormal trajectory data that have high risk about colliding with the bridge,so as to provide sufficient data for the data-driven warning of vessel-bridge collision.TGANs-MC adopts the Wasserstein distance based objective function with gradient penalty.The generator in TGANs-MC generates the fake trajectory samples based on gated recurrent unit(GRU).In the critic of TGANs-MC,1D convolution and 1D adaptive pooling are used to establish the variable-length sequence feature extraction module.For generating the multifarious trajectory samples,multiple critics with different combinations of the feature extraction modules are constituted to guide the updating of the generator.Because different critics have different levels of constraint on the generator,the trajectory samples with different patterns can be generated through the multiple critics based network.In training TGANs-MC,the strategy of curriculum learning is adopted to relieve the burden for both generator and critic in processing the whole trajectory sequence at the early stage of training.(5)A dual-task encoder-decoder network is proposed for warning vessel-bridge and vesselvessel collisions,whose innovation is to merge the trajectory anomaly detection which is for warning vessel-bridge collision and the trajectory prediction which is for warning vessel-vessel collision in one network.The encoder in the dual-task encodes the observed trajectory into a fixed-length trajectory information vector which is decoded for vessel trajectory anomaly detection and prediction in decoder.In the anomaly detection branch of the decoder,the fixed-length trajectory information vector is mapped to the risk degree of navigating,and the judgement about whether to issue the warning is made based on the risk degree.In the prediction branch of the decoder,the attention mechanism is adopted for relieving the burden of encoder to improve the prediction precision.If a distance between the different vessels’ current positions or each predicted positions is less than the safety threshold,which indicates that the vessels have the tendency to collide,a warning will be issued.A strategy of three-stage training is adopted to optimize the parameters of the dual-task encoderdecoder network.The two branches of the network are seperatedly trained firstly,and then both branches are jointly trained.A real-time collision warning system is developed,which is applied on Guangdong Jiujiang bridge.And the vessels which have high risk about colliding with the bridge are identified in monitoring.更多还原