基于深度特征交互与层次化多模态融合的情感识别模型

打开文本图片集

关键词：多模态情感识别；层次化融合；多尺度融合；特征融合

中图分类号：TP391 文献标志码：A 文章编号：1001-3695（2025）07-008-1978-08

doi：10. 19734/j.issn.1001-3695.2024.11.0487

Abstract：Multimodalemotionrecognitionhasrecentlybecomeanimportantresearchdirectioninafectivecomputing，aiming to moreaccuratelyrecognizeandunderstand human emotional states by integrating various modalitiessuchasspeechandtext. However，existingmethodslacktheprocessngofinter-modalcorelationsduringfeatureextractionandoverlook multi-scale emotionalcuesduring feature fusion.Toaddresstheseisues，thisstudyproposedadeepfeature interactionand hierarchical multimodal fusionemotionrecognition model（DFIHMF）.Inthefeature extraction stage，themodel enhanced interactionsbetweendifferentmodalitiesandextractedmulti-scaleinformationbyintroducinglocalknowledgetokens（LKT）andcrosmodal interaction tokens（CIT）.Inthefeature fusionstage，the model integratedcomplexmultimodalfeaturesandmulti-scaleemotionalcesusingahierarchical fusionstrategy.ExperimentalresultsontheMOSIandMOSEIdatasetsshow thatthemodel achieves accuracy rates of 45.6% and 53.5% on the ACC7 evaluation metric，demonstrating that the proposed method outperforms existing technologies in multimodal emotion recognition tasks.

Key Words：multimodal emotion recognition；hierarchical fusion；multi-scale fusion；feature fusion

0 引言

情感识别是自然语言处理（naturallanguageprocessing，NLP）中的一项核心任务，其目标在于分析和处理输入文本，以估计对象的情绪状态。（剩余21459字）