基于样本重要性的分布式深度学习通信优化策略

打开文本图片集

Distributed deep learning communication optimization strategy basedonsampleimportance

MENG Yugong （GuangxiCodemakerInformation TechnologyCo.，Ltd.，Nanning53ooo3，China）

Abstract：Thecomputing nodes indistributeddeep learning（DDL）need to frequentlyexchangegradientdatawith the server，whichresultsinlargecommunicationoverhead.Inviewof this，aDDLcommunicationoptimizationstrategybasedon sampleimportance isproposed.Itmainlyincludesthreecontents.Theimportancedistributionofdatasamples isexploredby confirmatoryexperiments.Theimportanceofdatasamplesisevaluatedbycross-entropylossIncombinationwiththenetwork statusawarenessmechanismandbytakingtheend-to-end network delayasthenetwork status feedback indicator，thecomputing nodesareusedtoadjustthecompressionratiosofthetransmissongradientdynamicall，whichreducesnetworktraficwhile ensuringmodelconvergence，therebyimproving thetraining eficiencyofDDL.Experimentalresultsshowthattheproposed methodanimprovecommunicationeffciencyefectivelyindistributed trainingsenariosofdiferentscalesIncomparisonwith the existing gradient compression strategies，the proposed method can reduce distributed training time by up to 40% ：

Keywords：DDL;stochasticgradient descent;sampleimportance;cross -entropy；network statusawareness;dynamic compression

0 引言

深度神经网络（DNN）被广泛用于支持越来越多的人工智能应用，如计算机视觉、自然语言处理2和网络优化等。（剩余7952字）