机器学习平台汇报.pptx

机器学习平台汇报Jone提纲基础准备CTR平台凸函数优化MPI基础BSPBatch Learning原理评测CheckerboardIncremental Learning原理评测后续方向CTR平台MPIMassage Passing Interface:是消息传递函数库的标准规范，由MPI论坛开发，支持Fortran和CMPI是一种标准或规范的代表，而不是特指某一个对它的具体实现MPI是一种消息传递编程模型，并成为这种编程模型的代表和事实上的标准MPI有丰富的接口提供实时的点对点、Group对Group、点和Group进行数据交互的API目前主流使用的有Mpich2和OpenMPIMPIP0P0P1P2dataprocessorMPIp0p0k=m/pp0p1p2p3p1p1p2p2p3p3Layout 向量按照节点进行分割，矩阵采用row shards的方式存储加速比不算All Reduce的开销，加速比是Processor数目MPI支持P是进程单位，Mpich2必威体育精装版版本支持同一个Node启动多个PBSPBulk synchronous parallel: BSP was developed by?/wiki/Leslie_ValiantLeslie Valiant?during the 1980s A BSP computation proceeds in a series of global?supersteps. A superstep consists of three ordered stages: 1) Concurrent computation: local data, independent, asynchronously? 2) Communication: At this stage, the processes exchange data between themselves.3) Barrier synchronization: When a process reaches this point1) No deadlock and data races2) Barrier permit novel forms of fault tolerance3) Parallel graph-processing-PregelParallel DSA a[0] a[N-1] P0A a[0] a[N-1] P0 P1 P2 P3bool CreateVector(Comm,GlobalSize,LocalSize,*Vector)bool VectorSet(*Vector,double)double Dot(Vec*, Vec *)double VectorNorm(Vec *,NormType,double*)AXPY AYPX……Class Vector 逻辑抽象和物理存储分开 存储和计算都并行 通信量很少Local VectorParallel Vector凸函数优化假设 f(x)有二阶连续偏导:优化的方法： 1) 最速下降法 2) NewTon法 3) Qusai-NewTon法 4) BFGS方法 5) OWLQN所有上述方法的核心：f的一阶导数——梯度，二阶导数——HessianExample: FindMinimumPlot[Cos[x^2 - 3 y] + Sin[x^2 + y^2],{{x,1},{y,1}}] {{-2.,{x = 1.37638,y = 1.67868}},{Steps = 9,Function = 13,Gradient = 13}凸函数优化 内部向量有60个Dense Vector的向量 1亿的Feature的内部存储需要48G数据 存储：训练数据如何存储？ CPU：内部有大量超高纬向量的运算 OWLQN或者说LBFGS不存储Hessian矩阵，而是存储以前迭代的梯度信息，直接计算有哪些信誉好的足球投注网站方向S，Y中存储前m次迭代的信息使用two-loop recursion方法直接计算有哪些信誉好的足球投注网站方向Batch Learning VS Online LearningOnline algorithms operate by repetitively drawing a fresh random example and adjusting the parameters on the basis of this single example only. Batch algorithms completely optimizing the cost function defined on a set of training examplescost function