1、卷积神经网络CNN及其变种,山世光 中科院计算所,CNN的早期历史,卷积神经网络CNN K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980 Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W.
2、Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541551, 1989 Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278
3、2324, 1998,2,DL时代的CNN扩展,A Krizhevsky, I Sutskever, GE Hinton. ImageNet classification with deep convolutional neural networks. NIPS2012 Y. Jia et al. Caffe: Convolutional Architecture for Fast Feature Embedding. ACM MM2014 K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale i
4、mage recognition. arXiv preprint arXiv:1409.1556, 2014 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),3,卷积示例,4,卷积形式化,积分形式 = 常用表达式 =()() 离散形式 一维情况 = = = + () 二维情况 , = ,= , , , = ,=
5、, , K称为kernel,5,卷积why?,1. sparse interactions 有限连接，Kernel比输入小 连接数少很多，学习难度小，计算复杂度低 m个节点与n个节点相连O(mn) 限定k(m)个节点与n个节点相连，则为O(kn),6,卷积why?,1. sparse interactions 有限连接，Kernel比输入小 连接数少很多，学习难度小，计算复杂度低 m个节点与n个节点相连O(mn) 限定k(m)个节点与n个节点相连，则为O(kn),7,卷积why?,1. sparse interactions 有限(稀疏)连接 Kernel比输入小 局部连接 连接数少很多 学习
6、难度小 计算复杂度低 层级感受野（生物启发） 越高层的神经元，感受野越大,8,卷积why?,2. Parameter Sharing（参数共享） Tied weights 进一步极大的缩减参数数量 3. Equivariant representations 等变性 配合Pooling可以获得平移不变性 对scale和rotation不具有此属性,9,CNN的基本结构,三个步骤 卷积 突触前激活，net 非线性激活 Detector Pooling Layer的两种定义 复杂定义 简单定义 有些层没有参数,10,Pooling,11,定义（没有需要学习的参数） replaces the out
7、put of the net at a certain location with a summary statistic of the nearby outputs 种类 max pooling (weighted) average pooling,Why Pooling?,12,获取不变性 小的平移不变性：有即可，不管在哪里 很强的先验假设The function the layer learns must be invariant to small translations,Why Pooling?,13,获取不变性 小的平移不变性：有即可，不管在哪里 旋转不变性？ 9个不同朝向的ker
8、nels（模板）,Why Pooling?,14,获取不变性 小的平移不变性：有即可，不管在哪里 旋转不变性？ 9个不同朝向的kernels（模板）,Pooling与下采样结合,更好的获取平移不变性 更高的计算效率（减少了神经元数）,15,从全连接到有限连接,部分链接权重被强制设置为0 通常：非邻接神经元，仅保留相邻的神经元 全连接网络的特例，大量连接权重为0,16,Why Convolution & Pooling？,a prior probability distribution over the parameters of a model that encodes our beliefs
9、 about what models are reasonable, before we have seen any data. 模型参数的先验概率分布(No free lunch) 在见到任何数据之前，我们的信念（经验）告诉我们，什么样的模型参数是合理的 Local connections；对平移的不变性；tied weigts 来自生物神经系统的启发,17,源起：Neocognitron (1980),Simplecomplex Lower orderhigh order,18,K. Fukushima, “Neocognitron: A self-organizing neural ne
10、twork model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980,Local Connection,源起：Neocognitron (1980),19,源起：Neocognitron (1980),训练方法 分层 自组织 competitive learning 无监督 输出层 独立训练 有监督,20,LeCun-CNN1989用于字符识别,简化了Neocognitron的结构 训练方法 监督
11、训练 BP算法 正切函数收敛更快，Sigmoid Loss，SGD 用于邮编识别 大量应用,21,LeCun-CNN1989用于字符识别,输入 16x16图像 L1H1 12个5x5 kernel 8x8个神经元 L2-H2 12个5x5x8 kernel 4x4个神经元 L3H3 30个神经元 L4输出层 10个神经元 总连接数 5*5*12*64+5*5*8*12*16+192*30，约66,000个,22,LeCun-CNN1989用于字符识别,Tied weights 对同一个feature map，kernel对不同位置是相同的！,23,LeCun-CNN1989用于字符识别,24,
12、1998年LeNet数字/字符识别,LeNet-5 Feature map a set of units whose weighs are constrained to be identical.,25,1998年LeNet数字/字符识别,例如：C3层参数个数 (3*6+4*9+6*1)*25 + 16 = 1516,26,后续：CNN用于目标检测与识别,27,AlexNet for ImageNet (2012),大规模CNN网络 650K神经元 60M参数 使用了各种技巧 Dropout Data augment ReLU Local Response Normalization Cont
13、rast normalization .,28,Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.“ Advances in neural information processing systems. 2012.,AlexNet for ImageNet (2012),ReLU激活函数,29,AlexNet for ImageNet (2012),实现 2块GPU卡 输入层150,528 其它层25
14、3,440186,624 64,896 64,896 43,264 4096 4096 1000.,30,Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.“ Advances in neural information processing systems. 2012.,AlexNet for ImageNet (2012),ImageNet物体分类任务上 1000类，1,431,167幅图像,31
15、,Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.“ Advances in neural information processing systems. 2012.,AlexNet for ImageNet,深度的重要性,32,Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification wit
16、h deep convolutional neural networks.“ Advances in neural information processing systems. 2012.,VGG Net (2014),多个stage 每个stage多个卷积层 卷积采样间隔1x1 卷积核大小3x3 1个Pooling层(2x2) 16-19层 多尺度融合,K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409
17、.1556, 2014,VGG Net (2014),几种配置 Cov3-64: 3x3感受野 64个channel,34,K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014,VGG Net (2014),35,K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recogn
18、ition. arXiv preprint arXiv:1409.1556, 2014,GoogLeNet (2014),超大规模 22个卷积层的网络 计算复杂度是AlexNet的4倍左右,C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),GoogLeNet (2014),超大规模24层网络 Inception结构
19、 提取不同scale的特征 然后串接起来,C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),GoogLeNet (2014),超大规模24层网络 Inception结构 提取不同scale的特征，然后串接起来 增加1x1的卷积：把响应图的数量缩小了,C. Szegedy, W. Liu, Y. Jia, P. Se
20、rmanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),GoogLeNet (2014),39,C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),GoogLeNet (2014),在ImageNet上1000类物体分类上的性能,40,C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014),41,谢谢！,42,
