1、第 卷第 期 计算机应用研究 年 期 Application Research of Computers 收稿日期： ；修回日期： 基金项目：国家自然基金项目“基于动态贝叶斯网络的头姿无关自发表情识别研究”资助（61463034） 作者简介: 自发表情识别方法综述*(南昌大学信息工程学院，南昌 330031)摘要： 表情识别研究热点正从摆拍表情转移到自发表情，本文介绍了目前自发表情识别研究的现状与发展水平，详细阐述了自发表情识别研究的内容和方法，以及自发表情识别研究的关键技术，旨在引起大家对此新兴研究方向的关注与兴趣，从而积极参与对自发表情识别问题的研究，并推动与此相关问题的进展。关键词：表情
2、识别；自发表情；特征提取中图分类号: * 文献标志码: A 文章编号：（作者可不填）doi:10.3969/j.issn.1001-3695 (作者可不填)A Survey of Spontaneous Facial Expression Recognition（ Information and Engineering college, Nanchang University, Nanchang, 330031,China)Abstract: Research Focus on facial expression recognition transforms from the posing e
3、xpression into the spontaneous expression. The actuality and the developing level of spontaneous facial expression recognition at the present time is introduced in this paper, and the key technology on the research of spontaneous facial expression recognition are paid particular attention to. This p
4、aper aims to arouse peoples attention and interests into this new field, to participate in the study of the spontaneous facial expression recognition problems actively, and to achieve more successes correlated to this problem.Key words: facial expression recognition; spontaneous facial expression; F
5、eature extraction 0 引言表情识别被视为未来情感人机交互的重要技术 1，吸引了国内众多高校和科研机构参与研究 2-6。但目前国内外研究中比较常用的表情数据库中的人脸表情大都固定为正面，任务仅限于识别 Ekma 提出的六种基本表情（愤怒、高兴、悲伤、惊讶、厌恶和恐惧）。这不仅与实际表情不相符，而且忽视了真实表情中特有的脸部肌肉形变与头部运动之间的时空相关性。谷歌申请的新一代表情识别专利，挤眉弄眼方可解锁手机。荷兰开发的诺达斯（Noldus）面部表情分析系统是世界上第一个商业化开发的面部表情自动分析工具，用户使用该系统能够客观地评估个人的情绪变化，可以跟踪人的表情变化。但二者均要
6、求测试者的头姿基本保持不动，实时连续头姿估计阻碍了其在手机等智能终端领域的应用。因此，面向实际应用的非正面表情识别研究在国外日益受到重视 78。近十几年来，开展自发表情识别研究的机构主要有美国的加利福尼亚大学，卡耐基梅隆大学机器人研究所，匹兹堡大学心理学系,伊利诺伊大学，沃森研究中心，贝克曼研究所，伦斯勒理工学院，麻省理工大学媒体实验室，丹佛大学，德克萨斯大学计算机视觉研究中心，新泽西技术学院，芬兰的奥卢大学计算机科学与工程系机器视觉组，荷兰的阿姆斯特丹大学信息学院，澳大利亚的昆士兰科技大学科学与工程学院，加拿大的麦吉尔大学，日本的庆应义塾大学，爱尔兰的爱尔兰国立大学计算机视觉和成像实验室 ，
7、中国的清华大学和中国科学技术大学等已经有人做了大量工作。目前该领域比较重要的国际会议如计算机视觉与模式识别会议（International Conference on Computer Vision and Pattern Recognition, CVPR）、模式识别会议（International Conference on Pattern Recognition, ICPR）、人脸与姿态自动识别会议（International Conference on Automatic Face Gesture Recognition, FGR）。关于非正面表情识别的研究文章逐年增多，但国内刚开始涉
8、足该领域的研究 9,一些非正面表情识别中的关键技术尚有待突破。但总的来说，对目前自发表情识别的研究和探索还处于初始阶段。对自发表情识别的研究还需要我们这一代人共同的努力。本文介绍了常用的自发表情数据库以及自发表情识别关键技术研究进展。1.自发表情数据库USTC-NVIE 数据库USTC-NVIE(Natural Visible and Infrared Facial Expressions)数据库 1011是由中国科学技术大学安徽省计算与通讯软件重点实验室建立的一个大规模的视频诱发的集自发表情和人为表情的可见光和红外自发表情数据库。其中自发表情库由自发的表情序列和夸张帧组成，人为表情库仅由中性
9、帧和夸张帧组成。该表情数据库包含年龄在1731 周岁的 215 名被试者的自发和人为六种表情。VAM 数据库VAM 数据库 12采用的是以参加电视访谈节目（ TV talk show）的方式诱发的自发表情数据库，记录了节目中年龄在 16 至 69 周岁的 6 位男嘉宾和 14 位女嘉宾总共 20位嘉宾的面部表情和语音信息。该数据集由 834 名评估者使用两种方式进行评估：（1）采用 Ekman 的 6 种基本表情类别进行标注，（2）在 3 个维度上使用强度进行评估，强度等级为-1、-0.5、0、0.5、1。MMI 数据库MMI 数据库 13是采用视频的方式诱发的数据库，其中包含人为表情数据库和
10、自发表情数据库，人为表情数据库中有 61 名成年被试者，自发表情数据库有 11 名未成年人和 18 名成年人被试者，该数据库的表情种类为基本的六类基本表情和单个 AU(动作单元) 和多个 AUs 组合。RU-FACS 数据库RU-FACS 数据库 14是采用视频的方式诱发的自发表情数据库，该数据库包含 100 名成年被试者，该数据库的表情的种类为 33AUs（动作单元）。UT Dallas 数据库页码 计算机应用研究 UT Dallas 数据库 15是采用采访的方式诱发的自发表情数据库，该数据库中包含 299 名成年被试者，该数据库包含 33AUs（动作单元）。AAI 数据库AAI 数据库 1
11、6是采用采访的方式诱发的自发表情数据库，该数据库中包含 60 名成年人被试者，该数据库包含 6类基本表情和尴尬，轻蔑，羞愧以及积极和消极等类别。Sebel 数据库Sebel 数据库 17是采用视频的方式诱发的自发表情数据库，该数据库包含 28 名被试对象，该数据库包含 4 类表情。Oulu-CASIA NIR Mahoor, Mohammad H. 页码 计算机应用研究 DISFA:A spontaneous facial action intensity database. IEEE Transactions on Affective Computing, 2013, 4(2):151-16
12、0. 9Wan Shaohua.Spontaneous facial expression recognition: A robust metric learning approachJ. Pattern Recognition, 2014, 47(5): 1859- 1868.10 Shangfei Wang, Zhilei Liu, Siliang Lv, Yanpeng Lv, Guobing Wu, Peng Peng, Fei Chen, and Xufa Wang. A Natural Visible and Infrared Facial Expression Database
13、for Expression Recognition and Emotion InferenceJ. IEEE TRANSACTIONS ON MULTIMEDIA, 2010, 12(7):682-691.11Wang S， Liu Z， Lv S, et a1 A natural visible and infrared facial expression database for expression recognition and emotion inferenceJ. Multimedia，IEEE Transactions on,2010, 12(7): 682- 69112Gri
14、mm M,Kroschel K, Narayanan S. The vera am mittag german audio-visual emotional speech database. Proceedings of Multimedia and ExpoC/2008 IEEE International Conference on. IEEE, 2008, 865- 868.13M. Pantic,M.F. Valstar, R. Rademaker, and L. Maat.Web-Based Database for Facial Expression Analysis C/proc
15、. 13th ACM Intl Conf. Multimedia (Multimedia 05), 2005: 317- 321.14 M. S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J.Movellan. Recognizing facial expression: Machine learning and application to spontaneous behaviorC/in Proc. IEEE Int. Conf. Computer Vision and Pattern Recogniti
16、on (CVPR 05), 2005:568 573.15 A. J. OToole, J. Harms, S. L. Snow, D. R. Hurst, M. R. Pappas, J. H.Ayyad, and H. Abdi. A video database of moving faces and peopleJ .IEEE Trans. Pattern Anal. Mach. Intell,2005 ,27(5) : 812816 .16G.I. Roisman, J.L. Tsai, and K.S. Chiang. The Emotional Integration of Ch
17、ildhood Experience: Physiological, Facial Expressive, and Self-Reported Emotional Response during the Adult Attachment InterviewJ. Development Psychology, 2004, 40(5): 776-789.17 J. F. Cohn, L. I. Reed, Z. Ambadar, X. Jing, and T. Moriyama. Automatic analysis andrecognition of brow actions and head
18、motion in spontaneous facial behaviorC/ in Proc. 2004IEEE International Conference on Systems, Man and Cybernetics, 2004: 610-616.18 Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z. Li & Matti Pietik?inenJ. Facial expression recognition from near-infrared videos. Image and Vision Computing, 29(9):6
19、07-619, 2011.19 Pfister T，Li X B, Zhao G Y, et al. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework C/Proceedings of the IEEE International Conference on Computer Vision Workshops. Los Alamitos: IEEE Computer Society press, 2011:868-8
20、75.20Wang Peng, Ji Qiang. Multi-View Face and Eye DetectionUsing Discriminant Features J.Computer Vision and ImageUnderstanding, 2007, 105(2):99-111.21 Li Yongmin, Gong Shaogang, Sherrah J,et al. Support Vector Machine Based Multi-View Face Detection and RecognitionJ.Image and Vision Computing,2004,
21、 22(5):413-427.22 Wang Yan, Liu Yanghua, Tao Linmi, et al. Real-Time Multi-View Face Detection and Pose Estimation in Video StreamCProc of ICPR 06,2006:354-357.23 Castrillon M, Deniz O, Guerra C, et al. ENCARA2: Real-time Detection of Multiple Faces at Different Resolutions inVideo StreamsJ.Journal
22、of Visual and Image Representation, 2007:130-140.24 Xiao R，Li M J，Zhang H J Robust multipose face detection in imagesJ IEEE Transactions on Circuits and Systems for Video Technology，2004，14( 1) : 31-41.25 王文宁，王汇源. 彩色图像中多姿态的人脸快速检测及定位J.计算机工程， 2005, 31(17):149-151.26 滕少华，陈海涛，张巍. 基于超球支持向量机的多姿态协同人脸检测J. 计
23、算机应用， 2013，33(7) : 1988 199027M.S.Bartlett, G.Littlewort, M.G.Frank, et al. Fully Automatic Facial Action Recognition in Spontaneous BehaviorC/Proc. IEEE Intl Conf. Automatic Face and Gesture Recognition (AFGR06), 2006: 223-230.28G.Guo and C.R.Dyer. Learning form Examples in the Small Sample Case: F
24、ace Expression recognitionJ. IEEE Trans.Systems, Man, and Cybernetics Part B, 2005, 35(3): 477-488.29 M.Pantic and I.Patras. Dynamics of Facial Expression Recognition of Facial Actions and Their Temporal Segments from Face Profile Image SequenceJ.IEEE Trans. Systems, Man, and Cybernetics-Part B: Cyb
25、ernetics , 2006, 36(2): 433- 449.30 M. S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan. Recognizing facial expression: machine learning and application to spontaneous behaviorC/ in Proc.Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conf
26、erence on, 2005: 568-573 .31 J. R. Delannoy and J. B. McDonald. Estimation of the Temporal Dynamics of Posed and Spontaneous Facial Expression Formation Using LLEC/in Proc. Machine Vision and Image Processing Conference, 2009. IMVIP 09. 13th International, 2009: 139-144.32 S. Fujie, Y. Ejiri, K. Nak
27、ajima, Y. Matsusaka, and T. Kobayashi. A conversation robot using head gesture recognition as para-linguistic informationC/in Proc. Robot and Human Interactive Communication, 2004. ROMAN 2004. 13th IEEE International Workshop on,2004: 159-164.33 S. Koelstra, M. Pantic, and I. Patras. A Dynamic Textu
28、re Based Approach to Recognition of Facial Actions and their Temporal ModelsJ. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010,32(11):1940 - 1954.34 J. F. Cohn, L. I. Reed, Z. Ambadar, X. Jing, and T. Moriyama. Automatic analysis and recognition of brow actions and head motion i
29、n spontaneous facial behaviorC/in Proc. 2004IEEE International Conference on Systems, Man and Cybernetics, 2004:610-616.35 A. Kapoor and R. W. Picard. Multimodal affect recognition in learning environmentsC/ in Proceedings of the 13th annual ACM international conference on Multimedia, Hilton,Singapo
30、re, 2005:677-682.36 R. E. Kaliouby and P. Robinson.Real-Time Inference of Complex Mental States from Facial Expressions and Head GesturesC/in Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW04) 2004:181-200.37 Y. Tong, J. Chen, and Q. Ji. A Unified Probab
31、ilistic Framework for Spontaneous Facial Action Modeling and UnderstandingJ.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2):258-273.38 A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon. The painful face - Pain expression recognitio
32、n using active appearance modelsJ .Image and Vision Computing, 2009, 27(12): 1788-1796.39 S. Lucey, A. B. Ashraf, and J. Cohn. Investigating Spontaneous Facial Action Recognition through AAM Representations of the FaceJ. Mammendorf, Germany: I-Tech Education and Publishing, April, 2007.页码 计算机应用研究 40
33、 M. H. Mahoor, S. Cadavid, D. S. Messinger, and J. F. Cohn. A framework for automated measurement of the intensity of non-posed Facial Action UnitsC/ in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2009:74-80.41Menghua He, Shangfei Wang, Zhilei Liu, Xi
34、aoping Chen. Analyses of the Differences Between Posed and Spontaneous Facial Expressions C/2013 Humaine Association Conference on Affective Computing and Intelligent Interaction ,2013: 79-8442 P. Wang and Q. Ji.Multi-view face and eye detection using discriminant featuresJ. Computer Vision and Imag
35、e Understanding, 2007, 105(2):99111.43Yanjia Sun and Ali N. Akansu. Automatic inference of mental states from spontaneous facial expressionsC/2014 IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP),2014:719-723.44贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述 J. 计算机辅助设计与图形学学报, 2014, 26(9):
36、 1385-1395.45Pfister T, Li X B, Zhao G Y, et al. Recognising Spontaneous facial micro-expressions C /Proceedings of IEEE International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press,2011:1449-1456. 46 Wenjing Yan, Xiaobai Li, Sujing Wang, Guoying Zhao, Yongjin Liu, Yu-Hsin
37、Chen & Xiaolan Fu. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLOS ONE, 9(1), 2014.47Xiaohua Huang, Guoying Zhao, Matti Pietik?inen ，Wenming Zheng. Robust facial expression recognition using revised canonical correlationC/2014 22nd International Conferen
38、ce on Pattern Recognition,2014：1734-1739.48 M.S. Bartlett, G. Littlewort, P. Braathen, T.J. Sejnowski, and J.R.Movellan.A Prototype for Automatic Recognition of Spontaneous Facial ActionsJ.Advances in Neural Information Processing Systems , 2003,15: 1271-1278.49 J.F. Cohn, L.I. Reed, Z. Ambadar, J.
39、Xiao, and T. Moriyama. Automatic Analysis and Recognition of Brow Actions and HeadMotion in Spontaneous Facial BehaviorC/Proc. IEEE Intl Conf.Systems, Man, and Cybernetics (SMC 04), vol. 1, 2014: 610-616.50 S. Ioannou, A. Raouzaiou, V. Tzouvaras, T. Mailis, K. Karpouzis, and S. Kollias. Emotion Reco
40、gnition through Facial Expression Analysis Based on a Neurofuzzy MethodJ.Neural Networks, 2005, 18:423-435.51 Q. Ji, P. Lan, and C. Looney. A Probabilistic Framework for Modeling and Real-Time Monitoring Human FatigueJ.IEEE Systems, Man, and Cybernetics Part A, 2006, 36(5):862-875.52A. Kapoor, W. Bu
41、rleson, and R.W. Picard. Automatic Prediction of FrustrationJ.Intl Human-Computer Studies, 2007,65(8):724-736.53 G.C. Littlewort, M.S. Bartlett, and K. Lee. Faces of Pain: Automated Measurement of Spontanous Facial Expressions of Genuine and Posed Pain. Proc.Ninth ACM Intl Conf. Multimodal Interface
42、s(ICMI07),2007:15-21.54 N. Sebe, M.S. Lew, I. Cohen, Y. Sun, T. Gevers, and T.S. Huang. Authentic Facial Expression AnalysisC/ Proc. IEEE Intl Conf. Automatic Face and Gesture Recognition (AFGR), 2004.55 Z. Zeng, Y. Fu, G.I. Roisman, Z. Wen, Y. Hu, and T.S. Huang. Spontaneous Emotional Facial Expres
43、sion DetectionJ .Multimedia, 2006, 1(5) :1-8.56 H. Wang and N. Ahuja. Facial Expression DecompositionC/proc. Ninth IEEE Intl Conf . Computer Vision (ICVV03), 958,200357 Lucey S, Ashraf A B, Cohn J. Investgating spontaneous facial action recognition through aam representations of the faceJ. Face recognition, 2007:275-286.
