数据科学中的R语言.pdf

1、.数据科学简介数据科学与R如何成为数据科学家数据科学中的语言李舰中国R语言会议（广州会场）2014年11月15日ChinaR 7th Guangzhou R in Data Science 1/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区目录1数据科学简介什么是数据科学数据科学的误区2数据科学与R3如何成为数据科学家ChinaR 7th Guangzhou R in Data Science 2/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区目录1数据科学简介什么是数据科学数据科学的误区2数据科学与R3如何成为数据科学家ChinaR

2、 7th Guangzhou R in Data Science 2/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区什么是数据科学？数据科学的来历Wikipedia上目前最早考据到上个世纪60年代Peter Naur提出了这个概念郁彬教授认为上个世纪40年代Turner和Carver等人就提出了数据科学的思想C.F. Jeff Wu于1997年非常旗帜鲜明地提出了“Statistics =Data Science?”从2008年DJ Patil和Jeff Hammerbacher把他们在LinkedIn和Facebook的工作职责定义为“数据科学家”的那段时期开始

3、，数据科学开始在业界流行起来定义数据科学是使用科学方法从数据中获取知识的学科Wikipedia上的定义：数据科学是一门利用数据学习知识的学科，其目标是通过从数据中提取出有价值的部分来生产数据产品ChinaR 7th Guangzhou R in Data Science 2/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区什么是数据科学？计算机科学机器学习统计学商业智能数据科学传统数据分析领域知识ChinaR 7th Guangzhou R in Data Science 3/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区基于统计的分析

4、x1Fertility Agriculture Examination Education Catholic bluelightmediumdarkfairredmediumdarkblacka710.5 1 2051015202530350.5 1 205101520253035Ascorbic acidOrange juiceSex: MaleAdmit?: YesSex: FemaleAdmit?: No119855714931278a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71

5、a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a711 2 3 4 524681012xya71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a7

6、1a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a7

7、1a7110.80.60.40.200.20.40.60.81mpg cyl disp hp dratwt qsec vs am gear carbmpgcyldisphpdratwtqsecvsamgearcarb100200300400500600700010203040012345IDTimehamdelta71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71 a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a

8、71a71a71a71a71a71a71 a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71 a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a7

9、1a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71 a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a

10、71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71Observe

11、dABSimulatedABLDChinaR 7th Guangzhou R in Data Science 4/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区基于计算机的分析210 1 2 3 4 501234x1x2x20x10x1+2x223x14x212x121012x210123z50010001500200025003210 1 2 3321012Component 1Component 2These two components explain 95.81 % of the point variability.a71a71a71a71a71a71a7

12、1a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71Neighbour distance plota71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71

13、a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71123456glucose = 0.13n

14、egnegneg posposposyes no6 8 1012141618202.02.53.03.5SVM classification plotHwtBwtoooooooo ooo ooooooooooooooo ooooooooo oooo ooo ooooooxxx x xxxxxxxxxx x xx xx xx xxxx xxxx xxx xx xx xx xxxx xxxxx xa71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71

15、a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71 a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71StandardizedResiduals:4ClassSex1st2nd 3rd CrewMaleFemaleChildAdultNoYesNoYesChildAdultChildAdultChild

16、Adultyearsgrowthriyalsunitedmuchpresent158productiongradeyorkplanssixforeignimportsquotedfreesweetoutputexplorationyesterdaybudgetdaily198586power198788alkhalifaannouncedministerexchangerevenuemoveslowqatarremainweakabdulazizfuturebuyersreiteratedhelpbringsanalystsproducerpctrevenuesslightly alsabah

17、highsetaddressloweredfixedcompanyboostyearindonesiaaccordreuterquotabpd1986publishedmonthprojectedexportsexpectedbutresearchemiratesspokeswomansharpdayintermediatearabiasprotectchangeindonesiaseffectivepositioncompanieswill nymexshipdecrease150spatheygroupbblmayrealproducingbillionsincecountrysnazer

18、todayestimatesefptradersnewmemberseconomicmaincrudesmaller1987developmenttexacoopecalqabasaroundmarchmeetingrecentfebruaryagreementarabagencyreducedduegulfhishamkuwaitagreedsellarabianbackamongabilityaskedpolicyemergencystatedecembermizrahidemandsectorthree1985comparedwayperenergypetroleumexpenditur

19、ecutcannamedpostedkingdomsstatesseverallastaddedcommitmentmckiernanmeetcalledproblem13nationmustincreasedollarsbarrelsfellinternationalbankmadetransactionholdstrategicbarrelappearstwosayingnowpumpingcontractfallsaytradingincmlnplantnextoneguardneverweekfourofficialsrulelocalaliapriltexasthedeniedcor

20、poilexportpressuresaidhowevermarketfuturesaccordingrefinerysheikhdomesticalsolowerreportsourcessaudirisetotalstudysaysaverageindustryceilingjanuaryreserveestimatewestgovernmentcurrentmeasuresworldeconomyweeksreservesthisdlrshighermarketsarabialevelspricepresidentopecskuwaitsofficialMatrix with 371 r

21、ules123456246810120100200300400Consequent (RHS)Antecedent (LHS)liftChinaR 7th Guangzhou R in Data Science 5/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区基于领域的分析ChinaR 7th Guangzhou R in Data Science 6/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区另一张流传很广的韦氏图注意其中的Danger Zoneaa作者是Drew Conway，最早发布于http:/ 7th Guangzho

22、u R in Data Science 7/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区目录1数据科学简介什么是数据科学数据科学的误区2数据科学与R3如何成为数据科学家ChinaR 7th Guangzhou R in Data Science 8/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区只会套用方法而不理解数据的不是数据科学a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71

23、a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a715 10 15 202628303234363840TimeY5 10 15 202628303234363840TimeYa71a71a71a71a71a71a71a71a71a71a71a71a71a

24、71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71a71ChinaR 7th Guangzhou R in Data Science 8

25、/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区寄希望于“黑箱”工具的不是数据科学ChinaR 7th Guangzhou R in Data Science 9/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学数据科学的误区无视数据而信仰“科学”的不是数据科学ChinaR 7th Guangzhou R in Data Science 10/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的R目录1数据科学简介2数据科学与R什么是R数据科学中的R3如何成为数据科学家ChinaR 7th Guangzhou R in Data Sc

26、ience 11/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的R目录1数据科学简介2数据科学与R什么是R数据科学中的R3如何成为数据科学家ChinaR 7th Guangzhou R in Data Science 11/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的R什么是R？R语言国人习惯于把能编程的东西成为语言R最大的特色是R语言通常用R语言来指代RR软件/ R軟體从应用的角度来说R是一个软件不同的程序包（package，也称套件）可以解决不同的问题R环境R的官方定义是一个统计计算和绘图的环境R既是语言、也是软件、也是开发和应用环境China

27、R 7th Guangzhou R in Data Science 11/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RR的历史（I）S语言是R语言的前身S语言诞生于贝尔实验室统计研究部。1976年Chambers和他的同事用Fortran实现了初步想法，称为“S1”。到1992年的时候，已经发展到了“S3”。1993年，S语言的许可证被MathSoft公司买断，S-PLUS成为其公司的主打数据分析产品，1995年发展到了“S4”。1998年美国计算机学会（ACM）授予了S语言的主要设计者Chambers“软件系统奖”。2008年，TIBCO收购了已改名成Insight

28、ful的原MathSoft公司，目前的S-PLUS已经纳入了Spotfire平台。R语言吸收了很多Scheme语言的特性Scheme语言1975年诞生于MIT，是LISP的一个方言。很久以前，有一次R语言的作者Ross准备用Scheme向别人演示词法作用域的时候，由于手边没有Scheme，就用S来演示却失败了，这让他萌生了改进S语言的想法。ChinaR 7th Guangzhou R in Data Science 12/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RR的历史（II）R语言诞生新西兰Ross Ihaka和Robert Gentleman在奥克兰大学成为同

29、事，他们最初希望在Mac环境下开发一个统计计算软件，于是模仿Scheme，使用C开发了一个解释器，并采用S的语法。1993年，Ross和Robert将R的部分二进制文件放到了卡耐基梅隆大学统计系的Statlib中，并在S语言的新闻列表上发布了一个公告。1995年6月，在很多人的建议下，R终于在GPL协议下作为开源软件发布了。1997年，R核心团队成立1997年第一批核心团队的成员数目为11位；2008年R核心团队成员数目增加到了19位；2011年至今，R核心团队成员数目达到20位。ChinaR 7th Guangzhou R in Data Science 13/33.数据科学简介数据科学与R

30、如何成为数据科学家什么是R数据科学中的RR的特点John M. Chambers在2009年第一期R Journal上对R的定义：an interface to computational procedures of many kinds（各类计算过程的接口）；interactive, hands-on in real time（具有可交互性，可以实时手动操作）；functional in its model of programming（函数式编程模式）；object-oriented，“everything is an object”（面向对象，“所有东西都是对象”）；modular，bu

31、ilt from standardized pieces（模块化，由标准化块构建）；collaborative，aworld-wide，open-source effort（协作性，全球范围的开源力量）。ChinaR 7th Guangzhou R in Data Science 14/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RR的优势灵活的语言为数据而生的程序设计语言一个设计理念：人的时间永远比机器的时间宝贵混搭的平台基于S/R语言进行数据操作、建模和绘图类似Scheme的词法作用域和内存管理机制调用C或Fortran进行底层运算早期版本大量使用Perl进行系统的

32、交互在业界中常作为运算和绘图引擎嵌入到JAVA系统中强大的社区CRAN上已包含超过6000个第三方包R社区最大的特点是来自具体领域的数据科学家数目众多R的使用者可以分为“用户”和“开发者”，这在编程语言中是非常少见的注意：使用R要学杨任不要学黄天化ChinaR 7th Guangzhou R in Data Science 15/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RR的性能问题固有的缺陷无法多线程权衡的牺牲解释型语言与交互式环境，可以使用compiler包来弥补免费工具与代数运算库，可以换用商业BLAS/LAPACK统计建模与计算机编程，可以使用C或者Fort

33、ran开发对性能要求高的函数（注意，要多使用内置函数）并非特有的缺点“基于内存的计算是个缺陷”。除了SAS等少数分析工具可以隐式执行内存外分析之外，数据量大都会撑爆内存。只是其他语言的用户不大容易遇到大数据分析的问题。解决办法都是内存外运算或并行，可以参考R中的bigmemory和parallel等包“难以处理大数据”。数据大到一定程度之后就不是编程语言的问题了，业界通常是借助于数据库或者并行系统来解决，可以参考R中的ORE或者Rmpi、RHadoop等包ChinaR 7th Guangzhou R in Data Science 16/33.数据科学简介数据科学与R如何成为数据科学家什么是R

34、数据科学中的R目录1数据科学简介2数据科学与R什么是R数据科学中的R3如何成为数据科学家ChinaR 7th Guangzhou R in Data Science 17/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的R哪些公司在使用R？ChinaR 7th Guangzhou R in Data Science 17/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RKDNuggets的调查2014年你在数据分析/数据挖掘/数据科学工作中使用过的编程语言或者统计语言有哪些？该项调查于2014年8月进行，共有719人参与了投票R语言得票率49%，排名第

35、一SAS排名第二；Python排名第三；SQL排名第四；Java排名第五在过去的一年里你在实际项目中用到的数据分析/数据挖掘/数据科学软件或工具有哪些？该项调查于2014年5月进行，共有3285人参与了投票R语言得票率38.5%，排名第二RapidMiner得票率为44.2%，排名第一Excel排名第三；SQL排名第四；Python排名第五ChinaR 7th Guangzhou R in Data Science 18/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的R数据科学中各部分的常用工具FortranPythonSASRapidMinerRSPSSExcel计算机

36、科学 统计学领域知识ChinaR 7th Guangzhou R in Data Science 19/33.数据科学简介数据科学与R如何成为数据科学家什么是R数据科学中的RR与工程开发ChinaR 7th Guangzhou R in Data Science 20/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学家数据科学家的必备技能目录1数据科学简介2数据科学与R3如何成为数据科学家什么是数据科学家数据科学家的必备技能ChinaR 7th Guangzhou R in Data Science 21/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学家数据科学家

37、的必备技能目录1数据科学简介2数据科学与R3如何成为数据科学家什么是数据科学家数据科学家的必备技能ChinaR 7th Guangzhou R in Data Science 21/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学家数据科学家的必备技能数据科学家是21世纪最性感的职业Data Scientist: The Sexiest Job of the 21st CenturyaaHarvard Business ReviewChinaR 7th Guangzhou R in Data Science 21/33.数据科学简介数据科学与R如何成为数据科学家什么是数据科学家数据科学家的必备技能谁更性感？1 2 3 4 5 6 7 801234Petal and Sepal Dimensions in Iris BlossomsLengthWidthssssssssssssssssssssssssssssSSSSSSSSSvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvVVVVVVVVVVVVVVVVVsSvVSetosa PetalsSetosa SepalsVersicolor PetalsVersicolor SepalsChinaR 7th Guangzhou R in Data Science 22/33