1、Software Testing - Software Testing Fundamentals,,MA, ChaoISS, WHU,1,Outline,Motivation Why bother with testing? The Psychology of Testing What is the goal of testing? The Testing Process What is relation to other programming-related tasks? Bugs Important properties of bugs Implementation vs. Specif
2、ication: Formal definition of a bug,2,Software Errors,Therac-25 Radiation Therapy 85-87 Massive overdoses (6 deaths / amputations)!Patriot Missile Guidance System 91 (Gulf War 1.0) Accumulating rounding errors deathsAriane V 96 (one of the most expensive bugs, ever) Conversion from 64-bit float to 1
3、6-bit signed int,3,Software Errors (contd),Train Control System 98 (Berlin) Train cancellationsMars Pathfinder July 97 Periodic resets Win95/98 w/ 3rd-Party Device Drivers late 90es Dysfunction (“blue screen of death”)!,.on mars!,4,Software Errors (contd),Mobile Phones 00- Freeze and odd behaviors (
4、really annoying)!Cruise Control System Model 86 (Grady Booch) Accellerated after car ignition car crashesBaggage Handling System 94-95 (at Denver Intl Airport) $ 360,000,000 USD,5,and what about?!,Surgical Laser Control SystemOops!Air Plane Control SystemDysfunction (plane crash)!Nuclear Powerplant
5、Control SystemCore melt-down (“China-syndrome”)!,6,Outline,Motivation Why bother with testing? The Psychology of Testing What is the goal of testing? The Testing Process What is relation to other programming-related tasks? Bugs Important properties of bugs Implementation vs. Specification: Formal de
6、finition of a bug,7,Testing Incomplete Process,A program has:many possible valid inputsmany possible invalid inputsmany possible executable pathsHence:,8,8,Testing can never prove absence of errors!,Testing is an incomplete process!,8,8,Testing Incomplete Process (contd),9,The Psychology of Testing,
7、XGoal: test a program to show that it works,”Testing is the process of demonstrating that errors are not present.”,”The purpose of testing is to show that a program performs its intended functions correctly.”,”Testing is the process of establishing confidence that a program does what it is supposed
8、to do.”,10,successful test case,unsuccessful test case,Problematic terminology:,Psychology of Testing (contd),Goal: find as many errors as possible Note: realistically assumes errors are present Constructive goal (actually destructive),”Testing is the process of executing a program with the intent o
9、f finding errors.”,vs,11,Constructive vs. Destructive Thinking,Constructive thinking:(e.g., programming) ”Test-to-pass”Often not a good idea to ”test” your own code :-(,Destructive thinking:(e.g., testing) ”Test-to-fail”Often better to test/break someone elses code :-),Recommendation:Have another gr
10、oup/person help you test your ”product”,12,Testers Goal,The goal of a software tester is:Testers (therefore) often arent the most popular members of a project team,- 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed”,13,Outline,Motivation Why bother with te
11、sting? The Psychology of Testing What is the goal of testing? The Testing Process What is relation to other programming-related tasks? Bugs Important properties of bugs Implementation vs. Specification: Formal definition of a bug,14,Maintainability,Reliability,Usability,Portability,How robust is the
12、 SW wrt. external network failures, C, .?,How easy is the SW to understand? and use?,How easy is it to transfer and adapt SW to new environment / platform?,How easy is it to modify the SW? And fix errors?,Software Quality International evaluation standard,Functionality,Efficiency,ISO 9126,Does the S
13、W do what its supposed to? Does it work as intended?,How much time/memory/space/- bandwidth/ does the SW consume?,15,Testing vs. Debugging?,Testing vs. Debugging?:,Functionality:,Efficiency:,Quality Assurance:,(Functionality) Testing,(Performance) Testing,Diagnosis:,Profiling,Purpose:,Regarding:,Deb
14、ugging,16,Testing vs. Debugging (contd),Evaluate test results,Fix problem (reprogram),01101021,Program:,01101011,(greater confidence!),SYSTEMATIC,Document test results,(confidence?!?),Re-,01101011,17,Performance Testing vs. Profiling,Evaluate test results,Improve program (reprogram),01101021,Program
15、:,01101011,(greater confidence!),Document test results,(confidence?!?),Re-,SYSTEMATIC,01101011,18,White-box vs. Black-box Test,Test Case: To test whether a component conforms to project specification Black Box Testing To test whether a component conforms to development design White Box Testing,19,Wh
16、ite-box vs. Black-box Test (contd),White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”) Test focus: source code,Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”) (aka., ”input-ouput testing”)Test focus: specification (or intention),Complementary Approac
17、hes!,20,Outline,Motivation Why bother with testing? The Psychology of Testing What is the goal of testing? The Testing Process What is relation to other programming-related tasks? Bugs Important properties of bugs Implementation vs. Specification: Formal definition of a bug,21,Definition: ”bug”,Main
18、 entry: 2bug Pronunciation: /bg/ Function: noun Etymology: origin unknown Date: 1622 1 a: an insect or other creeping or crawling invertebrate (as a spider or centipede) b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious c: any of an order (Hemiptera and especially
19、its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests called also true bug 2: an unexpected defect, fault, flaw, or imperfection 3 a: a germ or microorganism especially when causing disease b: an
20、unspecified or nonspecific sickness usually presumed due to a bug 4: a sudden enthusiasm 5: enthusiast 6: a prominent person 7: a crazy person 8: a concealed listening device 9: a weight allowance given apprentice jockeys,22,”The Harvard Mark II Bug”,Photo of first actual ”bug”:,“The first documente
21、d computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested” Harvard University, Sep. 9, 1947,23,Terminology (for Bugs),”Severe conditions”: Fault Failure Defect,”Unintended operation”: Anomaly Incident Variance Featu
22、re (sounds intended),”Generic terms”: Problem Error Bug,Typically imply blame; as in: - ”it was his fault”,Famous quote: - ”Its not a bug, its a feature”,well use these terms,24,Bugs,Bugs often occur in groups: If you find one, chances are there more are nearby This property is useful when testing p
23、rogram parts: i.e., figuring out where to concentrate testing effortsNot all bugs will be fixed: Too costly? Too risky? Not ”cost-effective”? Cause unknown?,25,Cost of Fixing Bugs,Cost of bugs increases exponentially (over time):,(log),spec,design,code,test,release,$1,$100,$10,000,$1,000,000,$100,00
24、0,000,cost,phase,26,Where Bugs from,27,”The Pesticide Paradox”,”The pesticide paradox”:,”The more you test a software, the more immune it becomes to your tests”B.Beizer, “Software Testing Techniques”, 1990,28,Bug Kinds (diff. kinds of errs),Syntactic errors: Mal-formed program:Semantic errors: Symbo
25、l errors Type errors Other semantic errors: (e.g. uninitialized vars)Logical errors: Compiler: ”no errors”,int square(int x) return x*x ,* syntax error at line 2 ; expected,int square(int x) return n*n; ,* symbol error at line 2undefined variable ”n”,int square(float x) return x*x; ,* type error at
26、line 2function returns float, not int,int square(int x) return x+x; ,no errors found!,29,Outline,Motivation Why bother with testing? The Psychology of Testing What is the goal of testing? The Testing Process What is relation to other programming-related tasks? Bugs Important properties of bugs Imple
27、mentation vs. Specification: Formal definition of a bug,30,Definition: ”Bug”,A ”bug” is a relation between: Specification & ImplementationWhether or not specification is: Explicit or Implicit Written or Oral Formal or Informal(e.g., ”product spec” vs. ”back-of-envellope”),31,Specification,A spec sta
28、tes things an impl: Should do! Shouldnt do! Unspecified? (unclear, unmentioned, or left open),Should do!,Shouldnt do!,Unspecified?,S,Specs are often intentionally under-specified. Its often better to not ”prematurely commit” to a particular solution (by specifying exactly how a task should be done)
29、and instead just state which overall tasks the system should do.,32,Formal Definition: ”Bug”,A (software) bug occurs when: 1. Impl doesnt do sth spec says it should:2. Impl does sth spec says it shouldnt:3. Impl does sth spec doesnt mention:4. Spec doesnt mention sth, but should:5. Impl is hard to u
30、nderstand/use by user(s):,(or does it incorrectly),S,33,Testing Fundamentals,Intro to test cases,34,Testing:,Testing is easy: (e.g., ”random experimentation”)Testing well is not easy; requires: A SYSTEMATIC approach Test case: production evaluation documentation,35,Components of Software,客户需求 - Cust
31、omer Requirements 市场需求文档 - MRD (Marketing Requirement Document) 软件规格说明书 - Specifications 技术设计文档 Technical Design Docs 测试文档 Test Documents 在线帮助 - Online help 产品发布注释 - Release Notes / Read Me 产品软件包 - Release packages,36,Other contents of Software Product,帮助文件 Help files示例 Samples and examples to illus
32、trate points产品支持文档 Product support information错误信息 Error messages安装手册 Setup and installation instructions用户手册 User manual(s)产品标签 Label and stickers产品广告或宣传材料 Ads and marketing material ,37,Software Development Models,Waterfall ModelPrototype ModelWaterfall + Prototype ModelRapid Application Developme
33、nt Model (RAD)RAD-based V Model,38,Waterfall Model,Return,39,Prototype Model,40,Requirements Analysis,System Design,Program Design,Coding Unit Test,Integration Testing,System Testing,Acceptance Testing,Operation maintenance,Prototyping,User/Customer,Waterfall + Prototype Model,41,Rapid Application D
34、evelopment Model (RAD),42,RAD-based V Model,43,Testing Objectives,Finding errors (Right) A high probability of finding an as yet undiscovered error (Good) Uncovers an as yet undiscovered error (Successful) Design tests that systematically uncover different classes of errors and do so with a minimum
35、amount of time and effort (Perfect),44,Testing Objectives (contd),Secondary benefits include: Software functions appear to be working according to specification. Performance requirements appear to have been met. Software reliability and quality are indicated to be good.,45,Representative?Comprehensi
36、ve?Quality?Quantity?,Appropriate Test Cases?,46,Myers 10 Testing Principles,1) expected output is part of a test case2) avoid testing own program3) avoid testing own (organizations) program4) thoroughly inspect result of each test case5) also write ”invalid / unexpected” test cases6) also check: doe
37、snt do what not supposed to7) avoid throw-away test cases8) never assume no errors when making test cases9) Prob(more errors) Prob(#errors already found) 10) Testing is creative+intellectually challenging task,a) destructive vs. constructive mode b) can be overall misunderstandings,also wrt. program parts,instead: systematic ”regression testing”!,a) b) ,47,Test Information Flow,48,Q & A,49,
