1、Shanghai Jiao Tong University M.A. Thesisproduces significantly less correct answers, with the other two procedures B (QSP) andC (AOP) falling between. Questionnaire responses suggest that subjects believe thatquestion preview can supply extra information and thus facilitate their listeningcomprehen
2、sion. The second experiment, an immediate retrospection study, attempts toprovide qualitative data by examining other 12 Chinese EFL learners listeningstrategies and tactics in completing the listening test. It is found that there are nosignificant differences between the four test procedures in tes
3、t validity.This paper concludes that the choice of different test procedures leads to differencesin test difficulty, but not in test validity. Test takers can benefit from previewingquestion stems since the previewed question stems provide extra useful information;whereas most test takers suffer fro
4、m the absence of both question stems and answeroptions before listening. The researcher recommends that the procedure allowing onlyquestion stem preview should be widely adopted.KEY WORDS: question preview, multiple choice question, listening comprehension,difficulty， validityIIIShanghai Jiao Tong U
5、niversity M.A. ThesisList of AcronymsCET-4: College English Test Band 4CET-6: College English Test Band 6TOEFL: Test of English as a Foreign LanguageIELTS: International English Language Testing SystemIBT: Internet-Based TestsEFL: English as a foreign languageSLA: Second language acquisitionVOA: Voi
6、ce of AmericaMC: Multiple choiceSAQ: Short answer questionMCQ: Multiple choice questionFQP: Full question previewAOP: Answer option previewQSP: Question stem previewNP: No previewLUS: language understanding systemGPS: general problem-solverVShanghai Jiao Tong University M.A. ThesisList of TablesTabl
7、e 1.1 Test procedures adopted in the four versions of listening testTable 3.1 Previous listening comprehension scores of the four groups in experiment 1Table 3.2 Description of four VOA news reports in the listening testTable 3.3 Previous listening test scores of the 12 participants in experiment 2T
8、able 4.1 Test scores of the four groups of students on the 4 textsTable 4.2 Multiple comparisons of the total scores in the MCQ listening test across thefour groupsTable 4.3 The descriptive statistics of the test scores of the four textsTable 4.4 Test scores across the four test procedures and texts
9、Table 4.5 The mean and standard deviation of the four MCQ test proceduresTable 4.6 Multiple comparisons of test difficulty of the four MCQ test proceduresTable 4.7 Item analysis result of the listening comprehension testTable 4.8 Studentssubjective judgment on their familiarity with four test proced
10、uresTable 4.9 Comparison of familiarity level with the four test proceduresTable 4.10 Studentssubjective judgment on the validity of four test proceduresTable 4.11 Comparison of studentsjudgment on the validity of the four test proceduresTable 4.12 Studentsrating of the relative difficulty of four t
11、est proceduresTable 4.13 Comparison of students rating of relative difficulty of four test proceduresTable 4.14 Rank order of studentsjudgment on four test proceduresTable 4.15 Test scores of participants across the four groups and textsTable 4.16 Taxonomy of listening strategies used for codingTabl
12、e 4.17 Further observations on frequencies in four test proceduresTable 4.18 Observed frequencies of valid and invalid test items in four test proceduresVI上海交通大学学位论文原创性声明本人郑重声明：所呈交的学位论文，是本人在导师的指导下，独立进行研究工作所取得的成果。除文中已经注明引用的内容外，本论文不包含任何其他个人或集体已经发表或撰写过的作品成果。对本文的研究做出重要贡献的个人和集体，均已在文中以明确方式标明。本人完全意识到本声明的法律
13、结果由本人承担。学位论文作者签名：刘淑敏日期： 年 月 日上海交通大学学位论文版权使用授权书本学位论文作者完全了解学校有关保留、使用学位论文的规定，同意学校保留并向国家有关部门或机构送交论文的复印件和电子版，允许论文被查阅和借阅。本人授权上海交通大学可以将本学位论文的全部或部分内容编入有关数据库进行检索，可以采用影印、缩印或扫描等复制手段保存和汇编本学位论文。学位论文作者签名：刘淑 指导教师签名：潘之欣日期： 年 月 日 日期： 年 月 日Shanghai Jiao Tong University M.A. ThesisAcknowledgementsFirst and foremost, I
14、 cannot sufficiently express my most sincere gratitude to mysupervisor, Professor Pan Zhixin, for her solicitude and thoughtprovoking talksthroughout my postgraduate study and especially for her enlightening instructions in mythesis writing. Her inspiration, encouragement, and guidance will have rem
15、arkableinfluence on my entire career in the future.I would like to express my gratitude to all the professors who have given thelectures and guided me into the investigation of English language during mypostgraduate study in Shanghai Jiao Tong University.I also wish to extend my thanks to all the te
16、achers and students who took thelistening comprehension test in this study with great collaboration.My sincere thanks also go to my classmates and my friends who have offered theirselfless assistance and generous encouragement, especially Zhao Lulu, Wang Yanyan,who helped me a lot in data collection
17、 in the present study.Last but not least, a special note of appreciation goes to my boyfriend, whosesupport has always been the source of my strength.IShanghai Jiao Tong University M.A. ThesisChapter 1 Introduction1.1 Background of the present studyAs is known to all, College English Test (CET), Tes
18、t of English as a ForeignLanguage (TOEFL) and International English Language Testing System (IELTS), havealmost the largest population of test takers around the world. All the three language testsadopt Multiple Choice Question (MCQ) test format in the part of or the whole listeningcomprehension test
19、; however, test procedures of MCQ vary one from another.For a long time, only two forms of MCQ test are adopted, which include theanswer option preview (AOP) form and full question preview form (FQP). Nowadays,with the increasingly widespread use of computer, internet-based test begins to beadminist
20、ered in EFL testing, which provides more options for MCQ test.For example, in 2008, CET utilized the internet-based test in some pilot testingplaces, which adopts some different practices from that of the traditional CET. In theinternet-based CET test, test takers are able to preview the question st
21、ems and answeroptions on the computer screen first, and then listen to the passage and finish all thequestions. However, the traditional CET, which is a paper-and-pencil test, has all theanswer options printed out on the test paper. In the listening comprehension part, thetest takers finish listenin
22、g to the whole passage first, and then hear the questionspresented on the tape. During the 15 second pause after each question, they answerquestions by choosing from the options the one that is most appropriate.TOEFL and IELTS are widely used as international tests of English skills. Formore than 40
23、 years, TOEFL has been regarded as the leading academic Englishproficiency test around the world and it measures the English proficiency of nonnativespeakers of North American English in reading, listening and writing. Most people whoare planning to study at colleges and universities where instructi
24、on is in English arerequired to take the TOEFL test. The old TOEFL, which is paper-and-pencil test, isquite similar to the traditional CET; whereas the new TOEFL, which is internet-basedtest, is administered on computer. And test takers are allowed to preview neitherquestions nor answer options. The
25、y need to finish listening to the whole passage first,1Shanghai Jiao Tong University M.A. Thesisand then read and finish the questions presented on the computer screen.IELTS is designed to assess the English language ability of non-native speakers inlistening, reading, writing and speaking. It is re
26、cognized by universities and employersin many countries, including Australia, Canada, New Zealand, the UK and the USA.Additionally, it is also recognized by professional institutions, immigration authoritiesand other government agencies. Practices in the old and new IELTS remain almost thesame in th
27、at full question preview is presented to test takers. That is, test takers areallowed to preview both the question stems and answer options first, and the listeningtext is presented afterwards. The difference is that in new IELTS, test takers are given20-30 seconds in advance to preview both the que
28、stion stems and answer options.1.2 Research purposeThe diversity of MCQ test procedures in listening comprehension test leads us towonder: Do different orders in which question stems, answer options and listening textsare presented lead to any significant differences in test difficulty and test vali
29、dity inMCQ listening comprehension test?By systematically combining the three basic elements of MCQ, that is, questionstems, answer options and listening texts, four plausible MCQ test procedures arefinally obtained. The four MCQ test procedures constitute:Test procedure A (FQP) allowing test takers
30、 to preview the full question, that is,both the question stems and answer options;Test procedure B (QSP) allowing test takers to preview only the question stems,but not the answer options;Test procedure C (AOP) allowing test takers to preview only answer options, butnot question stems;Test procedure
31、 D (NP) allowing test takers to preview neither question stems noranswer options.The present study conducts this investigation on the effects of the four differentMCQ test procedures on the difficulty and validity of a listening comprehension test.With the aim in mind, the study aims to answer the f
32、ollowing two research questions:Research question 1: Do the four different MCQ test procedures lead to anydifferences in test difficulty in MCQ listening comprehension test?Research question 2: Do the four different MCQ test procedures lead to any2Test Version Text1 Text2 Text3 Text4I A B C DII B A
33、D CIII C D A BIV D C B AShanghai Jiao Tong University M.A. Thesisdifferences in test validity in MCQ listening comprehension test?1.3 Research methodologyThe present study is an experimental study in which two independent experimentsare carried out to collect both the quantitative and qualitative da
34、ta respectively.In the first experiment, a MCQ listening comprehension test is conducted to 12randomly selected classes at Shanghai Jiao Tong University each of which has about 40students. There are in total 467 students who are non-English major freshmen. Based ontheir listening comprehension score
35、s in the final exam of the previous semester, all the12 classes are assigned to four parallel groups. Its indicated that there are no significantdifferences among the four groups. Their language proficiency level ranges fromlower-intermediate to higher-intermediate.First of all, a MCQ listening comp
36、rehension test paper is prepared, which entailsfour parts based on four different VOA news reports respectively. There are five MCquestions in each part and each MC question is assigned one point. Therefore, the totalscore of the whole listening test paper is 20 points. Then, four versions of the sa
37、melistening comprehension test are prepared, each of which is used among one group ofstudents. Each one of the four test versions entails the same four well-selected VOAnews reports respectively and these four VOA news reports appear in the same orderacross all the four test versions. In addition, e
38、ach one of the four test versions includesall the four different MCQ test procedures (procedure A, B, C and D); however, theorder in which the four test procedures arranged varies one test version from another.The purpose of doing this is to make sure that all the four VOA news reports are testedin
39、different test procedures in the four groups of students and all the four test proceduresare tried by each one of the 467 students (See Table 1.1). The form of four test versionsis shown in Table 1.1.Table 1.1 Test procedures adopted in the four versions of listening test3Shanghai Jiao Tong Universi
40、ty M.A. ThesisIn addition, at the end of each part, a questionnaire is attached to collect studentscomments on the four test procedures. Question 1 is aimed at investigating studentsjudgment on their familiarity with the four different test procedures; question 2 and 3 isaimed at explore students ju
41、dgment on test validity and test difficulty of the four testprocedures respectively. After the experiment, scores of the four groups will becompared and subjective judgments from questionnaires will be analyzed.The second experiment is an immediate retrospection study, which is designed toinvestigat
42、e the effects of four test procedures on test validity. This experiment isconducted among 12 participants who are, according to the final exam listening scoresof previous semester, assigned to four parallel groups. The language proficiency level ofthe 12 participants ranges from lower- intermediate
43、to higher- intermediate. For eachgroup of participants, one different listening comprehension test version among the fourversions is adopted. All the participants take part in the listening comprehension testindividually. They report their thought processes of reaching the answer to eachquestion aft
44、er they finish the 5 questions based on each listening texts. All the recallprotocols are recorded, transcribed, coded and analyzed. Based on their recall protocols,listening strategies they use are classified into valid and invalid ones. Frequencycalculation is conducted to compare test validity of
45、 the four test procedures.1.4 Organization of the thesisThe thesis is divided into five chapters.Chapter one gives a brief introduction to the research background, researchpurpose of the thesis, presenting a general layout. By reviewing the diverse practices inseveral large-scale standardized langua
46、ge tests, research questions of the present studyare put forward.Chapter two is devoted to a comprehensive review of theories and practices in thetest of listening comprehension. In the former part, a theoretical foundation is given byintroducing theories such as the nature of listening comprehensio
47、n, informationprocessing and three different models of listening comprehension. In addition, testprocedure effects on test takersperformance are reviewed and introduced, especiallyfactors affecting test difficulty and validity. In the latter part, summaries and briefcomments on previous related stud
48、ies concerning MCQ test format effect on test takers4Shanghai Jiao Tong University M.A. Thesisperformance are provided. And studies on the question preview effects are the mainfocus.Chapter three aims to address the design of the present study, giving a descriptionof the experiments. Detailed inform
49、ation is provided about the subjects, materials andimplementation in the two experiments.Chapter four describes, analyzes and discusses the data collected from theexperiments. By using tables and figures, test results are, first of all, displayed andillustrated to answer the research questions posed at the very outset and to seek forexplanations for the major results.Chapter five is the conclusion, showing the implications and limitations of thepresent study. Practical suggestions are also given for future research.5Shanghai Jiao Tong University M.A. ThesisChapter 2 Literature Re
