Centre for Research in English Language Learning and Assessment Однажды… CRELLA Centre for Research in English Language Learning and Assessment Однажды… ….. в 1974-oм году … CRELLA Centre for Research in English Language Learning and Assessment Assessing reading – how and why? An introduction to assessing reading for language teachers Dr Lynda Taylor CRELLA Designing tasks for testing L2 reading CRELLA We need to think about…. 1. What types of mental (or cognitive) processing are typically involved in reading? 2. What types of text are typically involved in reading activity? 3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests? Starting from the right place…. 1. the nature of the cognitive processes in reading 2. the nature of the reading texts 3. the nature of the reading question formats Starting from the WRONG place? 1. the nature of the reading question formats 2. the nature of the reading texts 3. the nature of the cognitive processes Good reading task design 1. What types of mental (or cognitive) processing are typically involved in reading? 2. What types of text are typically involved in reading activity? 3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests? What actually happens when we read? A brief look at cognitive processes in reading CRELLA Creating an intertextual representation: Construct an organised representation across texts Text structure knowledge: Genre Creating a text level representation: Construct an organised representation of a single text Remediation where necessary Building a mental model Integrating new information Enriching the proposition Rhetorical tasks General knowledge of the world Topic knowledge Meaning representation of text(s) so far Monitor: goal checking Goal setter Selecting appropriate type of reading: Inferencing Establishing propositional meaning at clause and sentence levels Careful reading LOCAL: Understanding sentence Syntactic Parsing Syntactic knowledge Lexical access Lexicon GLOBAL: Comprehend main idea(s) Comprehend overall text Comprehend overall texts Lemma: Meaning Word class Expeditious reading LOCAL: Scan/search for specifics Word recognition GLOBAL: Skim for gist Search for main ideas and and important detail Visual input Lexicon Form: Orthography Phonology Morphology Visual input Clearly, the starting point for any reading activity is the set of marks on a handwritten or printed page, or on a computer screen – i.e. a combination of letters, symbols, pictures, etc. Word recognition Word recognition involves matching the form of a word in a written text with a mental representation of the orthographic forms of the language. Field (2004:234) refers to this as “the perceptual process of identifying the letters and words in a text”. Lexical (vocabulary) access Accessing the lexical entry containing stored information about a word’s form and its meaning from the lexicon. The form includes orthographic and phonological mental representations of an item and possibly information on its morphology. The lemma (the meaning-related part of the lexical entry) includes information on word class and the syntactic structures in which the item can appear and on the range of possible senses for the word. Syntactic parsing Once the meaning of words is accessed, the reader has to group words into phrases, and into larger units at the clause and sentence level to understand the text message. Establishing propositional meaning at the clause or sentence level An abstract representation of a single unit of meaning: a mental record of the core meaning of the sentence without any of the interpretative and associative factors which the reader might bring to bear upon it. Inferencing (1) Inferencing is necessary so the reader can go beyond explicitly stated ideas as the links between ideas in a passage are often left implicit. Inferencing in this sense is a creative process whereby the brain adds information which is not stated in a text in order to impose coherence. Inferencing (2) A text cannot include all the information that is necessary in order to make sense of it. Texts usually leave out knowledge that readers can be trusted to add for themselves. If there was no such thing as inferencing, writing a text which includes every piece of information would be extremely cumbersome and time consuming. Constructing a mental model Once the reader has processed the incoming sentence and elaborated it where necessary and possible through inferencing, the new information needs to be integrated into a mental representation of the text so far. This process entails an ability to identify main ideas, to relate them to previous ideas, distinguish between major and minor propositions and to impose a hierarchical structure on the information in the text. Creating a discourse-level structure At a final stage of processing, a discourse-level structure is created for the text as a whole. The skilled reader is able to recognise the hierarchical structure of the whole text and determines which items of information are central to the meaning of the text. The skilled reader determines how the different parts of the text fit together and which parts of the text are important to the writer or to reader purpose. Establishing a mental representation across texts In the real world, the reader sometimes has to combine and collate macro-propositional information from more than one text. The need to combine rhetorical and contextual information across texts would seem to place the greatest demands on processing. Cognitive processing at A2 to C2 Word recognition Lexical access Parsing Establishing propositional meaning Inferencing Building a mental model Creating a text level structure Creating an organised representation of several texts KET A2 * * * * (*) (*) PET B1 * * * * * * FCE B2 * * * * * * CAE C1 * * * * * * * CPE C2 * * * * * * * * Goal setter The goal setter in the left hand column is critical in that the decisions taken on the purpose for the reading activity will determine the relative importance of some of the processes in the central core of the model. Types of reading Reading is either careful or expeditious and comprehension takes place at the local and global level. Local comprehension Local comprehension refers to the understanding of propositions at the level of microstructure i.e., the sentence and the clause. Basic comprehension questions are used to assess lexical, syntactic, and semantic abilities and the ability to understand important information presented in sentence-level propositions. The information used in the question and the information required for the answer are usually in the same sentence. Global comprehension Global comprehension refers to the understanding of propositions beyond the level of microstructure, that is, any macropropositions including main ideas, the links between those macro-propositions and the way in which the micro-propositions elaborate upon them. Careful reading Careful reading is intended to extract complete meanings from presented material. This can take place at a local or a global level, i.e. within or beyond the sentence right up to the level of the complete text or texts. The approach to reading is based on slow, careful, linear, incremental reading for comprehension. Expeditious reading Expeditious reading involves quick, selective and efficient reading to access desired information in a text. Expeditious reading would appear likely to include skimming, search reading, and scanning. Skimming Skimming is generally defined as reading to obtain the gist, general impression and/or superordinate main idea of the whole text. Search reading Search reading involves locating information on predetermined topics. The reader only wants information necessary to answer set questions or to provide data for example in completing written assignments. It differs from skimming in that the search for information is guided by predetermined topics so the reader does not necessarily have to establish a macropropositional structure for the whole of the text. Scanning Scanning involves reading selectively, to achieve very specific reading goals, e.g. looking for specific words/phrases, figures/percentages, names, dates of particular events or specific items in an index at the local word level. Types of reading tested at levels A2 to C2 Careful Reading Local Understanding propositional meaning at clause and sentence level Careful Reading Global Comprehend across sentences Comprehend overall text Comprehend overall texts Expeditious Reading Local Scanning or search reading Expeditious Reading Global Skim for gist Search reading KET A2 PET B1 FCE B2 CAE C1 * * * * * (*) * * * * * * (*) (*) CPE C2 * * * * Let’s look at a reading test task • Look at Task 1 which comes from a reading test. • Which of the core cognitive reading processes we have discussed does this task seem to elicit? • Which processes does this task apparently NOT elicit? Let’s look at a reading test task Word recognition Lexical access Syntactic parsing Establishing propositional meaning at clause/sentence level Inferencing Building a mental model Creating a text-level representation Creating an intertextual representation Let’s look at a reading test task Word recognition Yes Lexical access Yes Syntactic parsing Yes Establishing propositional meaning at clause/sentence level Yes Inferencing Building a mental model Creating a text-level representation Creating an intertextual representation ? When we design a reading test task… We need to be explicit about: • the types of reading that the task requires • the various cognitive processes we believe the reading task is eliciting • how well matched the cognitive processes are to the level of our students E.g. Is there a shift from tasks that focus on decoding to tasks that focus on meaning building from main ideas, to a text level representation to intertextual representation Cognitive validity in the testing of Reading The extent to which the tasks we employ in a reading test elicit the cognitive processing involved in target reading contexts beyond the test itself. What is the nature of the texts we read? A brief look at what makes a text difficult to read CRELLA Contextual features of texts and tasks in reading tests This relates to the appropriateness of both : the linguistic and content demands of the text to be processed (i.e. read and comprehended) and the features of the task setting that impact on task completion (e.g. responding to comprehension questions or writing a summary) Controlling contextual features of reading texts and tasks – 3 questions • What are the key contextual features (within-text) that we need to think about when selecting texts for reading test tasks? • What degree of complexity should we aim for in these features across each of the proficiency levels we want to test? • What are the methods that can help us to understand the difficulty level of a reading text? Contextual features in reading Some illustrative features relating to: • lexical complexity (decoding) • structural complexity (syntactic parsing) • cohesion (the construction of meaning) Some key lexical parameters L1 L2 L3 L4 L5 Syllables per word Type token ratio Word frequency Lexical density Proportion of academic words L1 : Average syllables per word • the mean number of syllables per content word multisyllabic words take longer to read and process than monosyllabic words [Rayner & Pollatsek, 1989] “In general, the more syllables per word and the more words per sentence, the higher the associated grade level of the text” [White, S. (2011) Understanding Adult Functional Literacy: Connecting Text Features, Task Demands, and Respondent Skills. Taylor & Francis] L2 : Type token ratio • the number of unique words divided by the number of tokens of the words Each unique word in a text is a word type. Each instance of a particular word is a token. When the type: token ratio is 1, each word occurs only once in the text; comprehension should be comparatively difficult because many unique words need to be encoded and integrated with the discourse context. A low type: token ratio indicates that words are repeated many times in the text, which should generally increase the ease and speed of text processing. [Templin, M (1957) Certain Language Skills in Children: Their development and interrelationships. Institute of Child Welfare Monograph Series No. 26. Minneapolis, MN: University of Minnesota Press] L3 : Word frequency • the relative frequency of occurrence of words Frequency effects have been shown to facilitate decoding: – frequent words are processed more quickly and understood better than infrequent ones (Haberlandt & Graesser, 1985; Just & Carpenter, 1980). – rapid or automatic decoding = strong predictor of L2 reading performance (Koda, 2005) – texts which assist such decoding (e.g., by containing a greater proportion of high frequency words) are easier to process…. The more frequent a word, the more likely it is to be processed with a fair degree of automaticity, thus increasing reading speed (even among lower level learners) and freeing working memory for higher level meaning building. (Crossley, Greenfield and McNamara, 2008) L4 : Lexical density • depends on distinguishing between different word types, i.e. lexical (content) and function words – lexical: verbs, nouns, adjectives, adverbs – function: auxiliaries, determiners, pronouns, prepositions, conjunctions Accessing the meaning of lexical items requires accessing the mental lexicon, function words can be dealt with by pattern matching. Reading focuses mainly on lexical items and readers tend to skip function words. L5 : Proportion of academic words • the incidence of academic words in a text • proved to be a good predictor of level in a study of FCE, CAE and CPE reading texts (Weir et al, 2012) Mean SD FCE (B2) 1.61% 1.26% CAE (C1) 1.63% 1.41% CPE (C2) 5.82% 2.84% Some key syntactic parameters S1 Sentence length S2 Readability formulae S3 Higher level constituents Syntactic complexity • linear processing of text in careful reading, with the reader decoding word by word • assembly of decoded items into larger scale syntactic structure (Just & Carpenter, 1987; Rayner & Pollatsek, 1994) • cognitive demands imposed vary considerably according to how complex the structure is (Perfetti, Landi & Oakhill, 2005) (Crossley ,Greenfield and McNamara, 2008) S1 : Sentence length in Cambridge ESOL reading papers Main Suite Level Average number Range of words per sentence KET (A2) 13.2 8 - 17 PET (B1) FCE (B2) CAE (C1) 14.9 18.4 18.6 10 - 20 11 - 25 13 - 27 CPE (C2) 19.6 13 - 30 S2 : Readability formulae • are long-established and widespread in use • rely heavily on word length and sentence length • ignore many language and discourse components that are theoretically expected to influence reading and comprehension difficulty …nevertheless… • texts with longer words and lengthier sentences are more difficult to read – longer words tend to be less frequent in the language and infrequent words take more time to access and interpret during reading – longer sentences place more demands on working memory – real-time processing means holding information in your head until you can parse sentences syntactically – the longer the sentence, the more difficult this may be S2 : Difficulty/readability estimates in Cambridge ESOL reading papers Main Suite level Flesch reading ease score Flesch-Kincaid Flesch-Kincaid grade level range KET (A2) 78.3 5.5 2 – 7.4 PET (B1) 64.7 7.9 5 – 10.1 FCE (B2) 66.5 8.4 5 – 12.3 CAE (C1) 58.4 9.6 5.7 - 16 CPE (C2) 57.7 9.9 5.6 – 16.1 S3 : Higher level constituents • the number of main verbs in a sentence is broadly indicative of the number of clauses - thus of complex syntactic composition • the more complex the syntactic composition, the greater the load on cognitive processing • the more clauses you have to process in a sentence, the more propositions you have to hold in working memory and link together Cohesion (and coherence) Cohesion is an objective property of the explicit language and text. There are explicit features, words, phrases, or sentences that guide the reader in interpreting the substantive ideas in the text, in connecting ideas with other ideas, and in connecting ideas to higher level global units (e.g., topics and themes). These cohesive devices cue the reader on how to form a coherent representation. The coherence relations are constructed in the mind of the reader and depend on the skills and knowledge that the reader brings to the situation…coherence is a psychological construct, whereas cohesion is a textual construct. [Graesser et al 2004: 193] Cohesion • two forms of textual cohesion can be estimated : • referential cohesion (the extent to which words in the text co-refer) • conceptual cohesion (the degree of similarity between concepts in different parts of a text) Let’s look at a reading text • Read the text and, as you do so, think about aspects of the text’s complexity related to : – the difficulty of the lexis/vocabulary – the difficulty of the syntax – the cohesion of the text • Don’t worry about answering the questions over the page. We shall look at those later. Analysing the text’s complexity K E T Overall Number Lexis of words Approximately Restricted to common 740-800words items which normally occur in the everyday vocabulary of native speakers. Structure P E T Approximately 1460-1590 words General vocabulary Mostly simple sentences but sufficient for most topics some use of relative and in everyday life. other subordinate clauses. F C E Approximately 2000 words Good range of vocabulary. Topics are addressed in detail and with precision. C A E Approximately 3000 words Broad range of vocabulary including idiomatic expressions and colloquialisms as well as language relating to opinion, persuasion and ideas. C P E Approximately 3000 words Mainly simple sentences A range of sentence patterns– from the simple to the complex. This level is typified by: many complex sentences Frequent use of modals Some use of ellipsis Complex approaches to referencing – range of pronouns and adverbials, as well as use of synonymy. Very wide range of Most sentences are long vocabulary including and complex. No restriction idiomatic expressions on the types of structure and colloquialisms as employed by the text. Many well as language examples of structures relating to opinion, typically used for effect in persuasion and abstract writing – sentences with ideas. several subordinate clauses, for example. What is the nature of the reading task? A brief look at question formats in reading tests CRELLA Response method • • • • • • Selected response Multiple choice (MCQ) True/false Right/wrong doesn’t say Multiple matching Gapped text • • • • • Constructed response Short answer questions Information transfer Random deletion cloze Selective deletion gapfilling • Reading into writing (e.g. summary) Position of the test questions • Before the reading text? or • After the reading text? Let’s look at some test questions • Question 1 • Question 2 • Question 3 • Question 4 Context validity in Reading Context validity relates to the appropriateness of both the linguistic and content demands of the text to be processed, and the features of the task setting that impact on task completion. When designing reading test tasks we need to take account of…. 1. What types of mental (or cognitive) processing are typically involved in reading? 2. What types of text are typically involved in reading activity? 3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests? Спасибо за внимание!