Assessing the Roles of Depth and Breadth of Vocabulary Knowl

Assessing the Roles of Depth and Breadth of Vocabulary Knowledge in

Reading Comprehension

David D. Qian

Abstract: This empirical study explored the relationships between depth and

breadth of vocabulary knowledge and reading comprehension in English as a

second language (ESL). Using multivariate analyses, the study examined the

roles of depth and breadth of vocabulary knowledge in assessing the

performance of a group of young adult ESL learners with a minimum

vocabulary size of 3,000 word families in carrying out general academic

reading comprehension tasks. The results support the hypotheses that (1)

scores on vocabulary size, depth of vocabulary knowledge, and reading

comprehension are highly, and positively, correlated; and (2) scores on depth

of vocabulary knowledge can make a unique contribution to the prediction of

reading comprehension levels, in addition to the prediction afforded by

vocabulary size scores. The findings from this study call for a recognition of

the importance of improving depth of vocabulary knowledge in learners' ESL

learning processes.

Ré

sumé

: Cette é

tude empirique examine les relations entre la profondeur et

l'ampleur des connaissances lexicales et la compré

hension de l'é

crit en

anglais langue seconde (ALS). A l'aide d'analyses multivarié

es, l'é

tude

examine le r?

le de la profondeur et de l'ampleur des connaissances lexicales

lors de l'é

valuation des performances d'un groupe de jeunes adultes

apprenant l'anglais langue seconde, ma?

trisant un vocabulaire minimum de

3000 familles de mots, lors de la ré

alisation de t?

ches gé

né

rales de

compré

hension de textes acadé

miques. Les ré

sultats confirment les

hypothè

ses suivantes: (1) il existe une corré

lation é

troite et positive entre les

scores portant sur l'é

tendue du vocabulaire, la profondeur des connaissances

lexicales et la compré

hension de l'é

crit, et (2) en plus des pré

dictions

suggé

ré

es par les scores portant sur la taille du vocabulaire, les scores de

profondeur des connaissances lexicales peuvent constituer un excellent

pré

dicteur des niveaux de compré

hension de l'é

crit. Les ré

sultats de cette

é

tude incitent à

reconna?

tre qu'il est important d'amé

liorer la profondeur des

connaissances lexicales dans les processus d'apprentissage de l'anglais

langue seconde.

Introduction

In first language (L1) research, it has long been recognized that vocabulary

knowledge makes an important contribution to reading comprehension

(Anderson & Freebody, 1981, 1983; Mezynski, 1983; Stratton & Nacke, 1974;

Tuinman & Brady, 1974). Second language (L2) research on the relationship

between vocabulary and reading is, however, just beginning to gain attention.

Within this area, the two ends of the scale are still rather unbalanced: the

greater part of the literature has been on how L2 learners acquire their

vocabulary through reading, while, except for a very limited number of

studies (e.g., de Bot, Paribakht, & Wesche, 1997; Hirsh & Nation, 1992;

Laufer, 1989, 1992, 1996), few studies have attempted to determine what

role vocabulary knowledge plays in L2 reading comprehension. Even within

the small number of studies that have assessed the relationship between

vocabulary knowledge and reading comprehension in L2, the majority focus

on breadth of vocabulary knowledge. Little recognition is accorded to the

roles other aspects of vocabulary knowledge play.

Defining depth and breadth of vocabulary knowledge

A recognition of depth and breadth as two primary dimensions of vocabulary

knowledge is essential to understanding the relationship between vocabulary

knowledge and reading comprehension (Qian, 1998). To map out a

conceptual framework for the present study, it is necessary to clarify what we

mean by `breadth' and `depth' of vocabulary knowledge. In the present

study, breadth of vocabulary knowledge is defined as vocabulary size, or the

number of words for which a learner has at least some minimum knowledge of

meaning. Depth of vocabulary knowledge is defined as a learner's level of

knowledge of various aspects of a given word, or how well the learner knows

this word.

Over the years, researchers such as Cronbach (1942), Dale (1965),

Henriksen (1999), Nation (1990), and Richards (1976) have proposed

varying, but generally complementary, conceptual frameworks of vocabulary

knowledge. However, produced in different contexts, these frameworks do

not provide an explicit description of what composes the mass of vocabulary

knowledge in the context of distinguishing depth from breadth. In order to

provide a theoretical framework for the present study, a working definition of

depth of vocabulary knowledge is proposed below. This framework takes into

account the merits of previous frameworks, particularly the definitions by

Nation (1990) and Richards (1976). The framework identifies various

important aspects composing depth of vocabulary knowledge for reading,

especially from the perspective of their possible contribution to reading

comprehension processes:

1. Pronunciation and spelling: how different forms of the word are

pronounced and spelled;

2. Morphological properties: the word's stem, its capability of inflection,

derivation, and other word formation devices, and its possible parts of speech;

3. Syntactic properties: the word's possible positions and its syntagmatic

relations, including collocational relations, with other words in a sentence;

4. Meaning: not only identification of the denotative meaning of a word in

context, but also, where applicable, knowledge of connotations, as well as

polysemy, antonymy, synonymy, and other paradigmatic relations the word

may have;

5. Register, or discourse features: including possible adherence to a stylistic,

social, or regional variety, and the field, mode, and manner of discourse

concerning the application of the word;

6. Frequency of the word in the language, or whether this word is a commonly

used word or one that appears only in some specialized texts.

As also mentioned in Nation's (1990) definition of word knowledge, the

present framework assumes that, while word meaning, register, frequency,

and syntactic properties are central components of depth of word knowledge,

pronunciation, spelling, and morphological properties are also considered

primary aspects of vocabulary depth for reading comprehension. This

inclusion is further supported by empirical evidence that knowledge of

morphology plays a key role in reading comprehension (Tyler & Nagy, 1990),

that phonological processing automatically takes place prior to meaning

access (Perfetti, Zhang, & Berent, 1992), and that fluency and accuracy in

phonological and orthographic processing are as important as syntactic and

semantic processing in reading comprehension processes (Nassajizavareh,

1998).

Previous research on the relationship between vocabulary

and reading

A word can be known in varying degrees (Baumann & Kameenui, 1991;

Graves, 1984; McKeown & Beck, 1988; Nagy & Anderson, 1984). In their

attempt to understand aspects of vocabulary knowledge, a number of L1

researchers have noted the importance of depth of vocabulary knowledge in

reading comprehension. Although there appears to be a virtual lack of

empirical research on this topic, the principle of the importance of depth of

vocabulary knowledge has been generally accepted in L1 research (Anderson

& Freebody, 1981, 1983; Beck, Perfetti, & McKeown, 1982; Mezynski, 1983).

The same, however, cannot be said of the situation in L2 research, where

there has been little recognition of the importance of depth of vocabulary

knowledge in reading comprehension, let alone empirical investigations on

the topic. The general lack of empirical research is especially evident with

respect to the relationship between depth of vocabulary knowledge and

academic reading comprehension. This is probably because the dimension of

vocabulary depth is more difficult to measure than that of breadth and

because, consequently, vocabulary size measures are relatively advanced in

comparison with depth measures (Schmitt & McCarthy, 1997). Although L2

researchers (e.g., Paribakht & Wesche, 1993; Read, 1989, 1993, 1995;

Wesche & Paribakht, 1996) have been developing instruments to measure

depth of vocabulary knowledge, only one known study (de Bot et al., 1997)

has in any way linked analyses about depth of vocabulary knowledge to

reading comprehension processes. However, the purpose of the de Bot et al.

study was to model lexical processing in reading, employing interview and

think-aloud protocols. The study neither attempted to conceptualize nor

focused on depth of vocabulary knowledge, even though some factors

investigated, for example, word morphology, word association, and

homonymy, did reflect aspects of depth of vocabulary knowledge proposed in

the present study. Questions on how and to what extent depth of vocabulary

knowledge contributes to reading comprehension still remain unanswered.

As mentioned earlier, in L2 research, a small number of studies (e.g., Laufer,

1989, 1992, 1996) have investigated the relationship between vocabulary

size and academic reading comprehension. Laufer (1996, 1997) found good

correlations between the vocabulary size tests and reading comprehension

tests she used. In one study (Laufer, 1992) with 92 first-year university

students whose native language was either Hebrew or Arabic, the correlation

between the scores on the Vocabulary Levels Test (Nation, 1983; see below

for details) and reading comprehension was .50 (p < .0001), and that

between the scores on a Eurocentres Vocabulary Test (Meara & Jones, 1989),

which requires the testee to say `yes' or `no' to indicate whether he or she

knows the meaning of a target word, and on reading comprehension was .75

(p < .0001). Reading comprehension in this study was measured by two

standardized reading tests: the reading comprehension section of Examen

Hoger Algemeen Vortgezet Onderwijs, consisting of two texts with 20

multiple-choice comprehension items, and an English sub-test of the Israeli

university psychometric entrance test, comprising 40 multiple-choice

questions. In another study involving 80 first-year university students of

similar L1 backgrounds in Israel (Laufer, 1996), a correlation of .71 (p

< .0001) was reported between students' scores on reading comprehension

and on the Vocabulary Levels Test.

Koda's (1989) study of 24 college students learning Japanese as a foreign

language found equally strong correlations between a self-made vocabulary

test and two reading tests, one of them a fixed-ratio deletion cloze, an

integrative procedure mainly tapping the testee's reading ability (Hale et al.,

1988), and the other paragraph comprehension. The latter contained four

passages with five short-answer questions attached to each. Koda reported a

correlation of .69 (p < .01) between the learners' scores on the vocabulary

test and their scores on the cloze test, and a correlation of .74 (p < .01)

between their scores on the vocabulary test and their paragraph

comprehension test scores.

In the behavioural sciences, a correlation r of .50 is generally regarded as

indicating a `large correlational effect size' (Cohen, 1988, p. 80), or at least

a `moderate positive relationship' (Hamilton, 1990, p. 481) between any two

variables considered, though this relation is contingent on sample size and

the distribution of variables. Following this general rule, the above reports

appear to indicate that there probably indeed exists a strong association

between the learner's breadth of vocabulary knowledge and their reading

comprehension levels, and therefore that scores on vocabulary size are

capable of predicting performance on reading tasks.

Threshold vocabulary for reading comprehension in ESL

The threshold hypothesis in reading comprehension (Clarke, 1979, 1980;

Cummins, 1979; Laufer, 1989, 1992, 1996, 1997; Nation, 1990) postulates

that, in terms of vocabulary size, there is a threshold level below which the

reader will be handicapped by a lack of comprehension and above which the

reader will be able to apply his or her reading strategies to help

comprehension and achieve better results. Laufer (1989, 1992, 1996) claims

that a threshold of 95% lexical coverage of a text is needed for minimum

comprehension, expressed in 56% on a reading comprehension test. This

95% lexical coverage translates into around 3,000 word families, or about

5,000 individual word forms (Laufer, 1996, 1997; Nation, 1993). Research by

Coady, Magoto, Hubbard, and Mokhtari (1993) confirms that explicit learning

of 3,000 high frequency English words produces considerable positive effect

on reading comprehension. Of course, this threshold merely indicates a

minimum desirable level for effective comprehension of general academic

texts; it by no means implies that 3,000 word families provide a sufficient

proficiency for satisfactory comprehension, or that, once beyond this 3,000

word-family level, the importance of contribution that breadth of vocabulary

knowledge makes to reading comprehension will diminish. In fact, recent text

readability research (Sutarsyah, Nation, & Kennedy, 1994) points to the need

for a minimum of 4,000-5,000 word families for comprehending a single

university economics textbook in English, which contains 5,438 word families

(Nation & Waring, 1997).

Relationship between depth and breadth of vocabulary knowledge

Although to date few studies have been documented on the relationship

between depth and breadth of vocabulary knowledge, support for a possible

strong link between the two dimensions is found in two recent studies

(Nurweni & Read, in press; Schmitt & Meara, 1997). Nurweni and Read, using

a self-made translation-based vocabulary size test of 200 items and a depth

of vocabulary knowledge test (Read, 1993; see below for details), found a

relatively high overall correlation (r = .62, n = 324) between the scores on

the two tests, although the sizes of correlation between the same two tests

varied considerably when the sample was split into three proficiency-level

groups. The Schmitt and Meara study on the English vocabulary knowledge of

88 Japanese young adults reports that, while the learners' knowledge of suffix

and word association and their vocabulary sizes were interrelated with one

another at various levels, the correlations between two elements examined,

namely, word association and vocabulary size, were fairly high (.62 for

productive knowledge and .61 for receptive knowledge, p < .05).

It thus stands to reason that breadth and depth are two interconnected

dimensions of vocabulary knowledge, the development of which are

interdependent to a substantial extent. With the exception of adults who are

particularly knowledgeable in a certain domain as a result of experience or

specialized training, it would be rare for an L2 reader to have superior and

in-depth knowledge of vocabulary in a given language in which his or her

overall size of vocabulary was very limited. By the same token, a large

vocabulary would probably not take a reader very far if his or her knowledge

of this vocabulary was shallow and superficial. Therefore, while the issue of

vocabulary size matters a great deal in assessing the relationship between

vocabulary knowledge and reading comprehension, the integral place of

depth of vocabulary knowledge in this relationship should also be accorded

equal recognition.

The present study

This paper reports on an empirical investigation to assess relationships

among vocabulary size, depth of vocabulary knowledge, and reading

comprehension in ESL. Specifically, it was designed to answer the following

questions:

1. How do scores on vocabulary size, depth of vocabulary knowledge, and

reading comprehension correlate with one another?

2. To what extent does depth of vocabulary knowledge add to the prediction

of reading comprehension scores, over and above the prediction afforded by

vocabulary size?

Based on the findings of relevant studies previously discussed regarding the

relationship between the learner's vocabulary size and reading

comprehension, and in line with the argument that breadth and depth of

vocabulary knowledge are closely interconnected and interdependent, the

following hypotheses were proposed:

Hypothesis 1

In ESL, scores on vocabulary size, depth of vocabulary knowledge, and

reading comprehension will correlate highly with one another at a minimum

level of r = .50 (p < .05).

Hypothesis 2

For ESL learners whose vocabulary size is beyond the 3,000 word-family

threshold level for reading comprehension, scores on depth of vocabulary

knowledge will make a unique contribution to the prediction of reading

comprehension scores, in addition to the prediction afforded by vocabulary

size scores.

Participants

Participants in the present study were all volunteers attending intensive

academic ESL programs in two universities in Ontario. Several criteria were

used to identify participants in this research. First, in line with Hypothesis 2,

this study was aimed at an ESL population whose L2 vocabulary size was at

the 3,000 word-family level or better. In other words, the participants should

all have reached the ESL lexical threshold level for reading comprehension.

Second, to minimize the influence of L1 on their ESL performance, especially

the factor of guessing in testing, the L1 background of the participants

needed be a language that is not cognate with English. For this reason,

Chinese and Korean ESL learners were sought as target groups. Although

Korean and Chinese are linguistically unrelated, the two languages are both

very different from English orthographically and phonologically. Therefore, in

comprehending an English word or text, these aspects of knowledge of their

mother tongues will not help them to guess the meaning of unknown English

words.

Third, to keep the educational levels of the participants as close as possible

while conforming to the first two criteria of recruitment, I decided that the

participants' educational levels should range from the completion of high

school, at the lower end, to the completion of university undergraduate

studies in their home countries, at the higher end. At these levels, learners

should be able to handle general academic texts, which composed an

important part of the instruments for this study.

All together, 80 volunteers were recruited for this study. However, to meet

the requirement that only learners with a threshold vocabulary of 3,000 word

families or better should be included in the sample, six learners were

excluded from further data analysis when it became clear that their scores on

the vocabulary size test (see below) were below the 3,000-word level. Of the

74 remaining participants, there were 41 Korean speakers, 5 male and 36

female, and 33 Chinese speakers, 5 male and 28 female. Their ages,

educational backgrounds, and duration of stay in Canada are reported in

Tables 1 through 3.

Instruments

Data for this study were collected through paper-and-pencil testing. Four

language tests and a questionnaire on the participants' background were

used for this purpose.

TABLE 1

Age ranges of the Korean group (n=41) and the Chinese

group (n=33)

Number of Learners

Age Range Korean Chinese

18-22 20 8

23-27 17 12

28-32 0 6

33-37 1 4

38-42 3 3

TABLE 2

Education levels of the Korean and Chinese ESL learners

Number of Learners

Educational Level Korean Chinese

High school graduates 3 6

University first year 1 1

University second year 7 1

University third year 9 1

University fourth year 5 0

University graduates 16 6

College graduates 0 18

TABLE 3

The Korean and Chinese ESL learners' duration of stay

in Canada

Number of Learners

Duration of Stay in Canada Korean Chinese

1-3 months 23 23

4-6 months 12 5

7-12 months 6 3

13-18 months 0 2

Background questionnaire

This questionnaire was designed to obtain background information on

participants' age, educational level, academic field, L1, length of time of

learning English, length of stay in Canada, and plans after completing the ESL

program.

Vocabulary size test

Originally called the Vocabulary Levels Test (Nation, 1983, 1990), this English

vocabulary size test (hereafter VS) has been accepted by a number of L2

researchers as an appropriate measure of vocabulary size (see Laufer, 1992,

1996; Laufer & Paribakht, 1998; Yu, 1996).

The VS (see Nation, 1983, pp. 19-24, for the complete test) is composed of

five parts representative of five different vocabulary size levels, namely, the

2,000 word-family level, the 3,000 word-family level, the 5,000 word-family

level, the university word list level, and the 10,000 word-family level. At each

vocabulary size level are six test items, each comprising six words and three

definitions. The test-taker is required to match the three definitions with

three of the six words provided by writing the corresponding number of the

word beside its definition, as in the example below:

1 ceiling

2 office 3 something that tells time

3 watch 5 main body of a tree

4 vehicle 6 a tool used for writing

5 trunk

6 pen

Each level contains 18 correct choices. The words at each level were selected

`so that they would be representative of all the words at that level' (Nation,

1983, p. 14). Because of the way the test was constructed, the chance of

guessing correctly is low, and testees' scores can be regarded as `a close

approximation to the proportion of words in the test that they know' (Nation,

1990, p. 262).

The 2,000 and 3,000 word-family levels of the VS include only high-frequency

words in English; the 5,000 word-family level is a boundary level between the

high-frequency and low- frequency levels; and the 10,000 word-family level

includes low-frequency words. The university word list level contains

specialized vocabulary needed for academic studies. The vocabulary items

selected for the university word list level represent words frequently

appearing in university textbooks. The list (Xue & Nation, 1984), with 737

lexical items covering over 20 academic disciplines, was developed based on

four existing lists (Campion & Elley, 1971; Ghadessy, 1979; Lynn, 1973;

Praninskas, 1972) for non-native English speakers attending universities in

English-speaking countries.

In scoring, each word correctly chosen is awarded one point. The maximum

possible score is 90 for the same number of words. In interpreting test scores,

Nation (1983), based on his research, states that a score of 12 or less out of

18 at a vocabulary size level is an indication that this level has not been

mastered. Because Hypothesis 2 in this study was aimed at a sample

population with a minimum vocabulary size of 3,000 word families, to be

eligible for inclusion in this study, a learner had to score 13 points or more at

the 3,000 word-family level.

Reading comprehension test

This test was a standardized multiple- choice reading comprehension test

(hereafter RC), taken from a form of the TOEFL (Educational Testing Service,

1987, pp. 93-100) released before the Educational Testing Service made

changes to the format of the reading comprehension section of the

paper-and- pencil TOEFL in July 1995, when the section containing discrete

vocabulary items were eliminated and vocabulary items were incorporated

into reading passages. Therefore, the RC did not contain vocabulary items.

The original RC test was composed of six reading passages and 30

multiple-choice questions to measure comprehension. Because of the

anticipated time constraints of administering the tests, considering that all

the participants were volunteers, I perceived a need to shorten the test. As a

result, two passages were randomly removed along with the comprehension

questions attached to them. The resultant test had four reading passages

with 20 multiple-choice questions. In scoring, each correct answer to a

comprehension question was awarded one point. The maximum possible

score was, therefore, 20.

Depth-of-vocabulary- knowledge test

Originally called the Word Associates Format, this

depth-of- vocabulary-knowledge test (hereafter DVK) was developed by Read

(1989, 1993, 1994, 1995) to assess depth of vocabulary knowledge in English.

A recent version of the DVK, made available for the present study, is

composed of 40 items designed to measure two aspects of the depth of

vocabulary knowledge: meaning and collocation, or the paradigmatic and

syntagmatic relationships of words. The aspects that the DVK measures

match some important components of depth of vocabulary knowledge

proposed in the present study.

Each DVK item consists of one stimulus word, which is an adjective, and two

boxes, each containing four words. Among the four words in the left box, one

to three words can be synonymous to one aspect of, or the whole, meaning of

the stimulus word, while among the four words in the right box there can be

one to three words that collocate with the stimulus word. Each item always

has four correct choices. However, these choices are not evenly spread. There

are three possible situations: (1) the left and right boxes each contain two

correct answers; (2) the left box contains one correct choice, while the right

box contains three correct answers; and (3) the left box contains three

correct answers, while the right box contains one correct choice. This

arrangement effectively reduces the chances of guessing. An example is

provided below.

sound

logical healthy bold solid snow temperature sleep dance

Most of the stimulus adjectives were selected from Barnard's Second and

Third Thousand Word Lists (Nation, 1986), which include some high

frequency academic vocabulary. The DVK was originally designed to measure

the depth of vocabulary knowledge of adult ESL learners. It went through a

process of repeated piloting and refining, including a large- scale field test (n

= 103; Read, 1994, 1995). In one trial (N = 94), a high correlation (r = .82)

was found between the DVK and a comparison vocabulary matching test. The

IRT-based Rasch reliability of the DVK in this trial was .93, and that of the

matching test was .90 (Read, 1995). In an earlier trial (n = 38), a correlation

coefficient of .85 (p < .01) was reported (Read, 1994) between the DVK and

the matching test. In scoring, each word correctly chosen was worth one

point. Therefore the maximum possible score was 160 points for the 40 items.

Morphological knowledge test

This self-made test was created specifically for the present study as a depth

of vocabulary knowledge measure complementing the DVK. The

Morphological Knowledge Test (hereafter MK) was designed to measure

morphological aspects of vocabulary knowledge, in order to obtain empirical

evidence that morphology plays a role in the process of reading

comprehension, as was assumed in the working definition of depth of

vocabulary knowledge.

The MK has 10 stimulus words shared by the two parts of the test. One part,

which contains 10 items, measures knowledge of lexical affixes and stems. All

10 stimulus words of the affix sub-test contain a prefix or a suffix. The test

requires the test-taker to define, in English or in their L1, the 10 stimulus

words, including the affixes. The other part measures knowledge of parts of

speech. In this part-of-speech sub- test, the stimulus words appear in two

forms: the stem and the stem plus an affix. This part requires the test-taker

to identify the correct parts of speech of the two forms of a given word. The 10

stimulus words of the MK were all selected from the reading passages in the

RC. The instructions and an example of the test items follow:

Instructions: Explain the meaning of the following 10 words in English, or

translate them into your first language. If the underlined part of the word has

changed the part of speech of the base word, please indicate this by using

appropriate terms provided below.

Possible terms: noun, verb, adjective, adverb, preposition

Example:

Word Meaning or translation The change in the part of speech

of the word caused by the underlined part

homeless without home n. ? adj.

In scoring, each item was worth one point. A wrong answer would get 0. In

the case of the affix sub-test, a half point was awarded to answers that bore

a close meaning to the stimulus word but failed to define the word accurately.

The maximum possible score on the affix sub-test was 10 for 10 correct

definitions; the maximum score for the part-of-speech sub-test was 20 points

for 20 correct identifications of part of speech, including 10 points for correct

identifications of 10 given forms and another 10 points for accurate

indications of changed forms under the heading `The change in the part of

speech.' The test-taker was required to indicate on the test paper the part of

speech of both the given and changed forms. If an item was left blank, no

score was awarded. Since the affix sub- test required some productive work to

define the 10 stimulus words, it was marked by two raters in order to

establish inter-rater reliability. When differences arose because of different

rater opinions, a third rater was called upon, in which case the score of the

controversial item was decided according to the majority opinion. Out of a

total of 740 items, the first two raters had different opinions on 37 items, or

5% difference. In other words, the inter-rater agreement was 95%. The 37

discrepancies were resolved with the input of the third rater.

Procedures and analyses