How to Test Speaking Skills in Japan

How to Test Speaking Skills in Japan: A Quick-Start Guide

2017年4月15日 初版第1刷発行 2017年11月11日 第2版第1刷発行 Author Jerry Talandis Jr. Publisher

株式会社アルマ出版 Alma Publishing www.almalang.com Tel: 075-203-4606 Fax: 075-320-1721 Email: info@almalang.com

Design

Inside: Jerry Talandis Jr. Malcolm Swanson Cover: Eric Vannieuwenhyuse

印刷・製本

株式会社太洋堂 乱丁・落丁はお取替え致します。

© 2017株式会社アルマ出版 Alma Publishing ISBN 978-4-905343-14-1

ii

Contents Introduction ........................................ 1 Part 1: Why should we bother with speaking tests? ............ 6 Chapter 1: The conundrum of teaching English in Japan ......... 8 The gap between stated aims.... . . . . . . . . . . . . . 8 …and the reality on the ground. . . . . . . . . . . . . . 9 What’s to blame for such poor English skills?. . . . . . . . . 10 A classic case of negative washback. . . . . . . . . . . . 12 Chapter 2: Embrace testing ............................. 16 Tests are not inherently evil. . . . . . . . . . . . . . . 16 Alltestsarenotthesame. . . . . . . . . . . . . . . . 17 Norm-referenced & criterion-referenced: two different solar systems . . . . . . . . . . . . . . . 18 Choose the right tool for the right job. . . . . . . . . . . 20 Key principles for evaluating tests: Reliability, validity, and practicality. . . . . . . . . . . . 21 Findtherightbalance.. . . . . . . . . . . . . . . . 25 Chapter 3: Work for positive washback ..................... 27 Recappingtheproblem. . . . . . . . . . . . . . . . 27 Flipthescript.................... 28 Harness your imagination…. . . . . . . . . . . . . . 29 …andbethechange.. . . . . . . . . . . . . . . . 30

iii

Part 2: How can we mark & administer speaking tests? ........ 34

Chapter 4: Holistic or analytic? Decide your overall approach . . . . . 36 What is holistic marking?. . . . . . . . . . . . . . . 37 What is analytic marking?. . . . . . . . . . . . . . . 39 Make peace with subjectivity. . . . . . . . . . . . . . 43 Whichwayisbest?.................. 44 45 Holistic approach: Define each level. . . . . . . . . . . . 46 Analytic approach: Choose your constructs wisely. . . . . . . 49 How to adapt descriptors from the literature. . . . . . . . . 52 Remember this when adapting rating scales . . . . . . . . . 59 A simplified combo rubric format . . . . . . . . . . . . 59 A great idea: Involve your students. . . . . . . . . . . . 62 Decide how to score each construct. . . . . . . . . . . . 62 Chapter 6: Plan out the details ahead of time ................ 65 Create a marking/feedback sheet . . . . . . . . . . . . . 65 The benefits of recording your exams. . . . . . . . . . . . 68 Make time for more extensive individual feedback. . . . . . 70 Holdapracticetest.. . . . . . . . . . . . . . . . . 72 What is the best test location? . . . . . . . . . . . . . . 73 Consider how you assign test partners. . . . . . . . . . . 75 How often should you conduct speaking tests?. . . . . . . . 77 A powerful record-keeping tool. . . . . . . . . . . . . . 78 Insummary.................... 81 Chapter 5: Identify & define your evaluation criteria

iv

Part 3: Three speaking test ideas ........................ 84

Chapter 7: Write & Perform a Conversation .....................86 Gettingready.................... 90 Step-by-step. . . . . . . . . . . . . . . . . . . . 90 Marking&admintips. . . . . . . . . . . . . . . . 92 Variationideas................... 93 Follow-upactivities.. . . . . . . . . . . . . . . . . 93 Finalthoughts................... 94 Chapter 8: Random Partner Conversation ................... 96 Gettingready.................... 98 Step-by-step. . . . . . . . . . . . . . . . . . . . 99 Marking&adminTips. . . . . . . . . . . . . . . 100 Variationideas................... 101 Follow-upactivities.. . . . . . . . . . . . . . . . 102 Finalthoughts................... 103 Chapter 9: Record & Transcribe a Conversation .............. 105 Gettingready................... 107 Step-by-step. . . . . . . . . . . . . . . . . . . . 109 Marking&admintips. . . . . . . . . . . . . . . . 112 Variationideas................... 114 Follow-upactivities.. . . . . . . . . . . . . . . . 114 Finalthoughts................... 116 Epilogue ......................................... 118 Appendices ....................................... 120 References ....................................... 126 Resources ....................................... 131

v

Introduction

In my professional travels, especially in Japan, I found that most teachers are intimidated by tests and maybe even a little fright ened of them. Why is that? I believe that one cause is the fact that no one has ever taken the trouble to clearly explain to most teachers how very useful tests can be to their work.

-J.D. Brown (1995: 12)

T his is a book for Japan-based EFL teachers interested in learn ing how to conduct effective in-class speaking tests in order to help their students speak English more fluently It’s written for new teachers, fresh off the boat, and also for those experienced in structors who would like a review of basic concepts and some fresh new ideas The basic premise is that testing should be embraced and utilized as a powerful tool to help our students learn Testing in Japan has a bad rap in the eyes of many students and teachers, of ten with good reason However, if we approach assessment from an integrative perspective, one where our tests closely resemble regular language practice activities, we can enlighten ourselves and our stu dents about the effectiveness of the learning and teaching that goes on in our classrooms (Brown, 2013) In other words, testing should reflect teaching (Mendelsohn, 1989) This book is my contribution towards making this happen for you in clear and practical ways

1

2

Introduction

I’ve been teaching in Japan now for over 20 years, since 1993, and during most of that time I rarely felt confident in my assessment skills, especially in oral communication classes I was under the mistaken impression that evaluating speaking skills was extremely difficult and time-consuming, so I either avoided it altogether or conducted unsatisfactory written exams When checking out some books on the subject, I was often put off by their dense and techni cal nature or overwhelmed by the amount of information provided Regression? Analysis of variance? 15 different types of validity? Hmm… my head hurts. Oh, look—sixty techniques for testing speaking skills! Great! But which one should I choose? My class starts in a few hours… Complicating matters was the fact that most of these books were written for a global audience, to be used in a variety of teaching contexts in widely differing cultures Hmm… interesting idea, but will this really work with my Japanese students? As a result, I rarely felt like the information spoke directly enough to my needs here in Japan, where we face our own particular set of challenges in getting our students to talk What I really could have used early on was a colleague to sit down and explain to me in words I could under stand just why I should bother with speaking tests and how I could administer some specific ideas that were tailored to my teaching sit uation So, while I’m no expert on language testing, I am willing to take on this role for you by sharing what I have learned over the years, one teacher to another Instead of a comprehensive, authoritative tome that addresses every possible teaching situation, with this book I’m aiming to gen erate value via a less is more approach that gives you, the Japan-based educator 1 , just what you need to get up and running with speaking tests Based on my experience, I’m convinced that once you under stand a few key basic principles, learn how to mark, administer, and 1 Even if you are not currently teaching in Japan, much of the information in this book should still proove valuable because the core theme of tests as learning tools still applies. Keep your local context in mind as you read and make use of what you can.

3

Introduction

evaluate an oral test, and then try out a few simple and proven activ ities, you will start to see significant improvements in your students’ speaking skills Once you taste a bit of success, my hope is that any lingering intimidation or fear surrounding language testing will be lessened, and you’ll be encouraged and motivated to seek out some of the amazing, higher-level literature and resources in order to de velop your own ideas and approaches

This book is a bridge between theory and practice

I therefore see this book as a bridge of sorts, between language testing theory and its actual practice here in Japan You know that old saying, the one about giving someone a fish versus teaching them how to fish? Well, I want to do both: give you some practical testing activ ities, marking strategies, and resources you can use today while simul taneously imparting basic background information so you can test your students with self-awareness and conviction for the remainder of your career Think of this book as your quick-start guide —that thin, colorful, and easy-to-read pamphlet included with the new electronic gadget you’ve just bought You don’t need to read the thick manual cover to cover before you can start using it Let’s jump right in and get started Transformation of your teaching practice awaits! How is this book arranged? There are three parts to this book In Part 1 , the main focus is on making the case for conducting and evaluating speaking tests by

4

Introduction

outlining the benefits and basic principles involved The job of Part 2 is to give you a clear idea on how to mark and administer a speaking test Here you will learn how to define various aspects of conversation in measurable ways, such as fluency, accuracy, and pronunciation Practical tips for administering oral tests will also be covered In Part 3 , three practical and proven speaking test ideas will be offered, each one presented in a detailed step-by-step for mat optimized for typical teaching situations—large classes filled with lower-level students, regular midsize groups, and small classes with higher-level learners The Appendices at the end of the book contain all of the test handouts you’ll need as well as a few model marking/feedback rubrics you can use for inspiration on construct ing your own Finally, the resources section highlights a few useful places to get information and tools that can help you take your lan guage testing skills to the next level Acknowledgements For the inspiration to begin this project and the encouragement to not give up, I’m deeply indebted to Bruno Vannieu, an award-win ning French teacher at Kobe University Bruno was also instru mental in helping develop the testing ideas put forth in this book and provided invaluable feedback Merci beaucoup, Bruno! I’m also grateful to my family for their patience while I struggled to get this project done by the deadline Next, I’d like to thank my students at the University of Toyama for granting me permission to take their photos during our speaking tests Thanks to their cooperation, the assessment activity directions in this book are now richly illustrated and much easier to follow Finally, I want to give a big shout out to Malcolm Swanson for some life-saving, last-minute help on the design and layout of this book

Part 1 Why should we bother with speaking tests? Chapter 1: The conundrum of teaching English in Japan Chapter 2: Embrace testing Chapter 3: Work for positive washback Communicative language testing and teaching are seen as two sides of the same coin . . . Having teaching and testing compatible is essential if we expect our students to learn what we teach them. -Meryl Swain (in Mendelsohn,1989: 107) B efore embarking on any new endeavor, it’s really important to fully understand what you are getting yourself into In this book, I’m basically asking you to learn new things, to expand your teaching and testing skills beyond what they are now In other words, I’m asking you to work harder, to put more time and energy into your job Being quite busy myself, I know full well what a big ask this is I therefore feel compelled to begin by directly addressing this critical why bother question When we have 59 other things we could be doing with our time, why should we invest our energy here, into learning how to con duct speaking tests?

6

7

Part 1: Why should we bother with speaking tests?

The case for upgrading your oral assessment skills begins with an overview of the conundrum we face when teaching English in Japan, caught as we are in the gap between stated aims for the lan guage and the reality of how it is taught and learned on the ground Next, we’ll investigate the key concept of washback effect , look at a critical distinction between test types, and then reacquaint ourselves with reliability , validity , and practicality —key principles we’ll need to understand in order to evaluate tests with insight and conviction Finally, some of the main benefits that a successful oral assessment approach can have for our students, teaching practice, and the field of ELT in Japan will be considered It’s my hope that reflecting on the powerful impact that good testing practice can have will help provide you with a motivating sense of perspective and clarity that leads to positive action

Chapter 1 The conundrum of teaching English in Japan

The gap between stated aims … If you’ve taught English in Japan for any length of time, you are no doubt familiar with the fundamental conflict at the core of ELT in this country: the vast gap between official language policy and what is actually done in the classroom (Tahira, 2012) As you know, En glish is one of the major core subjects taught throughout all levels of education It enjoys an equal status to other mainstays, such as math, science, social studies, and Japanese The language has been taught for decades during the six years of junior and senior high school Students going on to university are also often required to study it further In addition, now that English is making its way into many elementary school curriculums (Machida, 2016), it will be quite commonplace for most Japanese citizens to have had at least eight to ten years of study by the time they reach adulthood In fact, the emphasis on English is so strong that it has come at the expense of other foreign languages, none of which come close in terms of

8

9 Chapter 1: The conundrum of teaching English in Japan

status and support (Yoshida, 2003) In addition, English enjoys the full backing of the Japanese government, albeit largely for national economic goals (Hagerman, 2009) According to Tahira (2012), the Ministry of Education, Cul ture, Sports, Science, and Technology (MEXT) has been regularly publishing a set of educational guidelines known as The Course of Study since the end of World War II These are now considered the official principles for English education in all public schools Their role is to provide overall goals and appropriate curricula The follow ing excerpt from the MEXT English website provides a cogent snap shot of governmental hopes and dreams for English (MEXT, 2014): Amid ongoing globalization, the development of students’ pro ficiency in English, a common international language, is crucial for Japan’s future . Japanese people should expect to achieve top-level English proficiency in Asia. Under the reform of English education, not only the cultivation of fundamental knowledge and skills, but also such abilities as to reason, make decisions or express oneself in order to solve problems by oneself will be inevitable… With an eye to the year 2020 in which the Tokyo Olympic and Paralympic Games will be held, the government will proceed with studies on the new reform of English educa tion throughout elementary, junior high and high schools so that the reform will be performed in a stepwise fashion. Crucial for Japan’s future… achieve top-level English proficiency in Asia…the new reform of English education … these are all strong words that highlight bold and ambitious aims From these statements it is clear that MEXT wants the ELT system to be more effective in pro ducing students who not only understand English, but also actually use it to communicate effectively (Ainscough & Shiozawa, 2000) … and the reality on the ground Given all of the institutional guidance and resources put into En-

10 Part 1: Why should we bother with speaking tests? glish, you would think that most Japanese citizens could speak the language quite well, right? Well, as you may know from personal experience, this is not the case While many proficient speakers do exist, the majority of Japanese citizens really struggle with the lan guage Historically, Japanese have performed quite poorly on En glish standardized tests For example, Japan ranked at the very bot tom of rankings for TOEFL listening scores in 2006 and last overall in Asia (Hagerman, 2009) More recently, a Japan Times editorial entitled Disappointing levels of English (2015, March 28) noted that a majority of 3 rd year high school students were not able to meet the government target of at least Pre-Grade 2 on the Eiken test: Students’ English proficiency was especially low on the more ac tive, productive skills of speaking and writing. On the exam, 29.2 percent of students scored zero on the writing section and 13.3 percent also scored zero on the speaking portion . That is even more disappointing considering that only 20 percent of students even took the speaking portion at all. It is doubtful that the other 80 percent of students would have performed any better. What’s to blame for such poor English skills? Cultural and psychological factors that result in passive learning at titudes and silent classrooms notwithstanding (Harumi, 2011), the combination of traditional teaching approaches and the high-pres sure entrance exam system has received the lion’s share of the blame for the low overall level of English ability in this country For de cades, students have learned the language primarily via the Gram mar-Translation Method , which Larsen-Freeman (1986: 4) notes was designed to enable students to read literature in the target language and was never intended to foster communication skills:

The Grammar-Translation Method is not new. It has had many different names, but it has been used by language teachers

11 Chapter 1: The conundrum of teaching English in Japan

for many years. At one time it was called the Classical Method since it was first used in the teaching of classical languages, Lat in and Greek. Earlier in this century, the method was used for the purpose of helping students read and appreciate foreign language literature. It was also hoped that, through the study of the grammar of the target language, students would be come more familiar with the grammar of their native language… Finally, it was thought that foreign language learning would help students grow intellectually; it was recognized that stu dents would probably never use the target language, but the mental exercise of learning it would be beneficial anyway . As a result of this approach, Japanese English classes actually contain precious little spoken English This is, after all, not the main goal Students spend their time listening to their teachers explain compli cated grammar rules and vocabulary in order to translate from one language to another successfully Textbooks are packed with color fully convoluted charts, diagrams, and explanations The learning process is largely deductive, meaning that students are given rules and examples, told to memorize them, and then are asked to apply these rules to other examples (Larsen-Freeman, 1986) Listening and speaking activities, when conducted, are usually done in support of the grammar point of the day Instruction on how to conduct nat ural, pragmatically correct conversations is largely absent, and as a result, most learners are not confident enough to place themselves in authentic cross-cultural communicative situations (Harumi, 2001) Compounding this situation are the arduous entrance exams that students must pass in order to enter the high school or univer sity of their choice The English portion reflects the ways in which the language is taught Typical tasks include filling in blanks on long reading passages or short dialogs, putting words of a sentence into grammatically correct order, listening comprehension questions, and writing out short opinion statements on various topics Here is

12 Part 1: Why should we bother with speaking tests? a sample, taken from the 2016 exam ( 教英出版、 2016: 5) 1 :

(1)

A: Yumi, can ( pen / your / I / use )? B: Of course. Here you are. A: Thank you.

(2)

A: You have a nice watch! B: ( to / gave / me / it / my father / ) for my birthday. A: That’s good! A: Look at this picture. This is the longest river in Japan. B: Do ( long / is / you / how / know / it )? A: Sorry, I don’t know.

(3)

Although this question type features communicative exchanges be tween two speakers, it’s actually good reading and grammar skills that are required to answer it correctly As a result, teachers have to prioritize them over conversation practice in their classes What do you imagine are the long-term effects of constantly pairing communicative exchanges with grammar accuracy ques tions on high stakes exams? What sorts of attitudes about En glish communication are cultivated via such a curriculum? A classic case of negative washback The current entrance exam system has been heavily criticized as the primary reason why Japanese students fare so poorly at English The key problem is the influence these exams have on how the language is taught and learned As Hagerman (2009: 57) writes: 1 Kyoei Syuppan is a publishing company based in Shizuoka Prefecture that makes past en trance exams available. If you would like to get more acquainted with these kinds of tests, you can find them at your local bookstore.

13 Chapter 1: The conundrum of teaching English in Japan

The importance of university entrance exams (both national and private universities) cannot be underestimated… Students feel great pressure to do well on the exams. As a result, teachers have tremendous pressure from parents to prepare their children from the exams. Thus, regardless of what strategic goals MEXT sets out in 5-year plans, the immediate needs of students to do well on the entrance exam takes precedence for teachers and schools… The Center Test, with its mandatory English compo nent, is the largest impediment to effective language education in the country. So here we come to our first key concept in understanding the importance and role that testing plays on the educational process As you can see, the entrance exams have a direct effect on how En glish is taught and learned In language testing parlance, this effect is known as washback 2 Here in Japan, the washback effect caused by the entrance exams is decidedly negative, as it forces teachers at all levels to devote large amounts of time to preparing for them, thus taking valuable time away from practicing oral communication In addition, since these tests profoundly impact students’ future social and economic advancement, they effect Japanese society on a macro level as well (Bachman & Palmer, 1996) Perhaps you have encountered some negative washback effects in your own teaching experience Personally, I see it in terms of fostering negative, self-destructive ideas about English, such as I can’t do it , I’m not interested , or I’m too shy (Talandis, Jr, 2015) Another example stems from how the teaching methodology and tests reinforce the idea that there is only one correct answer, and that students need to be grammatically perfect in order to make them 2 This term is also referred to in the literature as backwash (Hughes, 1989). Mendelson (1989) uses both: positive influence from a test is called washback , whereas negative in fluences are deemed backwash . In this book, I will only use washback , which can be either positive or negative.

14 Part 1: Why should we bother with speaking tests? selves understood This contributes, in turn, to a fear of making mistakes, a debilitating state which further inhibits self-expression In reality, of course, you do not need perfect form and complete cor rectness in order to communicate, as native speakers make mistakes all the time (Schegloff, Jefferson, & Sachs, 1977) However, this is an idea that many students and teachers are not yet able to fathom Instead of embracing the mistake-filled adventure of language learn ing, students often retreat behind a wall of silence (Harumi, 2001) For teachers, the negative washback also has severe consequences on morale and job satisfaction Many recognize the demerits of the sys tem, have a sincere desire to put more emphasis on communication, yet feel trapped and powerless to do anything about it What can you do? The system is the way it is, so I just need to focus on doing my best to survive within it… In the end, English in Japan is largely taught as a subject for the test, not as a skill to be used in the real world Is it any wonder then that after years of study many learners are unable to hold a basic conversation?

Negative washback kills the spirit of learning.

15 Chapter 1: The conundrum of teaching English in Japan

Fortunately, a ray of hope comes from the fact that washback effect does not only flow in one direction It is absolutely possible to flip the script and create a positive, virtuous cycle of learning In the following chapter, we’re going to look more deeply at how to do just that

Chapter 2

Embrace testing

Tests are not inherently evil The first step on the road to realizing greater positive washback in our teaching is to understand that tests are not evil They are, like most things, quite neutral entities While it’s understandable that we hold negative feelings towards them, remember that tests are neither here nor there They are, as Underhill (1987:6) writes, more usefully thought of as tools: Tests are not inherently good or bad, valid or invalid; they be come so when applied to a particular situation. You cannot say how good a hand tool is unless you know exactly what it is used for; similarly, you can only evaluate a test in a specific context. This is an excellent metaphor to keep in mind because it can help dissolve some of the negativity surrounding testing Exams are, af ter all, just basic tools to be used for completing specific tasks Is a hammer bad or good? Well, it depends on what you use it for If you would like to make a piano, then it can come in quite handy to drive

16

17

Chapter 2: Embrace testing

in the nails that keep the instrument together However, I would not recommend using a hammer to actually play the thing! In other words, it’s the needs and goals of the teaching situation that enable proper evaluation of a particular test Testing, like it or not, is part of what we do Nation (2013) notes that it is one of the four core jobs of a teacher, along with planning courses, training learners, and teaching classes Testing is important because it provides us with clear feedback on student progress, helps students monitor their own progress, and can help motivate the students to learn A well-designed test can also serve as a diagnostic tool that enables us to teach more effective ly Madsen (1983: 5) writes persuasively on this point: Our tests can help us answer the important question, Have I been effective in my teaching? In other words, we can use them to diagnose our own efforts as well as those of our students. As we record test scores, we might well ask ourselves the following questions: Are my lessons on the right level? Or am I aiming my instruction too high? Am I teaching some skills effectively but others less effectively? What areas do we need more work on? Which points need reviewing? Should I spend more (or less) time on this material with next year’s students? As you can see, without testing, we would not be able to function effectively as teachers Many of the decisions we make about every aspect of our job come from the feedback we get from assessing our students The better we accomplish this essential task, the better job we can do All tests are not the same The next step towards realizing positive washback in your teaching practice is understanding that not all tests are the same This may

18 Part 1: Why should we bother with speaking tests? seem like an obvious point, but it’s one worthy of review Typical differences revolve around method and purpose, about how they are designed and used (McNamara, 2000) One of the problems in the current ELT scene in Japan is not that testing is bad per se, but rath er the types of tests used to assess language ability Again, if we see tests as tools, then part of the solution is understanding more deeply about their characteristics and intended uses so that we can choose them more wisely This makes sense—you would expect a carpenter to know precisely what each tool does in her toolbox, so it follows that we teachers should be able to differentiate between different types of tests and know how to use them appropriately Norm-referenced & criterion-referenced tests: two different solar systems In most books on language testing, there is usually a section devoted to defining two main categories of tests: Norm-Referenced and Crite rion-Referenced Yes, yes, I know—I’ve promised not to get bogged down in terminology, so let’s have a bit of fun and look at this im portant distinction through a metaphorical lens Let’s say this lens is actually part of a massive telescopic array at an observatory, like the one atop Mt Mauna Kea in Hawaii:

Observing the Language Testing Galaxy

19

Chapter 2: Embrace testing

On a dark, clear night, you peer through the telescope and zoom in on the Language Testing Galaxy Within this vast spiral of stars you search for two solar systems, one called Norm-Referenced Testing and the other Criterion-Referenced Testing Astronomers refer to them as NRT and CRT, respectively Zooming in on each one, you notice a number of planets orbiting each respective star Let’s say each planet represents a different kind of language test What can you tell about the kinds of testing that go on within each solar system based on the configurations of its planets?

Mid-term

IELTS Eiken SAT

Unit 3 test

CRT

Entrance exams

NRT

TOEFL

Final exam

TOEIC

Part 1 review quiz

Testing planets in the NRT & CRT solar systems

It’s clear from this analogy that NRTs are basically standardized tests that have been created as tools to sort students into groups, while CRTs are the kinds of exams that teachers normally admin ister in their classrooms Brown (2005) identified several key dif ferences between NRTs and CRTs, including how scores are inter preted, measured, and distributed There are also differences in test structure and whether or not students know what sort of content to expect The thing to keep in mind, however, is the purpose and sorts of decisions each type was designed for Brown (1995:18) offers the

20 Part 1: Why should we bother with speaking tests? following explanation:

The basic purpose of CRTs is to foster learning. Typically, teach ers administer CRTs in order to encourage students to study, re view, or practice the material being covered in a course and/or give feedback on how well they have learned the material. In contrast, the underlying purpose of NRTs is usually to spread stu dents’ performances out along a continuum of scores so the stu dents can be classified or grouped for admissions or placement purposes. While creating such groupings may be beneficial to learning, the NRTs are not typically designed to test material that is specifically and directly related to a single course or program. Thus NRTs are not created to foster learning. In the final analysis, it’s clear that each test type has its appro priate use cases It all really depends on what the main purpose is—do we want to know how our students compare to other students, or do we want to know whether they have mastery (or not) over the material we are trying to teach them (Bailey, 1998)? Choose the right tool for the right job Even within the world of CRTs, it is possible to contribute to neg ative washback by using the wrong kind of test to assess a particu lar skill To be effective, a CRT must match what is being taught in class (Brown, 1995) For example, if we are interested in truly assessing spoken interaction, then it doesn’t make sense to use mul tiple-choice, paper-and-pencil tests that contain no actual trace of talking (Weir, 2005) Simply put, if you want to encourage oral ability, then you have to test for it directly (Hughes, 1989) This naturally holds true for the other major language skills as well Tests that seek to assess any of the four basic skills need to feature them in their design As Breaux (2016) wryly observes, swimming is im proved by swimming; speaking is improved by speaking

21

Chapter 2: Embrace testing

Key principles for evaluating tests: Reliability, validity, and practicality

How can you tell if a test is doing its job? Well, as I’ve mentioned, the field of language testing is packed with technical terminology However, if you boil it down, there are basically just three main principles that form the core of the field: reliability , validity , and practicality Most books on testing feature these concepts promi nently They are deep and multi-faceted, like gems Fortunately, the central idea behind each principle appeals to our common sense, so they are not difficult to understand Let’s examine each one in turn and then see how we can use all of them to create a rubric of sorts for evaluating any test (speaking or otherwise) we wish to give Reliability This principle is all about the fairness and consistency with which your tests are administered Nation (2013) poses a number of help ful questions you can ask yourself when reflecting on the reliability of your exams: Do your students understand the directions? This is a key point—if some students do but others don’t, then your test is not reliable because a portion of the class did not know what to do This is especially important for speaking tests—your students will no doubt be feeling nervous, so being clear on the directions will help them calm down a lot The easiest way to think about this point is to put yourself in their shoes How would you feel if you had to take a test in Japanese but couldn’t understand the instructions? Do they know what they’ll be tested on? This is a key aspect of any CRT—in order to do well, students need to know what material will be on the test ahead of time (Brown,

22 Part 1: Why should we bother with speaking tests? 2005) For a speaking test, are your students clear on what topics you want them to talk about and what sort of language you’d like them to use? If they do, you’re all set, but if some don’t, then this is an issue you’ll need to address Do your students understand your marking scheme? Another key point—do your students understand your marking scheme? This is a topic I’ll be taking up in Part 2 Fortunately, this is not a difficult thing to do if you know how to go about it Related to this is the issue of marking consistency When you mark your exams, do you do them all the same way? With written exams that contain only one right answer per question, this is not really an issue When assessing speaking skills, however, it is a very big one Part of the art of successful oral testing resides in your ability to mark each student the same way Again, we’ll address specific tips and tricks for doing this in Part 2 If you had a colleague administer your test, would the students still get about the same scores, all else being equal? This is an interesting one to consider—imagine you were out sick, so you asked your co-worker to conduct and mark the speaking test you had scheduled for the day If all test conditions remained the same, would the students get similar scores from both of you? Achieving this sort of marking consistency would require you to have a really clear and simple test design, so pondering this question is quite helpful as you create your test because it will push you to do a better job Will your students take tests seriously? Fortunately, this is not a huge issue in Japan because most students have been deeply conditioned to respect authority and the impor tance of the testing process Still, it can be a problem in certain teaching contexts Keep an eye on this one—if you are dealing with

23

Chapter 2: Embrace testing

some maturity and discipline problems, this may have an adverse im pact on test reliability Reflecting on these questions with regards to your tests will steer you in a very positive direction Notice how they all exemplify common themes of clarity , transparency , and consistency Internalize these simple, common sense ideas and live up to them as best you can The more you do, the more confident you’ll feel that you’re doing your very best to administer highly reliable exams Validity A valid test measures what it is supposed to measure (Nation, 2013) This is pretty much the central concept in all of language testing (Fulcher & Davidson, 2007) There are entire books focused on just this one topic This is because when you break it down, there are, in fact, actually many kinds of validity: you have face-validity, con struct validity, content validity, and criterion-related validity Then there is predictive validity, concurrent validity, and test validity Oh, and don’t forget about convergent validity and uh discrimi nant validity Zzzzzzzz OH! Sorry, I dozed off there for a moment :-) Thing is, there are so many different facets of this concept that it is all to easy to lose sight of its simple and profound elegance

Test

Discriminant

Convergent

Predictive

Content

Face

Construct

Criterion-related

Concurrent

Validity is a gem-like, multi-faceted concept

24 Part 1: Why should we bother with speaking tests?

As you look upon all that terminology, remember you don’t need to understand all facets of a gem in order to appreciate its value and beauty For now, let’s begin by focusing on the gist of what validity is in practical terms When I talk about the importance of using the right tool for the right job , what I’m really getting at is this idea that tests should measure the language skill(s) you want to assess In other words, a speaking test is valid only if it really and truly measures speaking ability A classic negative washback-inducing error is to test oral com munication via typical multiple choice paper-and-pencil tests This sort of test is not suitable because students do not need to ac tually speak in order to pass Instead, they use other abilities, such as reading and their knowledge of vocabulary and grammar Invalid tests are a big problem because they are a chief cause of negative washback Mendelsohn (1989: 100) chronicles the trap that many teachers unfamiliar with the concept of validity commonly fall into: The unfortunate chain of events leading to negative backwash can go as follows: the classroom teacher does not really know how to design a communicative test of what she/he is teaching. This results in a test that does not reflect the teaching because the teacher has produced a traditional grammar-based dis crete-item test. This will, in time, have a negative backwash effect on the teaching. Practicality It may be obvious, but our assessment activities need to fit within the time and energy we have for conducting them Some types of speaking tests are easier to administer and mark than others For example, you could interview all of your students one by one and

25

Chapter 2: Embrace testing

then have your colleague do the same in order to compare notes This methodology would probably produce incredibly reliable and valid results, but at what cost? Nation (2013) identifies the follow ing characteristics of a highly practical exam: • Easy to make and mark • Easy for students to interpret Since we are all so busy, these are very important considerations to keep in mind As Bailey (1998) reminds us, the main purpose for assessing language is to help us gain critical information about our students’ abilities and progress in ways that are conducive to learn ing However, if our tests take too much time to make and adminis ter, if they are difficult for the students to understand, or if they take you weeks to mark, then they are of little use It’s not necessary to work yourself to the bone in order to as sess your students’ speaking ability well When you design your tests, keep practicality in the front of your mind If it’s all just too hard, then you’re not as likely to do it well, let alone at all The test ideas presented in Part 3 were developed with practicality in mind, if for no other reason than you are more likely to try them! Find the right balance In the end, your main challenge when evaluating a test is to find the right balance between the “big three” concepts of reliability, validity, and practicality Taken together, they provide a clear and useful rubric for reflecting on the quality of your assessment activ ities One challenge you’ll face, no doubt, is the constant push and pull between them If you emphasize one principle, it will often • Not expensive to produce • Fits within available time

26 Part 1: Why should we bother with speaking tests? come at the expense of another For example, you could give your oral communication students a written multiple choice test because you’re really busy and don’t have much time for marking and feed back There is, after all, a strong tendency in teaching to test what is easiest to test rather than what is most important (Hughes, 1989) However, you would take a big hit to reliability and validity if you did Conversely, you could go the other way by trying to assess all 45 of your students in one period via individual interviews Obvi ously, that won’t work either—you’ll never have enough time to get through everyone adequately, and you will end up exhausted

Validity

Reliability

Practicality

In the end, there will be trade-offs you’ll have to make given the limits of your time and energy. No test is perfect, after all. Testing often becomes the art of doing the best you can. Nation (2013) suggests a good rule of thumb to follow along these lines: When push comes to shove, it is better to emphasize reliability and validity over practicality to the highest degree you are able Learning to conduct speaking tests will, at first, require more effort from you However, if you stick with it, it does get easier Try to ap proach it as an investment of energy that will pay off down the line Make these core principles a part of your testing practice, and you will gradually feel like you really know what you are doing You’ll be teaching with more confidence because you know your assessment efforts are in line with your class goals and objectives This, in the end, is one of the most powerful things you can do to promote pos itive washback and productive learning in your classes

Chapter 3 Work for positive washback

Recapping the problem ELT in Japan is characterized by a debilitating disconnect between stated linguistic goals and an educational reality that works against them This inherent contradiction fuels the poor state of English communicative competence amongst many learners It’s no surprise that years of study do not seem to help students speak English much because the system in which they were taught has not been properly designed for producing fluent speakers English is a subject for the test, not a skill to be actually used Many educators and adminis trators with sincere interest in actually realizing some of the ambi tious MEXT goals work as hard as they can within the limits of the system It’s tough going, however, as the negative juken eigo (test English) washback puts a constant drag on morale and productivity In such a test-heavy educational culture, students have become conditioned to put their limited time and energy into covering only material that will be tested

27

28 Part 1: Why should we bother with speaking tests?

In other words, what gets tested gets done (Breaux, 2016). While communicative goals are admirable, students and teachers only have so much time and energy to strive for them. It would be ideal if we could teach communicative English all the time and help our students learn for the sheer intrinsic joy of human interaction, but the reality of the current system forces many of us to toe the line in order to protect student access to higher educational and employment opportunities (and our own jobs as well). What are we to do? Flip the script Ironically, the road to transforming this mess runs right through it Washback, the effect that testing has on learning and teaching, cuts both ways As a result, it is imperative to make any and all efforts to promote positive washback in our own classrooms if we truly want to help turn things around Yes, we do often have strict limits imposed upon us, but fortunately there is usually at least a bit of room to change and grow within them In other words, there is always something we can do better If we’re honest with ourselves, we know the choice to do so is there every day Simply by putting a bit more effort into doing our basic duties more effectively (such as oral testing), we put ourselves on a sustainable path that will not only lessen the sting of the current situation, but actually help us thrive within it One viable way forward is to accept reality and use the system’s energy against itself In other words, if our students will only put out significant effort to study what’s on a test, then let’s make use of that situation With a few simple adjustments, we can reorient this energy in a much more productive direction Gates (1995: 106) expresses this idea via a metaphor quite appropriate for our Japanese context:

29

Chapter 3: Work for positive washback

It is better to accept the existence of washback and harness it, much the same way an expert in aikido exploits the force of an opponent … Teachers who are responsible for their own cours es and tests can achieve positive washback fairly easily. If they choose course and test objectives on the basis that they share the same orientation, there should be no tension between exam work and learning needs. Students will realize that class work gives both practice for the test and preparation for using English in authentic situations.

YOU

JUKEN EIGO SYSTEM

This is you after upgrading your oral assessment skills. Feels good, eh?

Harness your imagination… Again, it’s important to remember that learning to conduct oral tests successfully is a skill you’ll need to cultivate, putting time and effort into it on a consistent basis There are no free passes here However, before you dive in, let’s imagine what your work life may look like once you upgrade your oral testing regimen… After a period of trial and error, you are now confident in your ability to conduct reliable and valid speaking tests that fit seam lessly and comfortably within your teaching workflow Instead

30 Part 1: Why should we bother with speaking tests? of inducing dread, your exams now promote positive attitudes towards English via the sense of achievement they provide Stu dents react favorably to your lessons because they know your tests are fair, consistent, and help them improve Fear of mistakes decreases as everyone realizes what a non-issue that is— they are too busy enjoying new found abilities Some students are con firming their interest in the language, while others are witness ing a sea change in their attitudes toward it As a result, they are expressing themselves more and more all the time As you witness their confidence growing, you feel amazed and renewed And who knew you could learn so much from your students? As you get more familiar with finer points of their linguistic output, your ability to provide clear and cogent feedback increases You’re feeling more accomplished as a teacher And, because your stu dents are expressing themselves more, you find yourself getting to know them better as people Hey, you know what? That feels really nice You’re reminded of why you became a teacher in the first place, and you start looking forward to your job whereas before you may have been struggling with it Having tasted some success, you discover hidden pools of energy and motivation to continue developing your practice in other areas You find your self seeking out other teaching resources, attending conferences just because, and getting active in the local chapter of a teacher’s organization Overall, you take great pride in knowing you have done your best to help your students learn You’ve done your job You’re part of the solution now… …and be the change Teachers who are dedicated to improving all aspects of their craft are essential if we’re to turn this negative EFL system in Japan around

31

Chapter 3: Work for positive washback

After all, it’s not going to change by itself The cultural, economic, and political interests that benefit from the status quo are not going to suddenly disappear, so there is no use waiting for this change to happen from the top In fact, research has suggested that even if the entrance exam system were abolished tomorrow, things would likely still stay pretty much the same (Watanabe, 2008: 131): The research into washback to date implies that an attempt to in novate in education could not be successful simply by changing the examination system. If this type of top-down approach does not work, then a more appropriate approach would be the one that starts at the level of individual teachers. As MEXT continues to place new emphasis on communicative goals in language teaching, there will be an urgent need in our pro fession for classroom teachers to equip ourselves with some measure ment tools in order to evaluate students’ oral proficiency (Nagata, 1995) As we develop these skills, our knowledge and influence can grow along with them We can then become more effective sources of information and inspiration to our colleagues in the field Despite strong headwinds, the simple truth is that there is al ways more we can do within the confines of our own classrooms and teaching practice than we realize Conducting speaking tests may seem like a trivial thing, and on one level this is exactly correct! Nevertheless, that old saying from Gandhi still holds true: Almost anything you do will be insignificant, but it is very important that you do it. This truism becomes abundantly clear if you consider what would happen if we did not invest our time and energy into oral testing Hughes (1989: 47) frames the consequences in stark terms: Before we can decide that we cannot afford to test in a way that will promote beneficial backwash, we have to ask ourselves a question: what will be the cost of NOT achieving beneficial back -