|
Article Summary: We don't think EFL is a good way to learn spoken English. (Read our link Why ESL doesn't work.) Therefore, we also think there is limited benefit in the grammar sections of an EFL exam (TOEFL). TOEFL recently added a new section called TSE (Test of Spoken English) which only uses 20 minutes of the four hour exam. Obviously, TOEFL is still primarily concerned with English grammar. Therefore, we are suggesting a new exam which is devoted entirely to spoken English. We call it TOSE2-Test Of Spoken English number 2. Please remember, this is only a suggestion. TOSE2 does not yet exist. This page is merely a suggestion of how a comprehensive spoken English language exam could be developed. |
TOEFL is the acronym for Testing Of English as a Foreign Language.
TOEFL is essentially a multiple choice exam which requires some listening to English but almost no spoken English. As of September 2005, a new section called TSE (Test of Spoken English) was added. The new TSE section only uses 20 minutes of of the four hour exam time.
Certainly, international students coming to the U.S. to study need to know how to read English, write a final exam, and do study papers. But more central to their overall success will be their ability to understand spoken English and respond verbally. (We think students will learn both written English and English grammar faster if they concentrate on spoken English first. Many students study grammar for two years and still cannot speak English. Yet, by using our course, students can learn to speak as well in four or five months as do many students who have studied English for two years. English grammar should be taught only after students have attained substantial English fluency. See our link Grammar and writing in spoken language study.)
Thus, in light of both the need to emphasize greater English fluency and the practicality of using English fluency as the basis for speedy acquisition of writing and grammar proficiency, we are proposing a comprehensive verbal English exam.
Aside from the requirements of the university and college communities, there is also a great need for a standardized spoken English exam in the business, import/export, and service industries which require English fluency of their employees.
Our design criteria for TOSE2 is:
Our design criteria may sound daunting. Yet, it could be done relatively simply using even older computer and audio technology. More to the point, grading the exam could also be done objectively, and in a way which would give a detailed follow-up report to the examinee.
Students typically buy TOEFL study guides to prepare for their exam. TOSE2 would also have a study guide. However, it would be the exam CD itself. Because the written TOEFL exam is dependent on precise published answers, the test answers are carefully guarded. For a spoken English test, however, the grade would be based on the examinee's ability to communicate verbally. Therefore, there would be no need to protect the exam material because it would consist of only the "question" part of the exam, never the answer.
With its simplicity and the lack of need for protection, the entire TOSE2 exam could be put on a single CD which the student could also use for study and practice prior to the exam.
The TOSE2 exam format
Since a picture is worth a thousand words, let's watch an examinee—her name is Monique—answer a question on the TOSE2 exam.
The examinee wears an earphone/microphone head set. It is connected only to the computer if the computer is used to record the audio. In our example, however, Monique is listening to the audio from the computer but she is recording her responses on a cassette tape recorder. Therefore, the headphones are connected to the computer and the microphone is connected to the cassette recorder.
(Note: The thumbnail display remains on the computer monitor while Monique evaluates the four pictures she initially selected. Double clicking on any picture on the thumbnail display will replace the picture on that column in her working row of four pictures. By depressing the control key when clicking on the thumbnail, the picture is placed in memory and can be pasted in any box on the working line by using the paste function (CONTROL V). When she is satisfied with her selection of four pictures, she will click ENTER to remove the thumbnail display from her computer monitor and activate the four pictures she selected. Each picture has a unique identification number which always appears under it.)
 |
Grading the TOSE2 exam
We told you that the section Monique just finished was a later portion of the exam. It had four pictures which were to be incorporated into a single response. However, before we see how Monique will be graded for her response to pictures #724, #1498, #627, and #1389, we should look at an early part of the exam using a single picture because it will be easier to explain.
For her response in Section I, question #7, Monique selected picture #1026. A similar thumbnail display with 24 pictures appeared on her computer monitor, but she only selected one picture. Mr. Smith, the grader, is using the floppy disc which was made during Monique's exam. It recorded the sequence of all her responses. Mr. Smith just completed grading her response for Section I, question #6, and is ready to grade her next response. He clicks NEXT and this active work sheet appears on his monitor.
| Monique Siqueira; 03/24/06; 8:13:06-8:17:57 Section: I #: 7 | ||
 #1026 | ||
| Narrator: | "Please tell me what you did while you were here." | |
| Other vocabulary: | mountain, river, tree, sky, blue, green, snow, beautiful | |
| Verb vocabulary: | to fish, to climb, to swim | |
| Bonus words = +2 | foreground, background, clear (sky or weather), gorgeous, to rise above, to fly fish | |
| Response answers statement = +4 Does not answer = 0, Answers = 4 | ||
| Each verb = +1 | ||
| Incorrect person/tense = -1 | ||
| Incorrect meaning = -1 | ||
| All correct = +5 | ||
| Inappropriate = -1 | ||
| Pronunciation: | Unsatisfactory = 0, Satisfactory =2, Excellent =4 | |
| Comments: | ||
| Total score | ||
In order to grade this response, Mr. Smith listened to Monique's description of the picture. Monique's recorded statement said,
"We went to the river so my friend could go fly fishing, but it was too cold to swim. It was a clear day with a blue sky. The mountain in the background still has snow on it. It was a gorgeous day. The trees were green and beautiful. We left the river in the afternoon."
To better illustrate what the grader is looking for, we added bracketed notations at the end of each sentence.
"We went to the river so my friend could go fly fishing, but it was too cold to swim [5 verbs=5; 1 bonus=2]. It was a clear day with a blue sky [1 verb=1; 1 bonus=2]. The mountain in the background still has snow on it [1 verb=1; incorrect tense=-1; 1 bonus=2]. It was a gorgeous day [1 verb=1; 1 bonus=2]. The trees were green and beautiful [1 verb=1]. We left the river in the afternoon [1 verb=1]."
First, Mr. Smith listened to the response in its entirety. He assumed for himself the understanding level of a native English-speaker with no language education training. He determined whether or not Monique's reply would be fully understandable to this hypothetical, untrained English-speaker without repeating any words or sentences. He gave her a 4 for pronunciation because her response was fully understandable the first time it was played.
Mr. Smith finished the remainder of the evaluation from both memory and by replaying Monique's response when necessary. Monique used four bonus words, earning 8 points, and nine verbs, earning 9 points. He entered an "=" to mark each bonus word she used. (If a bonus word is coupled with a verb, it counts as both a verb and as a bonus word.) Mr. Smith gave Monique a -1 and wrote a note because her description required a single tense for the entire account. Because of the -1, she did not get the +5 for all correct. The all correct bonus requires perfect agreement of number, person, and word order.
After he finished, Mr. Smith's work sheet looked like this:
| Monique Siqueira; 03/24/06; 8:13:06-8:17:57 Section: I #: 7 | ||
#1026 | ||
| Narrator: | "Please tell me what you did while you were here." | |
| Other vocabulary: | mountain, river, tree, sky, blue, green, snow, beautiful | |
| Verb vocabulary: | to fish, to climb, to swim | |
| Bonus words = +2 | foreground, =background, =clear (sky or weather), =gorgeous, to rise above, =to fly fish | 8 |
| Response answers statement = +4 Does not answer = 0, Answers = 4 | 4 | |
| Each verb = +1 | 10 | |
| Incorrect person/tense = -1 Everything was told in the past tense--you should say "mountain HAD snow..." | -1 | |
| Incorrect meaning = -1 | 0 | |
| All correct = +5 | 0 | |
| Inappropriate = -1 | 0 | |
| Pronunciation: | Unsatisfactory = 0, Satisfactory =2, Excellent =4 | 4 |
| Comments: | - - - - - - - - - - | |
| Total score: | 25 | |
The worksheet was generated for Mr. Smith from the program contained on the CD. When Monique clicked ENTER after selecting the picture she wanted to describe, the program wrote the grader's work sheet, selected the random Narrator phrase, and listed all of the vocabulary, bonus words, and grading criteria assigned to that picture. Of course, Monique saw none of this, but when Mr. Smith was ready to grade that part of the recording (which in this case was from Section I, picture #7) this information was available to him from the floppy disc Monique was running in her computer during the exam. All Mr. Smith needed to do was to listen to the recording, mark the bonus words Monique used, tally the verbs, and enter the totals in each right-hand column. He entered the note in the "incorrect person/tense = -1" row, and entered 0's and - - - - - in unused spaces to indicate that the category was not overlooked.
An explanation of the TOSE2 score sheet
Narrator. In order to test the examinee's ability to spontaneously speak English, all pictures in the CD's image file would be grouped so that each group could be represented by appropriately worded questions or statements. (The CD would contain approximately 2,000 pictures and 100 Narrator questions or statements.) When an examinee would select a picture for a single picture response, the Narrator's voice would randomly ask a question or make a statement appropriate to that picture. Multiple-picture exam questions would not use the Narrator, allowing the examinee to match his or her vocabulary with the pictures. Examples of these questions or statements would be:
"Please tell me what the people in this picture are doing."
"Please tell me what you think is going to happen."
"Why are these people talking?"
"Please tell me what you did while you were here."
Questions asking for simple responses such as, "Do you like this picture?" or "Is this person talking?" would not be used because they would stifle the examinee's response by suggesting simple "Yes" and "No" answers.
Other vocabulary and Verb vocabulary. These words would be given for study purposes only when the student would be using the study mode, which is explained later.
Bonus words. These words would represent specialized English vocabulary. For example, Monique said, "The mountain in the background still has snow on it." The use of the word "background" in the sentence adds description and clarity which requires greater fluency. Equally, in the example from Section IV, question #3, Monique could have said her friend did not have brown hair. Instead, she identified her friend as not being a brunette or a blonde. It was a creative way to use bonus words, and it displayed greater English fluency. However, there would also be a risk because inaccurate use of bonus words could result in lost points. Bonus words may only be viewed in the study mode.
Response answers statement. 4 points would indicate that the examinee understood the Narrator's question or statement and responded appropriately. 0 points would indicate that the examinee did not understand the question or statement. (Grades of 1, 2, or 3 points would not be given.)
Verb count. Because verbs are critical to the structure of an English sentence, each verb would be given one point. Compound verbs (could fish) would count as two verbs. Only verbs would be counted in order to discourage the long, rambling sentences which would result if all words were counted.
Incorrect person/tense. One point would be deducted for each incorrect verb person or tense.
Incorrect meaning. One point would be deducted for each occurrence of an incorrect meaning. Incorrect use of bonus words would be deducted here.
All correct. Five extra points would be given if there were no grammar or meaning mistakes in the entire response.
Inappropriate -1. We would take the point of view that the English used in this exam is that which would be appropriate in a professional job interview. There seems to be a trend among ESL students to incorporate a great deal of slang, vulgarity, and profanity in their attempt to speak English. Generally, this practice is distracting rather than helpful in the communication process. We are not passing judgment on its use; we are simply saying that the best English for the EFL student is English which is free of the potential hazards resulting from the use of slang and profanity in formal education and business environments. Therefore, we would mark a -1 if we heard "yup," "going like hell," and the like. This would not surprise the examinee because the expectations of the exam would be fully explained.
Pronunciation. The sentences would be graded as Unsatisfactory, Satisfactory and Excellent. If the grader could understand all the words in the full statement without paying more than normal attention during the first playing of the entire response, the pronunciation would be graded as Excellent (4). If the grader would have to listen carefully because of faulty pronunciation or inflection, but could still hear all of the words correctly in the first playing, it would be graded as Satisfactory (2). If the grader would be uncertain of specific words and would have to replay portions of the response to understand any of the words, the response would be graded as Unsatisfactory (0). The grader would have to maintain consistency throughout the entire grading process so that increasing familiarity with the examinee's pronunciation would not change grading. The grader would have to listen as though he or she were a normal English speaker without specialized language education training.
Monique's response in Section IV.
We have already seen Monique's response. In order to make the grading system clear, we are adding bracketed score values at the end of each sentence.
"My cordless phone rang [1 verb=1; 2 bonus=4]. I answered it quick [1 verb=1; incorrect grammar]. It was my friend Alexandra and she was in a hurry [2 verbs=2]. Alexandra is not a brunette like the girl in the picture [1 verb=1; 1 bonus=2]. She is not a blonde either because she has dark hair [2 verbs=2; 1 bonus=2]. She told me she has a bad problem on her computer and it was just fixed [4 verbs=4; incorrect tense=-1]. It is a desktop computer with a bad keyboard [1 verb=1; 2 bonus=4]. She had the same trouble with it before [1 verb=1]. She needs to write a paper on her computer and then go to the airport to meet a friend [4 verbs=4]."
This is how Mr. Smith scored her response.
| Monique Siqueira; 03/24/06; 11:27:43-11:34:07 Section: IV #: 3 | ||
| ||
| Narrator: | Ø | |
| Other vocabulary: | telephone, phone, receiver, LED display, key pad, pushbutton, woman, brown (hair), blouse, girl, yellow (hair), computer, airplane, airport, plane, monitor, | |
| Verb vocabulary: | to call, to ring, to answer, to wear (clothes), to fly, to board, | |
| Bonus words = +2 | =cord(less), =phone, touch keypad, =brunett(e), embroidered, =desktop, laptop, =keyboard, monitor, =blond(e), runway, tarmac, fuselage, jet engine, wing tip, | 12 |
| Response answers statement = +4 Does not answer = 0, Answers = 4 | Ø | |
| Each verb = +1 | 17 | |
| Incorrect person/tense = -1 she |
-1 | |
| Incorrect meaning = -1 | 0 | |
| All correct = +5 | 0 | |
| Inappropriate = -1 | 0 | 0 |
| Pronunciation: | Unsatisfactory = 0, Satisfactory =2, Excellent =4 | 4 |
| Comments: | - - - - - - - - - - | |
| Total score | 32 | |
Section differences.
Though the description of a four-picture sequence may be somewhat longer than that of a one-picture question, length would not be the purpose of the multiple pictures. Generally, it would be more difficult for the examinee to expand comments for multiple pictures because the pictures themselves are not as much the focus of attention as the sequence of events.
The one-picture questions would tend to produce embellishment on a single topic whereas the four-picture questions would force the examinee to describe a series of events or steps required for a completed description.
TOSE2 would be a timed exam with a fixed allotment of time for each section. It would serve the student's best interest to do a larger number of questions with shorter monologues than it would to attempt to include all possible bonus words in lenghty sentences. Using verbs would help the student. However, since the "all correct" bonus would be so heavily weighted, shorter sentences may produce better scores. (Notice that on the first score sheet, a single misuse of a verb tense changed Monique's score from 31 to 25.)
The final exam grade would be given for the total number of points earned in each section, irrespective of the number of questions completed.
We would suggest four forty-minute sessions for the exam. Section I and Section III would be identical, using a one-picture format. Section II and Section IV would also be identical, using a four-picture format. Alternating between picture format would offer some variety to the exam sessions.
In other applications where TOSE2 would be used for selection of employees and the like, it could be shortened to an exam of two hours in length by using only Section I and Section II .
How the TOSE2 test grader receives the information.
The grader would need two kinds of information. First, he or she would need to be able to listen to the recorded audio. Secondly, the audio recording would need to be coordinated with the picture numbers, the vocabulary which goes with each picture, student identification, and test data. (We call all of this the control information. The control information for each picture would include the student's name, the date of the exam, the length of time spent on that picture(s), and the exam section and picture number(s). There would be, of course, a summary page (not showm) which would include additional information and grades. After the grading would be complete, the student would receive a copy of each picture score.)
How the information would be recorded during the exam would depend largely on the technology available in the country administering it. The lowest technology level would use a cassette tape recorder for the audio information and a floppy disc in the computer for the control information. The TOSE2 exam CD would have a program which records the entire exam sequence and transfer the information to a floppy disc as the exam progressed. The control information would include each picture number and the time taken for each part of the exam. When the exam would be graded, the control information would be displayed on the grader's computer monitor (or printed) in the way the information is shown in our examples.
With appropriate computer technology, the same information could be recorded on a CD. With the necessary hardware and software—including MP3 burning capability—the CD could record both the control information and the audio. The advantage of using a CD for the entire exam would be its value as a backup. The university could mail one copy to the TOSE2 grading center in the United States and keep a backup copy in the event of mail loss. The backup copy would also be useful to the student in evaluating his or her responses after the grade was given. Equally, by recording the exam on a CD, the entire session could be transmitted to the grading center by way of the internet.
(Note 1: Upgrade to the floppy discs—or CD—would be done in real time rather than at the end of the exam. Unreliable electrical power in some parts of the world could disrupt computer function. The CD control information program would have to be written so that, in the event of a power failure, a minimum amount of information would be lost. Note 2: Circulation of a backup CD among future examinees would present no security risk. See the earlier comments regarding CD security.)
STUDY and EXAM modes.
The TOSE2 CD home page would allow a selection between one of two modes. In the study mode, the student could select either the single- or a multiple-picture mode. Each time the NEXT button was clicked, the thumbnail display would open and allow the student to paste the selected picture (or pictures) on the working line. In the study mode, the student would be able to double click on any picture on the working line and the grader's work sheet would pop up. This would give the student the "Other vocabulary," "Verb vocabulary," and "Bonus words" for study. It would also prepare the student for the kind of "Narrator" questions or statements which would be used during the exam. The Narrator questions or statements for that picture category would be played by clicking through the entire list.
The student could practice using the CD in the study mode. It would respond to the student exactly as the exam, except that the student could add the pop-up grader's work sheet. However, the control information function would not run and the computer would record nothing to a floppy disc. (Nonetheless, a student could use a live cassette recorder as a part of the practice session.)
However, in the exam mode, the pop-up work sheets would not be available and the control information function would run if a floppy disc were inserted into the computer. To thwart any possibility of cheating by using pop-ups during the exam, the computer would void the control information on the floppy disc if the examinee exited the exam mode in order to enter the study mode. (In order to prevent such an occurrence, a message would appear on the monitor saying, "WARNING: THIS ACTION WILL TERMINATE THE EXAM.")
TOSE2 applications.
TOSE2 has a large market potential which is entirely outside the reach of TOEFL and similar written examinations. In today's economic world, there are many fields in which spoken English fluency is mandatory. International business, tourist industries, call answering services, airport and aircraft hospitality, manufacturer representation, international sales, technical repair and service, computer and software support, advertising media, and banking—to name just a few—must often hire competent English-speaking employees. Yet, there is little standardized English testing available today aside from TOEFL. TOEFL would be entirely inappropriate for this level of testing.
The Educational Testing Service (ETS) distributes Test of English for International Communication (TOEIC). Nonetheless, it is a written exam rather than one which measures the examinee's ability to spontaneously communicate using spoken English. Their descriptive material says,
The TOEIC test is a benchmark of English language skills in listening and reading. The TOEIC test is a paper-and-pencil, multiple-choice assessment.
Educational Testing Service then poses this enlightening question and answer.
Why is testing only listening and reading skills better than testing speaking and writing?
Listening and reading skills can be tested objectively, cost-effectively, and efficiently. Generally, direct tests of speaking and writing are less objective and reliable—and more time consuming and expensive.
The purpose of TOSE2 is an objective and cost-effective exam which measures the examinee's ability to communicate in spoken English. Employers of those entering the large international work force need to evaluate prospective employees' ability to effectively speak English. A paper-and-pencil, multiple-choice exam tells the employer nothing about the applicant's ability to use the telephone.
A standardized TOSE2 exam for spoken English testing has huge market potential. The mere fact that a large English testing organization was unable to produce such an examination does not mean it cannot be done.
This proposed TOSE2 exam would not favor Spoken English Learned Quickly students. It would be independent of any specific course of instruction and would accommodate examinees irrespective of their means of learning English. Its sole purpose is to test spoken English fluency.
Nonetheless, the best preparation for the exam would be spoken English language courses such as Spoken English Learned Quickly, found at www.FreeEnglishNow.com.
Why ESL doesn't work (very well).
A technical comparison of Spoken English Learned Quickly and ESL.
Grammar and writing in spoken language study.
Can beginning and advanced students use the same lessons?
What is ASE (Accelerated Spoken English)?
Socialization versus language instruction.
Copyright notice: ©July 21, 2006, Spoken Language International. Spoken Language International (Lynn Lundquist) holds all rights to this material.