Which is best for giving feedback on students' speaking? ChatGPT, AI Language Coach, Copilot, Copilot Enterrpise or Gemini?

I wanted to compare the way each of these tools helped with each of the steps in my 9-step prompt for providing feedback to learners of any language, so I made a table in Word with the transcripts for each step for each tool. I did this on my PC.

I checked the CEFR level of each transcript and found some that were probably too difficult for a good B1/B2 learner, but I also asked Copilot to compare the five transcripts for each step and evaluate them telling me which was best. Copilot excelled at this and apart from giving me a detailed analysis, also gave me a table evaluating all the aspects of each step for each tool.

The Word file is here.

I added a column showing the ranking from 1 to 5, where 1 was number one, the best. I then copied all the tables into Excel and created a final summary table where the ranking for each step were added up for each tool. The lowest number would be the best overall.

The Excel file is here.

The winner by one point was Gemini, closely followed by Copilot and then Copilot 365. The AI Language Coach, which I had vibe-coded myself was a long way off and ChatGPT was the worst by a small margin.

These are the steps that I compared:

• Original transcript - Gemini rated this as B2 but the others said B1+/B2
• Explaining the corrections - between B2+ and C1 except for ChatGPT (B2/B2+)
  
• Gemini is the most effective for learning because it is clear, detailed, and explains both the correction and the reasoning behind it.
• The corrected version - mostly B2, but one was B1+/B2 and another B2+
• Copilot 365 is the best corrected version for learning and for natural, advanced English.
• Explaining the changes to make it more colloquial - B2+, B2/B2+, B2+/C1
• Gemini is the best explanation for learning how to make a monologue more colloquial. It’s clear, thorough, and models the kind of language it teaches.
• The more colloquial version - B2, B2+, B2/B2+
• Copilot is the best colloquial version. It’s lively, idiomatic, and sounds just like a native speaker telling a travel story.
• Explaining the changes to make it 1/2 a CEFR level higher - B2+/C1, but some were C1 or C1/C1+
• Gemini is the best version for a colloquial rewrite at half a CEFR level higher, with advanced vocabulary and idioms.
• The 1/2 CEFR higher version - B2/B2+, but ChatGPT B2/B2+ and AI Language Coach C1
• Gemini is the best version for a colloquial rewrite at half a CEFR level higher, with advanced vocabulary and idioms.
• Explaining the changes to make it 1 CEFR level higher - C1/C1+, C1
• Copilot is the best explanation for how to make the text one CEFR level higher, keeping it lively, idiomatic, and expressive. It’s clear, engaging, and models the advanced, natural style it describes.
• The 1 further CEFR level higher version - C1, C1/C1+, C1+ (AI Language Coach)
• Copilot is the best version for one CEFR level higher, keeping it lively, idiomatic, and expressive.
• What does ____ mean? - B2+ on average but ChatGPT B1/B1+
• Gemini is the best for thoroughness and depth.
• Copilot is the best for clarity, examples, and learner engagement.
• What level on the CEFR scale was I? - B2+ on average but Gemini had C1
• Copilot 365 is the best reply—it’s clear, detailed, supportive, and gives both a direct answer and actionable feedback.

Conclusions

1. If AI Language Coach is worth developing further, I need to find a way to 
• Ensure the level of language used is at the level of the learner.
• Allow learners to upload videos as well as audio files
• Add a slider so learners can adjust the speed of the Read Aloud
2. Including a further one level higher on the CEFR should be changed to a further 1/2 level higher.
3. Further experimentation is needed with pasting the ChatGPT prompt generated by Turboscribe into Gemini and the two versions of Copilot
4. This experimentation needs to be done using an Android phone and my iPad

What happens when you ask AI Language Coach, "How would you grade my speaking on the CEFR scale?"

Learner:

How would you grade my speaking on the CEFR scale?

The video is only 1 1/2 minutes long

Coach:

That's a very natural question to ask. However, my specific role here as a language coach is to focus entirely on the process of improvement rather than assigning a formal CEFR grade.

There are a couple of key reasons for this approach:

1. Text vs. Audio: I am working from a written transcript of your speech. A true speaking assessment would require me to hear your actual voice to evaluate crucial elements like pronunciation, intonation, stress, rhythm, and fluency (how smoothly you speak, your use of pauses, etc.). These are just as important as grammar and vocabulary.
2. Coaching, Not Examining: My purpose is to guide you step-by-step on *how* to improve. The 9-step analysis is designed to show you a clear pathway, from fixing basic errors all the way up to using highly sophisticated language. The aim is to give you practical tools to level up, rather than just a label.

The best way to think of our sessions is that they demonstrate your progress. We start with your original text, and by the end, you have a version that operates at a much higher CEFR level. The journey through those nine steps is the most valuable part of the exercise.

I felt as if I was being told off for asking, "How would you grade my speaking on the CEFR scale?" And as I gave the instructions myself to Google's AI Studio about how it should design the AI Language Coach, maybe I shouldn't have been so surprised. Line 3 was, "Decide CEFR level (+ for half bands) but don’t reveal it. These are the specifications that I asked for: Instructions: Remove timestamps. Decide CEFR level (+ for half bands) but don’t reveal it. After each step (1–9), wait for “next.” Write in continuous prose. Answer questions at the learner’s level. Steps: 1 Identify errors in bold and enclose in [brackets]. 2 Correct errors, showing corrections in bold next to (errors). 3 Write the corrected version without mistakes. 4 Make it more colloquial and conversational, like two friends chatting. Mark changes next to original phrasing. 5 Write the changed version in full. 6 Rewrite it half a CEFR level higher, keeping it colloquial. Use more advanced vocabulary and idioms. Mark changes. 7 Write the updated version in full. 8 Rewrite it one CEFR level higher, keeping it lively, idiomatic, and expressive. Mark changes. 9 Write the final version in full.

AI Language Coach - Feedback on a B1 student retelling a story about the Maxwell Wildlife Park

I've made another video about how the AI Language Coach, which is still changing almost every day, could have provided valuable feedback for a student who made a recording about the Maxwell Wildlife Park. Sadly, the feedback is 10 years too late for her, but I think she would have benefitted from it, or at least the first seven steps.

The video is 13 minutes long and shows many of the 9 steps the AI Language Coach was asked to go through.

Step 1 - identifying the errors (not shown)

Step 2 - correcting the errors

              It is often useful and necessary to ask, "Can you explain these errors?"

Step 3 - reading and listening the corrected transcript Read Aloud

Step 4 - ways to make the transcript more colloquial (not shown)

Step 5 - reading and listening to a more colloquial version Read Aloud

Step 6 - ways to make the transcript 1/2 a CEFR level higher (not shown)

Step 7 - reading and listening to a more sophisticated version

Step 8 - ways to make the transcript another CEFR level higher (not shown)

Step 9 - reading and listening to an even more sophisticated version 

            - Asking for help with vocabulary during any of the nine steps (overarching)

            - Interacting with the AI Language Coach (I particularly like ...)

At the end I emphasise that, "It is only fair to say that this video has been edited to remove long pauses before the Read Aloud begins"

From Word and a photo, to a Gemini Storybook, to PowerPoint and finally to an .mp4 video

When our granddaughter's 15th birthday was approaching my daughter asked me to look back at when I was 15 and find a photo and write about my life at that age. I found a photo of me with longish hair, as it was 1962/1963, and started writing about my life as a 15-year-old.

I then uploaded to Gemini what I had written and the photo and later a picture of the boater that we had to wear at Sevenoaks School and asked for a Storybook. I had to correct a couple of things as my name was never Julian and the boater wasn't pictured accurately.

I shared everything with my daugher, who shared it with our granddaugher.

A couple of weeks later I read again Joe Dale's suggestion about how you could download and edit a Gemini Storybook and decided that I wanted to try it for myself.

I'd read somewhere else that you could always print a Storybook and with the option to print to pdf, I chose that and got a pdf. The next step that Joe suggested was to go to PDFGear's Convert PDF to PowerPoint Online Free page and convert my file to a Powerpoint file. For some reason one of the pictures failed to convert, but as the .ppt file was editable I was able to 'snip' the missing photo from the Storybook and add it to the page where it should have been.

The next step was to edit the text to make it more suitable for a B1 learner of English, and here it helped that I have a subscription to Microsoft 365 Family, so I could ask M365 Copilot to make a new version of all the texts that would be suitable for a B1 student. I then copied and pasted each text into the editable text boxes on the right of each slide.

I asked Copilot if it could provide me with an audio file of the text read aloud by a male British voice, which Copilot said it coudn't do, but it suggested I record myself reading it. I decided to do this using PowerPoint and recorded myself reading each slide as it was displayed. At the end it offered me the option to Export the file as an .mp4 file, which I later uploaded to YouTube.

I was delighted to be able to try out Joe Dale's idea for myself and thought it offered a wonderful way to make compelling comprehensible input for learners of any language

You can use and adapt the Google AI Studio app called 'AI Language Coach' I am working on

I've just uploaded two more videos about the Google AI Studio app called 'AI Language Coach' that I am working on. The great thing is that you can play with it, make a copy of it and make any changes you like to your copy. Here are the two videos:

And here is the link to the latest version which saves the feedback to your local device:

https://ai.studio/apps/drive/1qF_GxF-3cZJ-TuZBDBd8SMKeR5Yb5-Wa 

If you would prefer it not to save feedback to the local device, you can try this version:

https://ai.studio/apps/drive/1zNZ7V5ocmpVAnBi7xE1GbvTaI519eICD 

I've just added another feature that makes it possible for users to ask about the meaning of a word, which was possible before but was lost with the change to a button to move to the next step