ChatGPT simply doesn’t understand how to grade or create texts on the IELTS scale

I offered a week ago to adapt my 9-step prompt for studentsto use to get feedback on their speaking from, which uses the CEFR scale and  the Global Scale of English to work with the IELTS scale. In many countries the IELTS scale is used more often than the CEFR scale.

Here is a table showing how grades on the IELTS scale correspond to the other two scales:

It also shows the levels of the different versions ChatGPT produced from the original transcript of “Bargain Sweater”.

Unfortunately, after experimenting with seven different variations of my 9-step prompt with various differences between steps, I began to suspect that ChatGPT had no idea about how to rate texts or create them at different points on the IELTS scale. I wanted to test this.

Daniel Tena Calderon had experimented with using Gemini to do something verysimilar to what I’m trying to do with my 9-step prompt, so I decided to see how the two, ChatGPT and Gemini, compared using the same transcript and the same prompt.

I asked each of the LLMs to grade the five versions on the IELTS, CEFR and GSE scales and they both asserted that they had done what they had been asked to do: produce texts that go from the original to half a level up and then one level further and one level further again.

	ChatGPT	Check	Gemini	Check
Original	4.5-5.0	3.5-4.0	4.0-4.5	3.5-4.0
Corrected	5.0-5.5	3.5-4.0	5.0	4.0
½ IELTS level up	5.5-6.0	4.0-4.5	5.5-6.0	4.5
1 more IELTS level up	6.5	3.5-4.0	6.5-7.0	6.5-7.0
1 further IELTS level up	7.0	4.5	7.5-8.0	6.5-7.0
Differences		9.5 too high		3.5 too high

 

Gemini appeared to grade the original more severely, but appeared to reach higher levels later in the sequence.

I haven’t got a tool to independently check if the LLMs were right, but I have used an average of the CEFR grades given by Text Inspector and Pearson’s Text Analyzer as a reliable way to grade texts. I then used the table above to predict the level on the IELTS scale and I’ve added a Check column with the results for each step for each GenAI tool.

Gemini was much closer to the checked levels and ChatGPT was so far out that I think it is fair to say that ChatGPT simply doesn’t understand how to grade or create texts on the IELTS scale.

Before writing this, I made a screen recording making the same point and illustrating it with the texts created by the two LLMs. As usual, I used Clipchamp to make a couple of edits and to generate the subtitles file, which I then used to create subtitles on YouTube. I experimented with various formats of the video and subtitles and uploaded the four formats to YouTube:

·         Wide screen with subtitles to the right

·         Portrait with subtitles at the bottom

·         With red subtitles on top of the screen recording

·         With subtitles you can turn on or off

NotebookLM Video Overview: Is it worth asking it to use language at a lower level?

The reason I am interested in the answer to this question is because I like the idea of students being able to generate content based on transcripts of their spoken output (or of their writing.) But it should be accessible to them and their classmates.

I've examined at least 9 different podcast-producing tools and analysed the length and level of the podcasts and the speed of delivery. The conclusion of that study is that this technique is most approproate for B1 and above students as the podcasts are mostly at B2. Many of these tools allow students to slow down the speed of delivery.

My comparison of podcast-producing tools all used the same recording and its transcript: The Story of the Spanish Couple on the Titanic recorded by a good B1 student in 2014.

Here I discuss how useful NotebookLM's Video Overview could be for a B1 level learner.

This is the uncustomised Video Overview:

Text Inspector         C1 59%

Text Analyzer B2 61 GSE

Average B2+ 60

Length in words      1044

Length in time        5 mins 45 seconds

Speed in w.p.m      182 w.p.m

This means that the Video Overview would be a bit difficult for a B1 student

Here's the version where I used the customisation, "Please use simple English suitable for a student of level B1 on the CEFR scale"

As you can see, it is slightly shorter and slightly easier.

Text Inspector      B2+    55%    (C1 59%)

Text Analyzer B1+ 58 GSE    (B2 61 GSE)

Average B2 56.5          (B2+    60)

Length in words     759                    (1044)

Length in time        4 mins 10 seconds    (5 mins 45 seconds)

Speed in w.p.m      182 w.p.m    (182 w.p.m)

My conclusion is that it is probably worth customising the Video Overview in this way. Could I have asked for the delivery to be a bit slower? It's still hard for a B1 student at full speed, but using NotebookLM it can be played at three-quarter speed 137 words per minute

JALTCALL 2025 International Keynote Mat Schulze

I enjoyed this enormously, particularly the last section. I asked NotebookLM to create a Video overview:

Is it fair to reduce the 68 minutes to just 6 minutes? Does the six minutes really express what Mat's main points were?

I also asked for an audio overview, which produced a 12-minute summary in answer to this customisation, "What does the speaker say about the main difference between humans and artificial intelligence and what are the implications of this difference?"

Mind the Gap!

Transkriptor send me an email and it proved to be very disappointing

I got an email from Transkriptor inviting me to try it again saying that it was now "free."

So, for the first time in just over a year, I tried Transkriptor again. This is what I found with my 'trial account', where I had plenty of minutes' credit left:

• Easy to edit out errors in the transcript
• Easy to export the transcript without speakers and timestamps on the app
• Easy to paste transcript into ChatGPT and then add 9-step prompt at the beginning

After that, it was all down to the ChatGPT app and everything worked just as well as with Turboscribe.ai

Here's the 57 second recording and edited transcript on Transkriptor.

So I made a video of the recording playing on Transkriptor and then of the 9 steps on the ChatGPT app on my Android phone and posted it on my YouTube channel. I have since deleted the video as I subsequently discovered this:

If you want a 'free account' you need to give your credit card details, which I wouldn't recommend and haven't done, so I will never discover whether you get 30 minutes every day split among any number of recordings or just one transcription of up to 30 minutes. I suspect the latter.

If you have a 'trial account' with Transkriptor, there are limitations on the length (5 minutes) and percentage (80%) of your recording that will be transcribed, so I can't really recommend that. The workaround is to keep recording anything whatsoever for a quarter of the time after the time you have recorded! 

But don't bother - stick to Turboscribe.ai