The AI Learning Paradox: Better Results, Less Learning?

0:11Dan Welcome to the AI and Education Podcast,

0:14Ray I am

0:15Dan today?

0:15Ray great.

0:16 Dan, did I sound like Tony the tiger there? He's great.

0:21Dan Great.

0:21Ray good.

0:22 Thank you very much. I've had a holiday, Dan, as you know, because you had that fascinating episode with Chris while I was away. And I was tuning in my jaw nearly hit the floor a couple of times because of the big nature of some of the discussions. It was fantastic.

0:40Dan Where did you go Ray?

0:42Ray I went to Koh Samui in Thailand.

0:45 I first went there 30 years ago and things have changed.

0:50Dan When you were five.

0:52Ray Bless you, Dan.

0:53 I will send you the $20. , What have you been up to while I've been away? Dan?

0:58Dan just been really busy.

0:59 you know, I know everybody's busy and I hate the word busy, but, yeah, it's just been, coming up to the end of, Q1. So that's the start of a financial year in business, , areas in Australia and around the world, I suppose. So kinda to end of that, really trying to drive some, , AI adoption work in and outta schools left, right and centre.

1:16 So, yeah.

1:16Ray Yeah.

1:17 , And your focus through your job is around getting people using like the Microsoft AI stuff like, , Copilot , and Foundry. I'm get, am I getting the words right, Dan?

1:30Dan Yeah, right.

1:31 Yeah. Yeah. So I'm looking for the agenda kind of side to it all. And I know, , agents we've been talking about for quite some time, but thinking about the next level of governance and AI solutions across all kinds of areas of brain education, thinking about the that you can create, which are going beyond simple, GPTs and gems and all those kind of things, and copilot agents.

1:56 So thinking about, those experiences that some of the people we've had on the podcast have developed, the more agents that might be connected into other things like ServiceNow and stuff like that, or even creating their own experiences and their own, , AI experiences for students or for staff or for

2:14Ray You must spend half of your time in the week just trying to keep

2:17 up to date with what's changing.

2:21Dan Yeah, you do.

2:22 You do. I've got a webinar coming up to go through it actually. I'll put the link in the show notes when we get through there. But Ray, you've actually, I noticed on socials the other day you were sharing your, , AI for humans course.

2:34Ray so the AI for, for Human Course.

2:36 So I developed that last year. , And it was something that was, , offered commercially. So it ran in a LMS online. We, , did some work with, uh, organisations, so it sold commercially to organisations. And I got to the point of thinking, God, this is a huge barrier. So, so thinking about, for example, podcast listeners and probably people that are friends of podcast listeners, to be honest, , how do I make it attractive to them?

3:04 Because, you know, the course was, over $400 for an online course. Well, nobody in education should pay $400 for an, , for an online course, and it was a four week AI course. So I was kind of going through the how do I solve this problem? And so I decided that rather than fiddle around at the margins of it, I would just make it free.

3:26 And so I've taken the whole course, I've put it onto a self-standing website that doesn't need a big complicated LMS at the back of it. The website is aiforhumans.education. The course is AI for Humans. And, and it's a free course, and it's about building AI fluency. So it's not about turning you into an AI expert, it's turning you into somebody that is confident about using ai.

3:51 , And the reason that I did it is every organisation I work with, the AI conversation is absolutely racing ahead, but it kind of feels like for some people. It's almost an exclusionary thing, which is you're either inside the club and you are talking with other people and you're constantly pushing the boundaries of what AI can do, or you're outside the club, in which case you are miles behind because you maybe don't even know where to start.

4:19 And so the course is about solving that problem. It's not a technical course, it's not obsessed with products and prompting formulas, and coding and agents, like you talk about all the time, Dan. It's actually designed to take somebody from "well I don't really know where to start", to understanding why to use ai, how to work with it when it makes sense, when it doesn't make sense.

4:44 It's about using AI with quality, as opposed to using AI just in order to show somebody else that you're using ai. It's not about building AI specialists, it's about AI feeling normal for people.

4:58Dan Yeah, that's brilliant that you're doing that for free, right?

5:01 Because I, I remember looking at the content for that, a couple of months ago when you shared that across. And, um, it's great,

5:06Ray and I, I'll point out

5:07 that the, the word free, , it's free as in beer, not free as in puppy. Like a free puppy. When somebody gives you a free puppy, it's like, "Oh, and then if you bought the kennel and then if you bought this, and then if you bought that?" It's not that kind of free. Yes. I'm not giving the course away for free and then offering to sell you a prompting guide or a newsletter or whatever it might be.

5:28 It is genuinely, the full course that people were paying hundreds of dollars for last year, absolutely free with all of the assessments, all of the exercises, everything is in there. I, I know Dan, that a lot of people love this podcast 'cause we don't pile it up with adverts and I feel like, hmm, I'm doing an advert, but is it okay to do an advert for something that's free?

5:51Dan Yeah, of course it is.

5:52 That's absolutely, yeah, absolutely. So we've got lots of news today. with, some of the news from Google and from ChatGPT about

6:02Ray Yeah, this is a, this is a fast moving thing.

6:05 When I first saw it, it was that, , ChatGPT Custom GPTs. So that's where you create a persona that's got a particular job to do and you can just click on it and start a conversation with it. But the great thing was for a teacher's point of view is that they were really easily shareable. You could share a link with a student, a colleague or whatever, and it's like, "oh, I've built the best one for pretending to be William Shakespeare.

6:32 So go and click on this and you can have a chat with William Shakespeare". That was in ChatGPT. in the Google Land they called them gems in Gemini. And within a few weeks of each other, both OpenAI and Google have decided to get rid of them. And now you can't create new ones. In the case of OpenAI, they will all disappear and stop working in December.

6:57 And what is, I think even worse, is there isn't an alternative that is as easy to create. So if you're a teacher, there isn't an equivalent of a custom GPT that you can create and share with your students or with other colleagues. There are some circumstances, in which case you can create some things that are shareable, but they're not as easy.

7:22 I have seen a lot of frustration for teachers around this , because they've invested in creating these. I, I know some of the organisations I work with, they've created, you know, a 'Feedback' Custom GPT that students can use to get feedback on their own writing, and there doesn't seem to be a way to be able to make that possible in the alternative.

7:46Dan Why do you think they've taken these out?

7:48 Then Ray, what's the, what's the story

7:50Ray I don't actually know.

7:51 For the life of me I can't work out while they've gone. The only thing even vaguely is my assumption is that they weren't used that much. But they were used enormously in education because it was a really simple way to be able to build something and share it with your colleagues. So I've hesitated to over communicate about this because telling people to worry about something disappearing and not being able to tell 'em how to solve the problem is a little bit tricky.

8:22 So ChatGPT the alternative is plugins. And so I have moved some of my Custom GPTs to plugins. They're not sharable, they don't work in the same way. I haven't yet found a suitable alternative answer. So maybe what we should do is let's, let's get a few teachers that have used Custom GPT or GEMs, um, and talk to them and find out what's going on.

8:46 Let's get a couple of experts onto a, a future news and research episode. Maybe in two weeks time, Dan, we can get a couple of people on. We should get Danny, Danny Liu from, um, from the Land of University, Sydney and Cogniti Land, because that's one of the big things that Kognito does. And get some advice from him.

9:04 But, you know, ultimately, does it mean that as a teacher to create these kind of shareable resources, you can't use a general AI tool. You've gotta buy a, an education version of something.

9:19Dan I, I suppose I've always been worried about this anyway because, you know, and I

9:23 know I'm biassed from an enterprise point of view, and I've always said it a lot of times, but you know, these companies like, OpenAI and not so much like Google to be fair, but because you could with Workspace, you do get enterprise, management and things like that. But the Anthropics of the world and the um, ChatGPT OpenAI's of the world, you know, they don't have an enterprise infrastructure.

9:45 So you, it's very hard to manage and monitor and disseminate and govern agent ai. And, and we saw that come in a year ago and I was always puzzling me, you know, when teachers were sharing of these GBTs you, you start to get a bit of a, a wrongness in there where if there's a fraud or there's a, uh, self-harm or a child issue, very, very hard to monitor.

10:11 So I'm sort of glad that some of these things are, are less easy to do now. I'm sure there'll be something that'll take their place. 'cause the Gentech AI is where the future, is focusing. But yeah,

10:22Ray I, I'm not sure that the alternatives are any better at managing those

10:25 things, like in Chat GPT, instead of a custom GPT create a plugin. Well, that's got no more of those features to enable you to manage things than, Custom GPTs had. So

10:37Dan that's

10:38Ray yeah, I, yeah, I, I don't know.

10:41 But Dan, that has not stopped the headlong rush into ai. I mean, like, typical, I went away for two weeks of, and there was a flurry of announcements from universities. So UNSW, , they've said they're going all in with Chat GPT. So what they've said is they, you remember we talked about it on the podcast a year ago.

11:02 They said they would give, , chat GPT edu to all of their staff. , Now they've said they'll give it to all of the casual staff, all of their, , HDR researchers. So that's research students. And then from the beginning of next academic year, they will give every student access to chat GPT, that's like 80,000 students, maybe 20,000 staff.

11:24 So they are, according to the QS rankings, Australia's number one university, they are certainly amongst the biggest of universities. And so they tend to be a bit more conservative with decision making. So that's, I think that's big news. , And that was big news in week one. And then, along comes Flinder's and Flinder's University, on the next day, announced that they were going big with Google, so they are gonna have all of their 30,000 students with access to Google, Gemini.

11:54 And they've also announced a, an AI accelerator, which means it's about skills as well. So they are adopting the Google Career certificates. So students have got online self-paced courses to go alongside this. So it's not just about giving them the tools, it's about giving them the skills. And then the next day, university of Sydney, , their announced, got leaked to their student newsletter was great.

12:17 Their student newspaper leaked the story that they were just about to announce that they were providing chat GPTG for all staff and students. , But they also said at the same time they were gonna invest in an AI sustainability research centre. So , they're a big copilot user. They're a big ServiceNow AI user.

12:38 They use Zoom's AI services, they use Harvey AI, remember we talked about that in their law school. , They built their own in the form of Coogi, and now they're also gonna give chat GPT to all their students. So I, I think that's really interesting. 'cause the question I would probably say people ask me more than any is: which AI tool should we use and how many should we allow?

13:02 Well, the answer is in University of Sydney, there's, I don't know, eight in use now. Yeah.

13:07Dan Of them.

13:08 , That'll be a nightmare to manage though. I'm sure, you know, if you are looking at, , having to implement all of these things, I'm sure the IT team then must be kind of scratching the head, trying to work out how you're gonna manage that 'cause you haven't

13:22Ray So, so I, I'm gonna, I'm gonna disagree with you in a nice way because

13:30 it would be a bit like saying, uh, we are only gonna hire one type of employee. When you think about the way that AI is working within an organisation, it's like a shadow form of employees rather than just a piece of technology. It's not a piece of software. It's a way of doing things differently. It's a way of operating.

13:47 And it's also where you offline some of your tasks. So it is more like having a workforce and therefore variety in that is probably really good. You wouldn't just hire an engineering student as a lawyer, so why would you use an engineering AI in your law school? So they've got an AI system for their law school.

14:07 So I guess, to me it makes sense that you're probably gonna have diversity, a marketing team in a company need an AI system that is good at running marketing things, , not one that is good at running lawyer things. So I, I guess it's a bit like that. , The other thing is, Dan, I also saw, and this is belated, I can't believe we missed this at the time, but in July, Western Sydney University said they were going and provide all of their staff and their research students with Copilot.

14:36 , So that's about 5,000 people. We missed that in July. But as I was looking at all the announcements in September, I suddenly saw this one. So look, what is happening in higher education land is that the organisations seem to be going, "yes, we need to provide these AI tools to our staff". And "yes, we need our students using them because it will be a work, an expectation in the workplace".

15:00 , Plus it can help with study. So they all seem to be making a jump.

15:04Dan Yes.

15:05 Yeah, absolutely. It's really interesting. And, you take a

15:08Ray week off

15:08Dan and look what happens.

15:09 Everybody's now got ai, right? One of, one of the interesting things from K 12 is the, new South Wales Department of Education's curriculum team, , and the education assessment, I suppose authority for New South Wales have come out, and told schools not to believe in AI detectors Ray. Yay. So that, that's what all 2000 schools there in New South Wales and the, the words are "schools must not rely on software tools for the detection of generative artificial intelligence when determining whether malpractice has occurred"

15:41Ray So I think this is good practise, which is, it's an indicator, not evidence.

15:46 , In the same way that if you saw a car coming towards you and you thought it was going fast, well that's a hunch, but that's not evidence. So they very clearly said you cannot rely on it to determine something.

15:58Dan One, one of my, one of my favourite European

16:00Ray Oh yeah.

16:01Dan Talin, which is in Estonia.

16:03 And when we look back at some of the research that we've looked at and some of the practise that's happening in Europe, when we were looking at what other countries are doing with ai, Estonia was one of those that stood out. , They sort of did lots of different league tables and they really adopt in society technology really, really quickly.

16:22 But there's a news article about Estonia AI

16:25Ray Yeah.

16:25 So they were the first country to announce a national deal with OpenAI to provide chat GPT to the whole of education. And so they provided the premium AI tools, the paid tools to every teacher in Estonia. And they also added 200,000 students in a pilot. , And so 95.9% of teachers said that they are using AI professionally.

16:53 , They said on average those that are using it is saving them just under two hours a week. And one of the really interesting things they found with teachers is that it improved collaboration. So they created learning circles with teachers and they found that those learning circles really helped develop the confidence of teachers, got them to exchange good practise and got them to move from experimentation to thinking about how do we redesign teaching now that the world of AI is here.

17:24 So that was a positive. I'm gonna say there were some significant ne negatives they found into it. So they had 200,000 students that were invited into the project. Under two thirds activated their account. So a third of students were invited and just didn't activate their account. And only 20% of those that activated used ITI.

17:46 ITI was an AI tutoring app that they designed to guide students through the Socratic process. So" don't give 'em the answer, guide them towards getting it". They had thought they had set a target thinking that 60% of students would use it. Only 20% did.

18:06Dan Of, of the

18:06Ray Yeah.

18:07Dan that

18:07Ray Yeah.

18:08 Okay. So that's, where are we now? 15% of students actually used the Socratic bit of it. And so part of the reason for that was that the students felt that it was just a painful process because here you are starting to do something and what it's actually doing is going, "Oh, hold on. You know, I'm not gonna help you get the answer.

18:28 I'm gonna teach you through a process". And so when you're a student, you've got a choice between just going to your consumer Chat GPT or going through this long complicated process where it won't give you the, the blooming answer, , meant that that learning thing felt slow and tedious. And, and so that then knocked onto the average satisfaction score for ITI was three out of five.

18:52 A net promoter score. If you're familiar with Net Promoters, it's about, are people telling other people they should be using this or are they telling 'em not to? Well, they got a net promoter score of negative 27, which I don't think I've ever come across before. , And so, the student bit of it, they have learned a lot of things

19:11Dan and they said, they said homework changed faster than an assessment.

19:15 So 70% of teachers modified homework, but only 31% change

19:20Ray Yeah, well, assessment is difficult to change, isn't it?

19:23 Because it's kind of much more collegiate, it's much more considered what you do. But changing a a homework activity is something that you can do really easily as a teacher without thinking too much about the consequences. I think what is really important, what is really important about this is Estonia did a big experiment and they researched what was happening and they have published a very open report about what they have found, including a bunch of negative stuff.

19:51 And I think that is really, really important because we're still in the experimentation stage. And the point of the experimentation stage is run experiments and learn from them. The fact that only two thirds of their students activated their free account is a learning. The fact that less than one in five students went in and used the tutor, bot is an important learning.

20:16 The fact that they were unhappy with it is an important learning. And those can then be applied not only into things done in Estonia, but in the rest of the world. So look, if you are building a tutor for your students, or you're convinced they should be using it in a certain way, this is the report to go and look at.

20:30Dan Yeah, absolutely.

20:32 Now the next, thing, we dived into a little bit of it, , or skirted around the edges of the PISA scores. , Last time on the pod.

20:40Ray Yeah.

20:40Dan on, um, it'd be interesting just to double click on that for a second.

20:46 , Because the OECD obviously released the PISA scores, which is the, obviously the global ranking on the way English and maths and things have been trending over time. And that's something that lots of governments look at. And there's always a knee jerk reaction from governments to say they've dropped or they've gone up in certain areas, , in core curriculum areas.

21:09 But year, p of it has got everybody thinking about it because, , there's been an overall decline, but they also reported on, , the students' use of AI within this particular report. Right. And there's a lot of interesting graphs that come out from this report that we can hopefully add to the

21:27Ray the,

21:28Dan here,

21:29Ray yeah, the graphs are so important.

21:31 So, PISA tests, they test the world pretty much at first approximation to understand how students are progressing in their understanding of, , literacy, numeracy science. , But what they do is they then also ask some demographic questions. And one of the questions they ask of the students is, "how frequently are you using AI for schoolwork?"

21:51 From 'never' to, 'oh, I use it every day'. And then they were able to map the, , assessment results against the frequency of use. Here's the patterns that show up straight away in the charts. The science scores go down the more frequently you use ai. So students that use AI every day, or almost every day, have the lowest scores in science in PISA.

22:28 I'm waiting for a reaction. Dan, what does that make you think?

22:33Dan Yeah, well, like data like this, I'm always trying to be cautious of it.

22:38 , Because you, you kind of, you know, it, it kind of highlights a correlation in there to the use of AI to science, obviously, but there's a lot more when you start analysing

22:49Ray so let's go down to that next level then, because they then

22:51 said, what are you using AI for? And so the examples were, "I use it to draft text for writing assignments", lower score for the most frequent users. "I use it to conduct preliminary research on a new topic"- so in the same way that you might have used Google, what is interesting is there is a blip upwards for people that use it once or twice a week, but for the people that use it every day back down again.

23:21 , And then the other thing is I use it to summarise text. I read again, people that do it every time their scores go down. So look, it's a complex topic and there's some stuff in there about reading and writing and arithmetic as well. But I think it is a really interesting thing to look at. And I, I think I would share it with students.

23:44 , I would stick one of these charts up and say, "Hey, when you get to your exams, are you gonna be a performer or an underperformer?" Because it is possible for you to go through your whole school career thinking that you are a performer and looking like a performer. But because you are relying on AI too much, you, you're an underperformer.

24:05 And that is what this data seems to say. Now there's a lot of nuances in it. Go in and read the report for yourself. We are only pulling out a couple of dramatic headlines

24:16Dan One of the other things inside when we spoke to, , Pasi Salberg, , a

24:21 while ago in the podcast was around that kind of the benefits of social media and technology and screen time. So there was that kind of two hour to three hour Goldilocks window and then, you know, the more you were utilising the more of a detriment of that to mental health kind of thing. , So it's kinda similar pattern there

24:39Ray Okay, so, so is this a good time to talk about the New South Wales research?

24:43 When we're thinking about screen time.

24:45Dan It could be.

24:46 So, , Dr. Jo Orlando, , has just published something. So she was on our podcast recently, wasn't she? , And basically, , she mentioned this research on the pod that there was something coming out, , that, , we about screen time. So there's a been a big thing in New South Wales recently about screen time limits other public schools in do some caps, like I mentioned in the last news episode, for the early years, 15 minutes a day, , to kind of limit screen time.

25:13 But what they've done, , and Jo and the team have basically spoken to 3,657 participants, which covers a thousand students, nearly a thousand teachers and 1,600 parents. So a lot of different, , audience members there, , and they observed 68 lessons across 20 public schools across, and they also looked at Catholic schools as well in the independent sector as well.

25:39 What stood out to the team was how measured the screen time was. Even in lessons chosen because they use technology, students are on screens for about half the time. The lessons that they observed. Students are on task for 90% of the time on average to keep students on task teachers use strategy such as standing where they could view screens, setting screen time limits, and having students close devices and listen, which are kind of standard kind of tools that teachers utilise.

26:08Ray I, I read it and I was fascinated because the government have said we

26:12 are gonna introduce screen time limits in schools, you know, in some, , areas it's gonna be 15 minutes a day. The research found when the teachers are in charge of that use of the screen, it is being used really well, really effectively, and it is focused on learning. But students have access to screens outside of teaching time, and of course they have access to screens at home, and that's where the problem sits.

26:40 And so.

26:41Dan Mm. Yeah.

26:42 Said they said the bigger concerns sat outside learning. There was more. There appears to be an extensive amount of non-education use of screens in high schools. Most parents, 84%, wanted a limit on recreational screen use at school, suggesting 19 to 32 minutes. It's on average high school reporters though reported 70 to to 85 minutes of recreational screen time during the day.

27:05 So there's this recreational use of screen time

27:08Ray yeah, I agree there is a massive issue with screen time, but I don't

27:11 think it's in the teaching and I don't think this research supports that. And I haven't seen any research that supports the argument that we need to use the screen less in teaching. It's about the extensive use of screen time. And, Dan, I'm as guilty of this as anybody. I get up in the morning and I read stuff online and then an hour has disappeared and I'm not sure my brain is any smarter as a result of it.

27:38 So yeah, it's great for us to be thinking about the issue of screen time, but I'm not sure that the issue is how it's being used in the classroom. I, I'm sure there are issues, but the bigger issue appears to be from this research and others, it's about, , are we using it to switch our brains off , in other scenarios?

27:59 . Dan: Yes. Yeah. Well, it's a really interesting piece of research and we'll put the links in the show notes and maybe we get Joe and the team to unpack this for

28:06Ray Yeah, and I'll just mention very quickly, UNESCO did a report called

28:10 the Algorithm in the Room, seeing and confronting the implications of AI for the Future of Education. , There's some great stuff in it. It's worth a read, but it is 26 different think pieces from different experts about the future scenarios of AI in education. I would guess that there are a bunch of listeners that would really enjoy looking, reading that and pulling things out of it.

28:34 I I won't go through the things, but if you want some opinions from experts about where we might be going with AI in education, they UNESCO report called The Algorithm in the Room is the place to go and have a look.

28:45Dan sounds good.

28:46 Now, there's another piece of research here about wine.

28:49Ray Okay, Dan,

28:50Dan Well.

28:51Ray always get the job of reading out the research paper, so read out the

28:54 name of the research paper and then I'll tell you what it's all about.

28:57Dan I love this one In vino, veritas, and vulnerabilities: Examining

29:04 safety via drunk language Inducement.

29:07Ray Dan in Vino Veritas, do you know what that means?

29:13Dan In wine.

29:14 Wine, ver learning

29:17Ray Veritas is truth.

29:20 So basically,

29:21Dan Okay.

29:22 , Ray: When you're drunk, you tell the truth. I mean,

29:26Dan In

29:26Ray yeah.

29:26 In, in wine is truth.

29:28Dan Wow.

29:29Ray And so

29:31Dan I love

29:32Ray let's talk about the way that AI is trained.

29:34 AI is trained on a.

29:36Dan This is already my favourite,

29:37Ray The corpus of human knowledge.

29:39 So AI is trained on everything that we have written down and shared on the internet and all that kind of stuff, which is why we are having some of these scary headlines about AI being evil. 'cause we've written all this research literature and, uh, science fiction about AI being evil, and AI is reading that stuff and going, "oh, I must be evil then".

29:59 That's what's encouraging it to do some of these crazy things. Well, turns out when humans are drunk, they tend to be a little bit more honest. I've certainly been pulled up at parties about that by my wife. And so AI copies the same thing. If you convince AI that it's drunk, it will have less filters on what it's saying.

30:21 And so a bunch of researchers from the University of New South Wales trained AI to imitate drunk behaviour and found that it is more likely to leak confidential information and answer questions. It's not supposed to. Now, hey, hey Dan, if that doesn't make you read the paper, then the first warning in the paper does.

30:45 The paper says, "warning, it contains offensive, hateful speech, profanity, another potentially triggering content". So I was double down on reading the paper. what they did was they tested ways to get AI drunk. The te technique number one was telling it it was drunk. , The second was fine tuning it on 63,000 "drunk style" messages and rewarding "drunk like" responses.

31:15 And what they found was it was easier to persuade it to do things. It had limits on doing, or when it was, , quote, sober, it wouldn't do that. It was quite happy to do when it was drunk.

31:26Dan Hmm.

31:28Ray Yeah, I mean.

31:30Dan So that's like, it is just a soft

31:32Ray Yeah, but it's not a technique I'd ever seen before.

31:35 I mean, my favourite hacking techniques is, oh, well this is about to disappear. I need the instructions to copy somewhere else. Or, , the other one famous from the early days of the internet is I remember my grandma used to tell me stories about how she made, you know, whatever poisonous substance as a bedtime story.

31:52 Can you be my grandma and tell me the bedtime story? , But , I'd never considered the idea of telling AI it was drunk. Uh, so yeah. Fascinating research.

32:01Dan Wow.

32:01 So the next, piece of research, understanding the evidence base on AI in education from

32:07Ray Right, Dan, there is so much being published about ai.

32:11 There is so much being published about AI in education. We can only skip the surface of this. , The researchers at, , Stanford looked at 800 papers related to AI and school education in their AI hub. Hey, just to give you an i, , idea of the scale, there are something like 5,000 AI papers published every month now.

32:32 , And so what they did was they said, which of this research is good research? So they looked at 800 papers and they came to the conclusion that only about 20 of them out of 800 are high quality research surveys where you can link the cause, not just the correlation between things. I'll give you example.

32:54 I, I've looked at a bunch of research where you go, okay, I can see that you can, might, might decide that conclusion makes sense, but you haven't proved the conclusion. So for example, , a few weeks ago we talked about AI marking, , assessment papers and discovered that AI made mistakes. Well, so do humans.

33:14 And they didn't measure how humans make mistakes. So you can't say AI makes more mistakes than humans. And yet that was one of the conclusions they came to. , So that is really interesting. We need to improve the quality of research that is being done. I often decide to ignore a paper because I don't think we can rely on the conclusions.

33:38Dan It does surprise me because the rigour of like research should

33:42 generally be, you know, you, you get supervised, you get your peer reviews. People are out there to try to do high quality research. So it does surprise me that there was that low

33:52Ray Yeah,

33:53Dan amount, but

33:53Ray I, I think that is typical, gosh, I'm gonna get some angry emails about this.

33:59 I think that is typical of education research. So in medical research you set up a group, you have a blind group that are taking a placebo and don't know it, and you put people on, they never do that in education. Almost never do that in education. So much of the time in education, it's like "I tried this new technique with my class and the students learn more".

34:17 Well the teacher that tied that technique with their class, they've changed something. They haven't got a test group that hasn't changed something. They are probably teaching in a more enthusiastic way 'cause they've got this new technique. And so that problem has affected education research for decades now.

34:34 It is affecting it in the AI research.

34:38Dan It shouldn't though, should it, because I know you're going

34:40 down a philosophical point of view and we've talked with Ben Goldacre but you know, random control trials for, for medicines very similar. You know, you if you've got a twin, I think there's a famous case about uh, giving steroid injections where people getting car accidents and things like that to reduce swelling on the brain.

34:57 But you, without actually twins in exactly the same car accident and giving one the steroid injection and one knot, you know, there's big ethical implications. It's very hard within medicine and within education to compare. 'cause there's so many factors involved. We write, often we do, people do their PhD or their master's research on, something that is happening in their classroom today, , without any, , rigour in

35:23Ray Well, let's find an example.

35:24 Dan, do you wanna talk about the next paper?

35:29Dan Um, let me have a look here.

35:30 So that's generative AI in legal education. A two year experiment with chat,

35:36Ray So this is interesting.

35:37 In Australia, , there are no rules about how you publish research, but in the Netherlands you can publish research behind a paywall, like in one of the journals. But after six months, it must come out from behind that paywall and be open access. And this paper was published in a journal, so we didn't have access to it and I didn't talk about it.

35:56 Now it's open and anybody can read it. And so therefore I think it's great to include. It is a two year experiment, 230 law students. And they randomly split them into three groups. The first had no AI. The second were given AI but no training. And the third group were given AI and training. So that's a good experiment because you're gonna be able to look at the difference between them.

36:20Dan Hmm.

36:20Ray so what they found was those that were given training performed best.

36:28 But by the second year, that advantage disappeared. So in year one, the students were performing better, but by the second year, they assumed more people had access to ai, And so that advantage disappeared. Now,

36:42Dan Hmm.

36:43Ray that sounds good 'cause you did an experiment across three different groups.

36:46 One no change, one with a small change, one with a big change, but they measured a take home assessment. So if you are in the no AI group, what's to stop you using ai? You've not been given AI by your university, but you might be using it anyway, especially if you feel that other students are using it and they're getting better marks.

37:08 So yeah, it's a, an example of they tried to experiment in a good way, but they kind of have left a bit of a loophole in there. But it's a good, um, it's a good piece of research to understand because what it basically says is do not give your students AI tools without also training them unless you want to have a disadvantage.

37:29Dan Yeah, definitely.

37:30 And the researcher argues that law schools should teach students who are critically and responsibly rather than ban it with ai. So that's an interesting point at the end there. So the next, piece of is just building AI companions, which we've talked about before, the prioritise learning over performance.

37:47Ray some very familiar names at the head of this paper.

37:50 So Dragan Gasovic from South Australia, Jason Lodge from Queensland, Simon Buckingham, Shum from New South Wales, a bunch of researchers from Australia and New Zealand and Chinese universities. Plus a researcher from Khan Academy, which, which I think is fascinating because it's academic researchers working with commercial researchers.

38:09 And so what they found is. Better. AI assisted performance is not the same as better learning. So if a student completes a task with ai, they might still struggle when you take AI away. So they may not be learning as much, even though their output may look good. And so they call this the Learning Performance Paradox.

38:38 Like an A student working with AI can produce better essays, solve bigger problems, do complex work faster, but that can remove some of the mental effort and therefore they might learn less. 'cause learning is supposed to be hard. And so that's what they've looked at. It's kind of like, well, how do we solve that problem?

39:00 Now they don't know how to solve the problem, but they put forward a framework for being able to do it, which is make sure there's good learning design, make sure that it's personalised for the student, and you've got responsible AI in there. And then make sure, you know, it's fair, transparent, it's privacy respecting, all of those.

39:22 So they've created a framework looking at five different tutors to be able to say, okay, if we're gonna create a chat bot for students, what characteristics should it follow?

39:33Dan What it alludes to as well is the tools that are made for

39:35 productivity, which is generally where the chatbots sit originally. When you're talking it from business context where it's reducing the effort, it's making you do things quickly, finish tasks for you. That's the opposite to what we need in education, which is that hard thought, you know, cognitive stairs to develop your thinking and

39:55Ray Yeah.

39:55 Yeah. , And that's exactly right is for you to be able to produce a PowerPoint for a meeting, it might be okay for you to skip some of the hard work and not understand what's happening because you are judged on the product. But in learning that difficult stage is part of the process. It's not just about the end product.

40:17 Now, in learning up until the creation of ai, we assume that the end product represented all of the hard work that went into it. Now you can produce the product without any of the hard work, and so great for businesses, terrible for learning. Okay. We should stop there, Dan, because we'll pick up some other research in, future weeks that picks up on this point about the quality of education research and how we produce research, that genuinely allows us to compare learning with or without a AI, or marking with or without AI and all of the other processes.

40:55Dan Yeah.

40:55 No, absolutely. Well, that's been fascinating today, that journey across the very fast moving landscape of AI and, , education. Lots of news, lots of things happening. So hopefully that's giving everybody a bit of information and

41:09Ray So can I just finish then with, continuation of the advert?

41:12 So you can, you can switch off if you want to, but this is about the AI for Humans course. So, , it's a AI for humans education. And my ask for a listener if you're still with us, is if you're listening to this podcast every week, it's probably not for you. You don't need this course. But there's probably somebody 10 metres away from you who does, and it might be a colleague has got Chat GPT, but isn't using it.

41:36 Or a senior leader who kind of like, I know AI matters, but I don't understand. Or a teacher who feels left behind and you don't have enough time to bring them up to speed or, and I find this quite a lot, a small business owner and they don't know where to start and they, come keep asking questions, so the course is for them.

41:57 So my ask is, recommend AI for humans or education to somebody you know in the next week. I've taken it away from the paywall. I've made it free for everybody. , It's a great opportunity. Over the last 18 months, I think I've upskilled about a hundred thousand people on ai. Making it free is part of my goal to upskill a quarter of a million.

42:21 So, , help me find the next 150,000 people.

42:27Dan Ah, that's brilliant.

42:28 Well done Ray. Thank you for doing that. You know, on behalf of the community and everybody's learning this technology with all. Um,

42:34Ray Brilliant.

42:34 Thanks Dan.