Are AI Tutors Making Students Learn Less?

0:11Dan Welcome to AI in Education podcast.

0:14 Good morning, Ray,

0:15Ray are you?

0:17Dan and very well, thank you.

0:19 , Yeah, nice. , Morning in Sydney here. So getting, ready for the start of the week. So how are you doing?

0:24Ray I am great.

0:25 So last week I ran a couple of panels at the AEIEC conference, the Australian International Education Conference. it is

0:34Dan Right Okay.

0:35Ray big conference centre in Sydney.

0:38 I ran a panel and participated in another one. One of the panels was about international student employability. , Couple of interesting things. One was what do employers want? Well, number one, they want, they want graduates that them in the eye and hold a conversation. And that's relevant to ai, I think because.

1:02 I've certainly come across scenarios where students aren't comfortable doing that. They would rather have

1:07Dan Yes.

1:07Ray that, "Oh let me just ask AI the question before I give you the answer".

1:11 And the other one was. Interesting contrast, and I think this applies to listeners in the UK and the US as well, which is in the context of international students. They might come to say Australia or the UK or the US in order to get an educational experience that they think helps make them more employable at home.

1:31 But when it comes to AI

1:32Dan Yeah.

1:34Ray we have a different approach to AI than they might

1:36 have in their home country. So for example, in Australia, when we are evaluating our workforce training and knowledge of AI, we are 44th out of 47 large countries. India is fifth, China is sixth. When we think

1:54Dan Right

1:55Ray do Australians trust AI?

1:57 We are 37 out of 44 India a second, China a fourth, we are 37 out of 47 for our acceptance of AI. India are second, China are fourth. So , the point that we ended up talking about was we can't our attributes and, and believe that they are global if we are hesitant to use AI, but some other countries are going all in with AI.

2:24 That then is interesting from a student perspective.

2:28Dan Yes.

2:28Ray interesting from a, , an education, a business perspective.

2:31 If it's deeply built into the curriculum in China, for example, students in a few years time will come out with different skills and attributes than our own students.

2:42Dan Yeah, yeah.

2:43 Which is, which on one hand could be quick and soon. And on another hand, you know, some people are probably thinking, "why do we need to be like China? Why do we need to be like some of these companies?" So that's, so it's very

2:54Ray But, but it was an eyeopener really about that

2:57Dan Yeah.

2:57Ray attitudes of employers in different areas.

2:59 The other thing I've been doing is I've been creating some new AI courses. So I'm working with the Australian Human Resources Institute, so that's a member group for HR leaders and managers. Um,

3:11Dan Hmm

3:12Ray I've been creating is, "AI: Re-engineering, your HR team".

3:17 And so that's all about, well, two things. One is what does your team look like in a world of ai? Because perhaps you will have more agents doing tasks that, uh, helping people do things. So what does your team look like and your team dynamic when you are

3:33Dan Hmm

3:34Ray people and agents?

3:36 part of it is, , helping HR people to understand the impact on the rest of the organisation. one. The other one is, , AI 101 for HR. So a bit like my AI for Humans course, specifically

3:48Dan Yeah.

3:49Ray And I've been co building that with Claude.

3:52 So I have a system in Claude that has access to all of my collected content, And so really

4:01Dan Yeah.

4:01Ray helps me structure courses, you know, find the right case

4:05 studies and stuff like that. I loaded into it,

4:08Dan Yeah.

4:08Ray but also helped me create SCORM packages.

4:11 Dan, have you ever created a SCORM package before?

4:14Dan Yes, I have.

4:15 And it was a pain in the bum. Like I remember Moodle was kind of a, uh, you, you know, the, the problem I, I, let me see if I can remember it right. Uh, you fire on spot, it s score was Self-Contained Object Resource Management or something like that as an acronym. And, and basically if you creating a course courses consisted of like text video, but then things like quizzes and like other bits and pieces.

4:41 And it was always horrible because there was this single scum that you could, like if you created a course in one LMS and you move this somewhere else, it was always like this panacea of you create into a SCORM package, which is like a zip file, but it would never kind of work properly. It was really hard, , 'cause you'd miss out some of the interactive things, if you got polls and things like that.

5:01Ray need

5:02Dan so yeah.

5:03Ray software, like Storyline or

5:05Dan Yeah.

5:05 Yeah.

5:06Ray to produce it.

5:07 So yeah,

5:08Dan Yeah.

5:09Ray interestingly, six months ago, I couldn't have done this,

5:12 but in the world of AI, after I had assembled all of my stuff, and so I created some new videos for the courses, I created some assessments. I then got Claude to create the pages for me and then to create

5:26Dan Okay.

5:27Ray package.

5:27 And, and literally I said to it, "I want a SCORM package that I want to be able to test". So it created it, it

5:34Dan Yeah.

5:35Ray it itself.

5:37 I then took it to SCORM Cloud to test it. , There was a bit of a problem getting it into Canvas, and so I diagnosed that problem with its help, and it's like, "oh, well that's because it's been loaded in the wrong way. Here's the instructions to give to them to load it in the right way". So

5:52Dan Beautiful.

5:53Ray would've had to spend or four weeks putting that all together myself,

5:57 doing a whole series of manual tasks

6:00Dan Yeah.

6:00Ray buying a bunch of

6:01Dan Hmm

6:02Ray to be able to create the scorn packages.

6:04 Now I can create those, and whichever client I'm working with it, and whichever LMS they're working with, I can create a package for them. It is really fascinating how I don't need high level technical skills anymore to create technical objects. And that's because the AI systems are getting incredibly smart around being able

6:28Dan Yeah,

6:29Ray to do some of the technical process.

6:30Dan Site.

6:31 Totally agree. Yeah, I think I mentioned last time that I was doing something, , with, with some of the code stuff and put that out onto my, um, WordPress site. And yeah, some of those things, you know, it can create plugins for you quite easily and it's great. I, I've, I've been so sort of analogue, um, the last couple of weeks, , well say last week actually saying that I took a day off work last week and went to Adelaide Uni, where I'm studying obviously, and we had a research conference, uh, at the School of Education and Behavioural and Social Sciences, which is really good.

7:02 So all the research has come together. , Again, like, because my research has got with Gen AI in it, uh, a load of people kind of gathered around and were interested to understand, you know, about gen AI in education and the policy stuff that I'm looking at. There's a couple of other people doing comparisons between Australia and different countries of policy.

7:22 Um, obviously I'm focusing on Australia itself, but really interesting. Lots, it's good to see the research that's happening. There was some research on Minecraft as well, which is great, talking about that impact on learning. So, that's kind of, there was a kind of a bit of an academic download for me, which was great.

7:38 I spent a day with the school, Knox Grammar, shout out for them, Miles and Christine, Diniz, and Kate, and Nick, at Knox. They doing some amazing work for the traditional AI stuff that we used to talk about before Gen AI on the pod where you're talking about using AI to predict, student wellbeing and, and the NAPLAN results and HSC results in New South Wales and , student 360 stuff, which was brilliant.

8:01 So it's, it was, that was really nice to chat with them. We, I went in for the entire day there and it was, , really, really good. So if you're interested in data and analytics, reach out to Christine Denise , at Knox. She's just phenomenal. And I did some analogue stuff as well. I did volunteering, um, some give month at Microsoft, so they do double donations and things like that.

8:21 So I went to my favourite charity in Sydney. I've talked to her in the pod before. ReLove. They do a heap of stuff with, rehoming indigenous and dv and people from prisons and things like that. Setting up their flats and things is really, really good. So then we got, we've got lots of news. While we doing all this fun stuff at the same time, there's lots of news happening.

8:41 And I suppose just framing, just from an interesting point of view, it's not in our notes or anything, but the last 10 days of been, let's call it an season. , And you know, I always look at that because I've got chairs in different companies and whatnot myself, very personally. And it's always interesting as well to track the journey of AI and, you know, how companies are, you know, positioning themselves, you know, like the big players like Open AI and Anthropic and things.

9:08 They look into IPO coming up. But you've got the, the core players who are already on the stock market, like the Microsofts and the Googles and things like that, that they release in their earnings. And, and people look closely at that to see, "well, what are the trends?" You know, "what is going on?" Because actually the, when you look at, say somebody like Atropic, and there's a lot of analysis going on about that at the minute, um, nobody really knows the business models behind that.

9:33 Whereas you can look at a company like, say Google or Microsoft, who are actually selling this stuff properly, and you can actually see trends and see what's actually happening in the market. So I'd encourage, , our listeners , to look at the earnings calls, so the different companies, , that are doing things in, in ai.

9:52 So you can see where the, where the trends are going from a real perspective as well. So that's been happening over the week. , So there's lots going on.

10:00Ray Oh, another thing that happened, I, I got a massive delivery, about 10 days ago.

10:06 I'd forgotten about it, was the new Ethan Mollick book. I pre-ordered.

10:11Dan Oh,

10:12Ray 10 copies

10:12Dan intelligence.

10:13Ray got your copy ready for you, Dan,

10:16Dan Oh wow.

10:16 Thank you.

10:17Ray so his first book, Co-Intelligence, was the book that I put more post-it

10:22 notes in than anything I'd ever read. His new book, Co-existence. I'm, halfway through it at the moment. It is about how do you keep your senses and your intelligence working in an age of AI. So where, where, where do you choose to work with it? how do you choose to work with it? Just don't work over to it without being thoughtful.

10:46Dan that's gonna be a fascinating read following on from the last,

10:49 actually something coming up as well. We, we, we, we are going on, going on rabbit holes here, but something coming up next week, , is the Castlereigh, catch up again for K 12. , So some people listening in might be, are you going along to that one Ray, would you be there?

11:02Ray I'll do, Dan, because it is always great to, to meet

11:05 listeners when we go to an event. So I am gonna bring two copies of Co-existence and the first two people that come up to me and ask the copies

11:15Dan Oh wow.

11:16 , Ray: Who have listened to this episode, I will hand over a copy of his book and you can enjoy it hopefully as much as I do.

11:23Dan Oh, fantastic.

11:24 So what's happening on the news then? So what have you seen recently?

11:29Ray so this one's about Microsoft Dan.

11:31 Um, sure,

11:32Dan Oh, okay.

11:32Ray got the inside line in advance of this.

11:34 I had to wait until I saw it in public, but basically Microsoft have announced, , a medical AI model - So it's being used for diagnostic services - which is 1.7 times more accurate than doctors. So this, this is really interesting. I think we had this before in the past with, , Dan: Hmm.

11:57Ray where turns out that AI is more accurate than a, a medical

12:01Dan Radiologist.

12:02Ray And so

12:02Dan Hmm

12:03Ray this new model, MAI-DxO.

12:08Dan Oh, some guys created that clearly.

12:10Ray I mean you can tell it's from Microsoft with a product name like that.

12:15Dan Yeah.

12:15 Yeah. Yeah.

12:16Ray it.

12:17 tested it by, , doing diagnostics, , versus a, a standard set of diagnostic cases. what they did was to the, to look to see how accurate the AI model was compared to how a normal physician, , analyses things. And they found it is both more accurate and it's way cheaper. And, and by cheaper it's like, you know, obviously running it, but also things like ordering the right in enabled to be able to problems.

12:46 , So I know that isn't directly relevant to education, but here's the interesting thing. In the licence exams they give to doctors, what they do is they give you a scenario with all the symptoms of a patient, and you have to work out, of the tick boxes, which are the full diseases or whatever. that doesn't reflect how things happen in real life.

13:10 'cause you go into the doctor and you go, I've got a cough. And they ask you a couple of questions and then they maybe start to narrow it down and then maybe they order some blood tests and eventually they come out with a conclusion. But in the way they do exams, they give you all the information you then decide on the answer, well that isn't real life.

13:31 so they decided to build a bunch of that they can test with AI around giving a little bit of information and the AI being able to do the diagnostic process on

13:45Dan Right.

13:45 Okay.

13:45Ray And it made me think about some of the exam processes

13:50Dan Hmm.

13:51Ray If they're testing doctors

13:51Dan Yeah.

13:52Ray by giving them all the relevant information, rather than get getting

13:55 doctors to elicit the information from a patient, I wonder if we're doing that in other areas of assessments where,

14:04Dan Yeah.

14:05 That's interesting.

14:05Ray really evaluating you in a real life scenario.

14:08 So, um,

14:09Dan Hmm.

14:10Ray using exams as a benchmarks for ai, but perhaps there

14:13 isn't the same as real world. So it's interesting that the AI companies are now saying we are building better exams.

14:22Dan Yeah.

14:23 Yeah, no, absolutely. That, that's, that's an interesting reading between the lines, that, to drive that up and, and think just from, um, just a little bit of a wit bit or information, when you see some of these models that come out that say Microsoft, they usually start with MAI. So that's Mustafa Suleyman's kind of org system, Microsoft AI groups.

14:42 So when you're looking at the models like GPT and Gemini and things like that, the first party models they've got are called MAI ones, just out of interest. , And that's the medical one.

14:52Ray the medical one.

14:53 They've also got, , that I've tested the, the leading voice one. Text to speech, their, their text to speech model, I think is, is leading the benchmarks at the moment. So it is interesting that we're starting to see more of these AI models coming

15:08Dan Hmm

15:09Ray rather than I think what typically Microsoft customers have used in the

15:12 past, which is they've used OpenAI or Anthropic or whatever Microsoft platform. Now we're starting to see Microsoft's own specialist modules coming out.

15:24Dan Like Harvey.AI for law and things like that are built into those.

15:27 Yeah, it's interesting. Interesting. So then the next, next article here that you've picked up on is that the Dartmouth Provosts resign over AI writing. then, and as soon as I saw this, I'm like, oh no, not another one.

15:39Ray it's been an interesting couple of weeks because on

15:42 the one hand we've got, yeah. So the provost at Dartmouth had been writing research papers with AI and writing newspaper articles with AI in journals and newspapers that required a, , declaration that AI had done the writing. And didn't do that. And so, eventually it turned into a bit of a mini, controversy and they ended up resigning.

16:07 The fundamental thing is, if you are create, creating stuff for a journal or a research paper where they say either "you can't use AI" or "you have to declare your use of ai", then that's what you should have to do. I think I've said to you, I hesitate to write much with AI now, am very comfortable with declaring where my, my AI use is like the Responsible AI course.

16:31 Every single page in the course had a little dropdown that explained how AI was used in the creation of the learning resources, the, the quizzes or whatever it does it is. , And I think that kind of transparency builds trust.

16:46Dan A hundred percent.

16:47Ray uh,

16:48Dan Yeah.

16:48Ray that happened, so two more American universities went all in with chat GPT.

16:53Dan Saw that.

16:54Ray went in and they're gonna give it to their staff and all of their students.

16:59 It still is a demonstration of the fact that it's very divisive. 'cause in one university we've got somebody resigning, in another university, we've got them handing it out for free to everybody. It was Princeton, that, that caught my eye. But then I discovered that Carnegie Mellon had said the same thing.

17:16Dan Mm.

17:16Ray Virtually every news episode over the last two months, we've

17:20 seen a university providing AI tools to all of their students and staff that is continuing at a pace.

17:26Dan Yes.

17:29 Hmm.

17:29Ray at some point, we're gonna have to have the reckoning

17:31 about cognitive offloading. Are you making yourself dumber

17:34Dan Yeah.

17:34Ray by just giving jobs to AI that you should be doing as a human?

17:39Dan And, and universities and, and obviously just delineating it again

17:44 and just what, while we're having this discourse, universities are thinking about this from a academic efficacy and philosophical reasoning approach. And then K 12 are also thinking about it in terms of learning development. So you've almost got like the two ends of the learning spectrum here. It's kind of like, you know, the purpose of university education is original thought pushing boundaries, coming up with something, , to aid and, and support society in a bigger way.

18:15 Um, where you can kind of do deep research and, and you're doing things that are scientifically rigorous. And then you've got the K 12 people, which have got their angle of either, are we doing cognitive offloading and are we, you know, supporting younger kids? And when do we start to bring that in K 12 education?

18:33Ray So that leads into the next story that I saw.

18:36 And Dan, this is a breaking story that is happening. We're recording this on Wednesday morning. By the time the listeners listen to this on Friday, things will have moved on. But the state of play

18:46Dan Yes.

18:47Ray Wednesday morning is that Google have announced that they've listened.

18:52 Do you remember maybe a couple of months ago, Dan, we had a bit of a conversation on the podcast 'cause Google had made their Gemini AI tool on by default for all schools.

19:03Dan Yes.

19:04Ray I think we talked about the fact that it was probably wrong to make it on by

19:08 default, especially for younger age kids. Why would a technology company say, we are gonna give these tools to all of your students unless you block it? So now reversed, they've heard that feedback and they've said they are now gonna turn off AI by default for all schools, and allow schools to switch it on.

19:30 Now that controversial. I think that's the right way to work. But what's controversial is they said if you are gonna turn off AI, then you're not gonna be able to use Google's Search either. So, you know, in

19:46Dan Hmm

19:47Ray much now whenever you do a search, you don't get the old

19:50 fashioned lots of blue links. You get the AI written response. So they're saying if you turn off AI, it's gonna turn off search. Totally. Not just the AI answer bit.

20:03Dan Right.

20:05Ray I've seen some reporting in the Wall Street Journal, I've seen some

20:08 reporting in the New York Times, I've looked at Google's original blog post, it seems to say that you can turn off each of those things. I wonder if there's a misinterpretation. I guess we will know. But it is interesting that there's this kind of default thing of added AI into search, and now the implication seems to be that you can't turn that

20:34Dan Yes.

20:35 Yeah, I know. And there is, I did notice this is a bit of, um, nuance in this as well, and it was a bit, there could be misinformation there. So we need to see what comes out in the, in the wash with that one, because yeah, if, if it does, then it's gonna, you know, where you don't wanna have the position of the, you need to turn AI on to actually have access to the default search.

20:56 But yeah, it's an interesting one.

20:57Ray the lesson is we continue to see that AI is divisive.

21:01 if you mention AI

21:02Dan Yes.

21:02Ray people mentally flip to one of two directions

21:06Dan But there's a driver isn't there as well, because I forget if it's in,

21:10 it's the UK and there's stuff coming out now in to do with the process automation stuff in the Privacy Act in Australia. There's a lot of policy that's driving some of the companies to follow some of these guidelines now because I think the US say that you've gotta be able to turn it off if you want to, and you've gotta be transparent and things like that.

21:30 So the companies are having to add these features in, , which they might not have done before.

21:34Ray So from December, organisations [in Australia] have to have in their

21:37 privacy policy, this is where we are using personal data to make automated decisions. It isn't an AI thing, a, a formula in a spreadsheet counts as much as a piece of ai, but it is driving more towards transparency about using AI in outputs.

22:00Dan Yeah.

22:00 Yeah. So the, from some of the news that we've just been talking about and kind of going into research a little bit, just one other thing worth mentioning, which is in the news just recently as well, Jason, uh, Jason Claire, if I can say his name properly, the education minister, for Australia, posted something recently.

22:17 And basically what he said is, so what's happened in Australia, just for people who are not in Australia, is each individual state has their own education ministers. And they all met over the last couple of weeks to talk about things to do with technology in classrooms and AI and education, behind closed doors.

22:33 And they've basically come out and we've got several different bodies in Australia that manage education. We've got ACAR, which is for reporting curriculum. We've got AITSL, which looks after teaching standards and school leadership. You've got AERO which is a kind of controversial, , education research organisation and ESA, the Education Services Australia.

22:53 We've had on this podcast as well, talking about policy. And they're bringing all those together in one non-governmental organisation, or however you wanna call it, over here a minute. They're gonna try to manage that under one body to be able to kind of bring research and school leadership and everything together, and policy, which, which sounds quite a sensible thing to do.

23:14 But again, there's a lot going on under the covers here about AI and technology in the classroom. And certain states Prue Carr came out in New South Wales saying that they won, go back to pen and paper for the NAPLAN testing and or, and taking away certain elements of testing. So there's a lot going on.

23:32 And the AI kind of point the end of this is pushing, a lot to some of this rhetoric and policy and, you know, all, all this stuff down, down into certain roads, I think. Have you got any points on this? Have you, have you followed this at all closely?

23:46Ray I don't think it's as much a backlash as it is, recognising

23:49 that we've gone a bit too far. know, if you are a young child at school, probably doing your tests on paper is the right thing to do. Especially in an environment where we know that with children are having too much screen time. So

24:05Dan Hmm.

24:05Ray I think it's just a balancing thing.

24:08 . And so all of these projects, I feel, tend to cause things to slow down until they're resolved. So I wonder if we're gonna get a bit less progress in regulation and guidance around AI, for example, because for the next two years people will be squishing all these organisations together as opposed to organisations working independently and trying to move forward at speed.

24:35Dan Yeah.

24:36 Yeah. So going into research, the first news research article here is about the UK's AI pilots and research. So that's based on the thing that we were talking about that was in the news a while ago, right? About the fact that the UK would investing in AI tutoring.

24:53Ray So they put about 23 million, pounds as an investment plan to, one of the

24:59 projects was around creating AI tutors. And the idea was that they were gonna run pilots, with industry and teachers evaluating AI tutors. And not just go for one, but evaluate a few. Now, this was a while ago, 'cause their goal is to get 450,000 students. I'm gonna say though, that it wasn't that long ago, 'cause it was only February.

25:21 Now the first research report has come out from the first pilot. that is

25:28Dan Wow.

25:28Ray incredibly fast, I think,

25:30Dan It is, yes.

25:32Ray they announced they were going to tender select some suppliers, and

25:36 here we are, first week of November, we've got the research out already. That's astonishingly fast. So comes from one of the trials, which was, , Google and their DeepMind group have been working on specific tools for AI, for learning. And Eedi, which is a, a maths platform for students. The has been published by them.

26:05 But having reviewed it, I think they are being pretty open about what they found. Good news, it's a random control type trial. So

26:16Dan Woo.

26:16Ray they take a group of students and randomly split them into different

26:19 groups that receive different processes. Just over a thousand students. Studying in maths. And they put them into this randomised control trial, which means students when they were answering a question incorrectly in the maths platform got three different types of responses. One group got a static hint, you know, the usual thing of, "Hey, you got that wrong. And here's, here's a, a hint".

26:46 The second group got assigned to getting help from a human tutor. when they got something wrong, a human tutor would intervene with them and go, oh yeah, I can see what you've done. Let me give you a few hints. then the

26:59Dan Hmm

27:00Ray third group got an interactive AI tutor, which at the

27:04 teaching resources and then had

27:07Dan Yep.

27:07Ray underneath kind of giving the student tutoring.

27:11Dan Okay.

27:13Ray that is good.

27:15 The surprising finding from the research two things. AI tutoring was as effective as expert human tutoring, at helping pupils to understand the math problems. And students that received the tutoring rather than just the stock answer did better in a mid project test, like five weeks through, they were doing better, but at the end of the 10 weeks, they tested all the students - Those that got no tutoring, those that had human tutoring, and those that had AI tutoring, they all scored the same in the end, course,

27:52Dan Okay.

27:53Ray the 10

27:54Dan Right.

27:54Ray Um, now

27:55Dan Hmm

27:55Ray This needs some exp explanation.

27:59 So the first thing is the AI tutor was designed so that a human reviewed the answer it was gonna give to the student, before it gave it to the student. And what they found was when they did this a year ago, 76% of the answers were accurate. So a quarter of them

28:21Dan Right

28:22Ray needed some help,

28:24Dan Yes.

28:25Ray this year, 95% of the answers didn't need any modification.

28:31 So

28:32Dan Okay.

28:32 Okay, so the ai, yeah.

28:34Ray AI is writing a great tutoring answer in the view of

28:38Dan Hmm

28:39 , Ray: And that that is changing just a year. thing, this, this again an interesting thing. They compared human tutors and AI tutors, and they found that one quarter of 1% of answers had maths errors in them. So the AI answers one quarter of 1%, So that's like one in 400, answers had a maths error in it.

29:03 And turns out that's almost exactly the same as human tutors. So yes, the AI is still making errors, but so are the humans.

29:15Dan Human.

29:15 Yeah. Two.

29:16Ray Now comes the really important point.

29:18 What they found is that students that had tutors progressed slower through the course. So I think we've seen this in other research that we've been talking about. When you make a mistake and the tutor steps in to help you, they're effectively slowing down students because they're going, "Oh no, we want you to understand this properly".

29:40 So they're kind of going back and, and reteaching it.

29:44And so what they found was students had access to the AI tutor studied about a

29:52 third less than the students that didn't. So I'm going through, I'm doing some math practise exams when I'm getting things wrong. It's just going, "Oh Ray, you forgot this". Or it's going, "Hey, let me tutor you and do the Socratic thing and ask you our whole lunch of questions". I am getting through a third less of the curriculum content.

30:12Dan Right.

30:13 Yeah. Okay.

30:14Ray that against human tutors, they found out that human tutors, you get

30:18 through about half of the content. So you slow down even more. So that is a really interesting finding. So adding a tutor seems to help you understand more, but it slows you right down.

30:31Dan Yeah.

30:31 That is fascinating. That is,

30:33Ray That I think might explain why at the end of the course when they

30:36 gave students an exam, that have been tutored scored less that hadn't.

30:43Dan Because they may have missed other concepts out because they

30:45 were slowing down a little bit. Yeah.

30:47Ray that and said, oh, we probably need to allow for that in future research.

30:51 , So yeah, that, that was, , interesting. And the students liked it because they didn't have to keep waiting for an answer. could just go straight into AI tutoring mode

30:59Dan Yeah.

30:59Ray and get support straight away.

31:02Dan Totally.

31:03 So the next, piece of research, AI humanises and the political economy of academic integrity in higher education. Humanizers? what's, a AI humanizer, right?

31:14Ray A Humanizer, Dan, is something that takes a piece of AI writing and make it

31:19 seems like it's been written by a person. And

31:22Dan Hmm

31:23Ray curiously humanizers are ai, they're a bit of AI that's been told.

31:27 Don't use the, it's not this, but this. Don't use M

31:29Dan Mistakes.

31:30Ray that kind of stuff.

31:31 So this is work

31:32Dan Yeah.

31:33Ray people we know and love.

31:34 Jasper Roe from Durham, Mike Perkins

31:36Dan Yay.

31:37Ray British University of Vietnam.

31:38 And, Leon Furze from Deacon University, and I would say rural Victoria. So, so they are the AIAS people remember the traffic light assessment thing.

31:47Dan Yep.

31:48 . Ray: So they look at, to see how these humanizer are being pushed, and these humanizer basically are saying, "oh, we are helping you poor students from being caught out by AI detectors".

32:00Dan Yeah.

32:01 Yeah.

32:02Ray and so they're doing that kind of lovely messaging, but to me it's about the

32:06 escalating war between AI detectors, AI Humanizers, and the poor students stuck in the middle, basically getting more

32:14Dan Yeah.

32:15Ray are they gonna get detected and accused of cheating, whether

32:19 or not they are cheating.

32:20Dan Yeah.

32:21 , Hmm

32:22Ray the same as like essay mills where you would go to an essay mill and say,

32:26 "quick, I need an assignment writing". Or even,

32:29Dan Oh, an essay mill.

32:30 That's an interesting one.

32:31Ray PhD writing though, write your PhD for you, Danny, if you want.

32:35Dan Yeah.

32:35 Yeah.

32:35Ray but now it's like you go to AI to write it then you go into an

32:39 AI humanizer and pay them to make it look like an AI didn't write it.

32:44Dan Mm.

32:44Ray Now in Australia,

32:46Dan Yeah.

32:46Ray Essay Mills are illegal.

32:48 TEQSA

32:49Dan Hmm.

32:49Ray notify the internet service providers of websites to block

32:53 that contain Essay Mills, but I don't think AI humanises are yet. you

33:00Dan Hmm

33:00Ray an arms race, isn't it?

33:03Dan Yeah.

33:03 Yeah, exactly. You've got, you've got these two things going, like the clapper's. Very hard to keep up with that. Yeah.

33:11Ray um, I, I've written stuff that AI detectives will say all

33:14 that sentence, don't like that. 'cause that looks like it was AI written, but I wrote it. And we know from the research that, you know, certain types of students can get falsely accused. So I dunno, it's just this is just the continuing battle between students and the institution. And in the middle are tech people going, oh, we can help solve the problem.

33:37Dan Yeah.

33:37Ray time, there's other tech companies that are creating the problem.

33:42Dan Yeah, true.

33:43 , So next piece of research, "the effects of course, integrated ai, tutoring on student performance and engagement. A randomised university trial, right?

33:54Ray This is two RCTs - randomised control trials

33:58Dan I.

33:59Ray in one week.

34:00 , So

34:01Dan Yeah.

34:01 Amazing.

34:02Ray split the students into different

34:03Dan Yeah.

34:04Ray This is and a half thousand students, 30 lecturers, a bunch of subjects

34:10 like science, maths, and humanities. And

34:13Dan Hmm

34:14Ray was they gave

34:15Dan good.

34:15Ray an AI tutor and they found that the students that were given

34:20 access to the AI tutor scored less.

34:26Dan Okay.

34:29 , But these are the AI tutors that are embedded into a course as well, so Okay.

34:33Ray yeah.

34:33 But, the other interesting thing is they found that those had access to a tutor engaged less with the learning platform and participated in fewer activities. So they were less engaged learners. But here's the twist, Dan. Only 15% of students that were offered the tutor actually even used it once. And those that used it

34:55Dan Right

34:56Ray just went in and go, "oh, just gimme the answer. None of this Socratic stuff,

34:59 just tell me the answer to this question"

35:02Dan Hmm

35:02Ray this thing to me.

35:05Dan Right.

35:06 It's interesting, isn't it? 'cause just as an anecdote, like as lots of our courses in Microsoft now, there's the content as you're going through, you get the, intermediate questions and things like that, quite three, four things. But at the end there's usually now a, , complete module you do, which is an AI like discussion with, a end user, so like a CIO or whatever.

35:29 And then it's grading you on that conversation discourse. So I think it'll be quite interesting over time to start to understand what AI tutors and what the support is this given, or how the things are assessed. It'll be quite good to understand that as, as this debate gets more and more mature, I suppose.

35:48Ray Here's the point.

35:48 So this research, of the suggestions from the researchers is that simply adding AI into the process might change the way that students think about learning, even if they don't use the AI. And

36:03Dan Yeah.

36:03 No, totally. Yeah.

36:03Ray scenarios, I know you've talked about the scenario where

36:07 you, do effectively a role play, you

36:11Dan Hmm

36:12Ray more engaged because you feel that you're getting something

36:14 that you can't get otherwise. But if you add a bit of AI into some of your teaching practise, some students might feel aggrieved that they're not getting what they think they're paying for, which is you teaching them.

36:27Dan Yeah.

36:27 Yeah, yeah, yeah. No, true.

36:29Ray other interesting thing about this research is it found that some student

36:32 groups were affected more than others. So the example was first in family students, they did worse. So that's a really good example of you come to university as a first in family student, and you expect to get taught. And so by simply adding a bit of AI into the process, you might be disengaging those students from the learning process

36:55Dan Right.

36:56 That's interesting. Yeah.

36:58Ray this is all this stuff coming out on tutoring I think is so important

37:02 because what we're seeing is the impacts on, learning the impacts on final exams. Much of the impact on final exams is either non-existent or negative,

37:13Dan Hmm

37:13Ray then there does it knock on into engagement.

37:16 And that's certainly one of the things we're seeing is: simply providing an AI tutor doesn't necessarily mean that students are gonna use it. So yeah, we've got this issue around: how do we motivate learners to learn? We're back to the basics, aren't we?

37:32Dan Yeah.

37:33 Yeah. I, I feel as if the, you know, I'm not saying Microsoft's gonna right by any stretch, but, you know, internally, it's almost like, as you were talking, it's almost like our, our learning content has almost got like a mini viva at the end. So it's like, "okay, you've done all of this stuff. Now let's speak to somebody about it", it's obviously got rules and regulations in play there. You'd ask your questions and say, "so tell me your challenges in your organisation". And then, then you like hone in on some of those and you, you offer some ideas and plans, and then they'd repeat back to you and say, "well, actually I don't really wanna do that.

38:06 I'm more concerned about something else". It's just like, it feels like you're being tutored in the exam part at the end. You know? It's, it's really interesting. It's like a coaching conversation, I suppose.

38:17Ray but,

38:18Dan hmm

38:19Ray that is useful if you are motivated to get that coaching conversation.

38:23 If you're not motivated

38:24Dan Hmm

38:24Ray it, just go in, answer some of the questions, you know,

38:29 pretend to have a conversation and

38:30Dan Oh yeah, totally.

38:31 . But that also happens with real people as well, doesn't it? I had to do a course last week on, , situational fluency's called, to do with like when you're in this situation in an organisation. And, and it was face to face and it was role playing it. And then I broke into, groups of four, you know, they did a breakout room and one of the guys said, "look, I'm really busy.

38:50 I am been engaging in this. I'm here for the credit. so I'm not gonna engage in the discussion at all". , And then you've gotta give him fair deals. You're like, okay, fine. You know? , So then the three of the three of us had this discussion, and rather than four of us, and, and you had to do kind of role play thing.

39:05 And one guy was really eager to do, and he said, "well, I've got a customer meeting tomorrow. It'll be really helpful for me". So, you know, by osmosis, you know, and everybody would, I was thinking about it as I was going through it, you know, like, AI might have been better. It might have been, you know, but people still at disillusioned, whether you're using AI or not, that was like a, a face-to-face, well, well, kind of virtual training, right?

39:29Ray You're describing every university group project ever, aren't you?

39:32Dan Yeah.

39:32 Yeah, yeah, yeah. I know. Yeah. True.

39:34Ray seen is much research around the idea of adding AI into the conversation.

39:39 It tends to be

39:40Dan Hmm

39:40Ray on one-to-one tutoring or one-to-one scenarios.

39:43 But I wonder what

39:44Dan Hmm

39:45Ray when you add AI as a member of the conversation, not as the

39:51 other side of the conversation.

39:53Dan Yeah, that would've been quite interesting because the three of us

39:55 would've been able to chat about whatever that particular concept was at the time, and then that, that that AI person could have been a pretend customer, could have been a coach, whatever. Yeah. Interesting.

40:06Ray I've been building a simulator for, , emergency incidents that

40:10 we've been testing out with, organisation policies and everything. But that's a one-to-one thing, and next step is to have it coming in and participating in the exercise with a group of people. So in, I don't know, six weeks time, I'll know what is possible in that space

40:27Dan Yeah, maybe you should add one into, maybe you should add one into a podcast

40:31 episode one day when we are talking about the research and see what it can add.

40:34Ray Maybe I should start writing research papers, Dan.

40:36 Okay. Hey, great to see you.

40:38Dan Thanks again.

40:39Ray we came into this thinking we didn't have that much to talk

40:41 about, but we've gone over time. Again, Dan.

40:44Dan I know.

40:45 Yeah. Well, thanks. Thanks again, Ray, for doing all that research and hopefully that's been useful for our listeners. Take care.