0:00
ChatBT recently released study mode.
0:02
It's OpenAI's version of trying to make
0:04
ChhatBT even better for studying and
0:07
learning. But how good is it really?
0:09
I've been a learning coach for over 13
0:10
years, and in this video, I'm going to
0:11
do an in-depth review on ChatBT's study
0:14
mode. I'll go over the good parts, the
0:16
current issues with it, and some tips to
0:17
make the most out of it. It's definitely
0:19
a powerful tool, but there are some
0:22
nuances with how you need to use it. Uh,
0:24
and in my testing, I found that if you
0:26
don't use it in the right way, it was
0:28
actually slower than like just literally
0:31
reading a book the oldfashioned way. But
0:32
if you're not familiar with study mode,
0:34
what is it? A lot of people are already
0:36
using chatbt for learning. But with any
0:39
new tool, there is a right and a wrong
0:41
way to use it. And I've made previous
0:43
video about how if you use chatbt in the
0:45
wrong way, it's actually going to be
0:47
more harmful than beneficial for your
0:48
learning. And it just creates this
0:49
illusion of learning where you feel like
0:51
you're understanding everything but
0:53
you're not actually gaining knowledge
0:55
that sticks with you. And one of the big
0:57
reasons for that was because it's so
0:59
easy to extract information from chat
1:01
GBT that it's easy to become passive in
1:04
our learning and passive learning is
1:06
ineffective. So OpenAI clearly was also
1:09
worried about the same thing because
1:10
study mode is basically their attempt to
1:12
address this. In their release, they say
1:15
that it offers stepbystep guidance
1:18
instead of quick answers. It's really
1:20
easy to activate. You just go onto the
1:21
website, you you click this button that
1:22
says study, and it will just put it into
1:24
study mode automatically. And now it's
1:26
like having a personal tutor right next
1:28
to you. And OpenAI say that they have
1:30
worked with learning science researchers
1:32
to get chatbt to interact with you in a
1:35
way that actually supports real learning
1:37
is what they say. The idea is that it's
1:39
meant to be more interactive, more
1:41
engaging. It challenges you and
1:43
facilitates your actual thought process
1:45
rather than just spoon feeding you
1:47
information which is the big problem.
1:49
And this is something that I'm really
1:50
excited about because if they can get
1:52
this right that is a huge step in equity
1:55
of education. A lot of you probably
1:57
don't know this but uh you know I've
1:59
been a learning coach for over 13 years.
2:01
More than half of that time at the very
2:03
beginning was in the nonprofit space. So
2:05
I was uh going into schools from lower
2:07
socioeconomic positions and I was
2:09
teaching them uh these academic skills.
2:12
One of the major issues I saw was that a
2:14
lot of these students are driven and
2:16
motivated but they just do not have
2:18
access to top academic resources. Like
2:20
they can't get a private tutor. A lot of
2:23
their teachers are turning over very
2:24
quickly. One of the schools I was
2:25
partnered with in a single year half of
2:28
their entire teaching staff turned over.
2:31
like they left for a different school
2:32
cuz they're just so underfunded and
2:34
underresourced. And so at the time my
2:35
solution was to just go into these
2:37
schools and and tutor these these kids
2:39
for free. But obviously that's not
2:41
sustainable. And so if ChatgBT is able
2:44
to be like an unlimited almost free
2:47
tutor available 24/7 for these students,
2:50
that is a big deal. So to see how good
2:53
study mode really is and how reliable it
2:55
is for you guys to use, uh I ran a
2:57
series of tests on it for around four or
2:59
5 hours of studying. These are the tests
3:01
that I ran. First, I studied a topic
3:02
that I have some background knowledge
3:05
on, but I'm certainly not an expert. I'm
3:07
trying to simulate like a first year
3:08
university undergrad kind of knowledge.
3:11
And so, the topic that I picked for this
3:12
was LLMs and transformer architecture.
3:15
And I tried studying as two different
3:17
types of learners. The first type of
3:18
learner was someone who isn't very
3:20
initiated with learning science or
3:23
learning methods uh and doesn't have a
3:25
lot of metacognition, which means that
3:26
they're not really aware of their own
3:28
thought processes. So, I'm asking fairly
3:30
basic questions. If I'm feeling
3:31
confused, I'm not using the strategies
3:33
that I know to work through that
3:35
confusion. I'm trying to work through it
3:36
in a way that I know that most students
3:39
uh would would typically work through
3:41
this, being really led by the teacher,
3:44
in this case, study mode. If I got stuck
3:46
or confused, I would just ask questions
3:48
that I felt naturally a student might
3:51
ask about the topic. In the second type
3:52
of learner, I fleck open to a much
3:55
higher order of thinking. I was more
3:57
metacognitive. And this would reflect a
3:59
learner who is much more experienced
4:00
with their learning, maybe has been
4:02
trained on learning methods. I would
4:04
evaluate and reflect why I'm getting
4:06
confused and why I'm getting stuck. And
4:08
I'd ask very targeted questions to see
4:10
if it's able to lead me in the right
4:11
direction. Typically, when you use
4:13
chatpt without study mode, the value of
4:16
the interaction is really dependent on
4:18
the way you interact with it. So someone
4:20
who is in that second type of learner
4:22
who is doing that higher order thinking
4:24
and asking really targeted reflective
4:26
questions, they're able to get to the
4:28
answer uh that's meaningful for them
4:30
much more quickly. So they can make it
4:31
click very rapidly. Whereas the first
4:34
type of learner that's a little bit more
4:35
passive uh and doesn't really know how
4:37
to work through their own confusion,
4:39
they're going to struggle. I then made a
4:40
new chat with a blank history and then
4:42
repeated those tests for medicine. This
4:45
is to see if study mode interacts
4:47
differently for different types of
4:48
subjects. Medicine is a subject that I
4:50
have background knowledge on. I have a
4:52
medical degree and I was a medical
4:53
doctor. Uh but also I used to teach
4:56
those subjects extensively like hundreds
4:58
maybe even thousands of times. So I'm
5:00
very intimately familiar with the types
5:02
of questions that students tend to ask
5:04
and where they get stuck on certain
5:06
concepts. So I'm able to simulate a
5:08
beginner learner very accurately. And
5:10
then once again I made a new chat and
5:12
then I tried it on learning science
5:14
specifically with self-regulated higher
5:16
order learning which is my area of
5:18
expertise. This was to see if study mode
5:20
is useful even if you are an expert at
5:22
that subject. This is the type of
5:24
information that there isn't really a
5:26
wealth of training data widely available
5:28
on the internet. Uh you'd really be
5:30
getting most of your information from
5:32
up-to-date recent research papers. In
5:34
this final test, I assume that if you're
5:35
already an expert, you're already
5:37
thinking at that higher order. So I
5:39
didn't simulate a lower order, more
5:41
passive learner. So that's how I tested
5:43
it. And now on to my findings. Let's
5:45
start with what it did really well.
5:47
First of all, the accuracy. I don't have
5:49
enough expertise around AI and LLM and
5:51
transformer architecture to know if
5:52
there was some subtle hallucination
5:54
there. But at least for medicine and
5:56
learning science, there was no
5:57
hallucination or the information was
5:59
very accurate across the entire study
6:01
session. So, especially if you are
6:03
studying a curriculum or like an
6:05
established field where you know that
6:06
there's lots of information available on
6:08
this online, um I would feel that the
6:11
information provided is pretty accurate.
6:13
The second part is that it does seem to
6:14
be a lot more interactive. Normally,
6:17
when you're using chatbt, you're asking
6:18
something and it will just just give you
6:20
the answer. Like, it's just responding
6:21
to you. In study mode, it does a better
6:24
job at asking questions, having
6:26
follow-up questions, kind of guiding
6:28
you, and walking you through that
6:29
process. The answers that it gives is
6:31
definitely more sequential. It's
6:33
designed in a way for you to logically
6:34
be able to follow along. It's also
6:36
pretty good at testing you. So with
6:38
normal chatbt, you have to do a little
6:40
bit of prompt engineering to give it
6:41
some guidelines on how to test you and
6:43
what types of questions to ask and how
6:45
to sort of figure out the right level.
6:47
In study mode, you can basically just
6:49
say, I want you to test me on what we're
6:51
learning. And it will do a really good
6:52
job at generating some meaningful,
6:54
relevant, targeted test questions. So
6:56
straight off the bat, just with those
6:57
improvements, it does seem to be a
6:59
direct upgrade from normal chatbt for
7:01
studying and learning. So if you're
7:02
using chatbt for learning, just flick it
7:05
on to study mode. it just seems to be
7:07
better. And another benefit which I
7:09
think a lot of people will be able to
7:10
resonate with is that you can feel free
7:12
to be dumb when you're learning through
7:15
study mode. You don't have to worry
7:17
about someone judging you for asking a
7:18
stupid question. You can be as dumb as
7:20
you want with chatbt and it's not
7:22
judging you as far as I'm aware. And it
7:25
may seem like a simple thing, but having
7:26
that psychological safety to just ask
7:29
and explore your own thoughts and your
7:31
confusions very freely without fear of
7:33
judgment is an important thing for
7:35
learning. So overall, I give study mode
7:37
generally a thumbs up, but there are
7:39
some caveats and I think these caveats
7:41
are crucial if you want to make the most
7:43
out of this tool. So let's talk about
7:45
the issues because there are some
7:47
issues. First one I'll get out of the
7:49
way uh is one that OpenAI themselves
7:52
have admitted as a current limitation,
7:54
which is that it's not very good at
7:56
teaching for your exact level. It's
7:59
generally just not very good at figuring
8:01
out why you might be confused with
8:02
something. If you're teaching someone
8:03
face tof face, a good tutor or a good
8:06
teacher is able to explain something and
8:09
then get immediate dynamic feedback. As
8:12
you explain something, you can figure
8:14
out that this person has not understood
8:15
this particular part of the concept. And
8:17
so instead of just finishing off your
8:19
monologue of explanation, you would
8:21
stop, pause, pivot, check the
8:23
understanding. Study mode is not going
8:24
to do that. It's going to give you a
8:26
pretty decentsized explanation. And if
8:29
you're confused about it, it's up to you
8:31
to be very clear and explicit about
8:33
which part of that is confusing for you.
8:36
For a beginner learner, that's
8:38
incredibly challenging. When you're a
8:40
beginner learning something, you don't
8:41
know what the right knowledge is meant
8:44
to look like. And so, it's hard to see
8:45
if it's right or wrong. All you know is
8:48
that something isn't clicking and it
8:49
feels confusing. So this is a problem
8:51
that open AAI has acknowledged and based
8:54
on what I understand about the AI
8:55
technology, I think that this is a
8:57
problem that they will be able to solve
8:59
eventually. It's probably just more of a
9:00
cost and compute issue at this stage.
9:02
Likewise, the second limitation which I
9:04
definitely felt myself uh that OpenAI
9:06
also acknowledges is the fact that it's
9:09
not really good for uh multimodal types
9:12
of learning. So, for example, if you are
9:15
reading a really textheavy, conceptually
9:17
dense explanation, it's really helpful
9:19
just to have a diagram or an image or a
9:21
flowchart or something like that to
9:23
anchor your thoughts on and it makes it
9:25
much easier to understand. At the
9:26
moment, it's pretty much all textbased.
9:29
And you can get it to generate an image
9:31
for you to help you understand it, but I
9:33
found that the images weren't
9:34
particularly that useful and I found
9:36
myself reaching for Google images
9:38
instead because you know that those
9:39
images are crafted by experts who really
9:41
know what they're talking about. And so
9:42
again, based on my understanding of the
9:44
underlying technology, uh I think this
9:46
is something that CHWT will be able to
9:48
solve as well eventually. Again,
9:50
probably a cost and compute problem. But
9:51
in the meantime, until they solve that,
9:53
I would definitely recommend uh having
9:55
something like a Google images tab uh
9:57
open next to you so that you can look
9:58
for images when you feel like you don't
10:00
see how something fits together. It's
10:02
going to be a lot faster than just
10:03
trying to piece together the words in
10:04
your head. But now, let's move on to
10:06
what I think may be the bigger issues.
10:09
The main problem that I felt was that
10:12
the interaction of using study mode is
10:15
very very userled. And if you're talking
10:17
about chipt as a general tool, you know,
10:20
it's like that's just how it is. That's
10:21
how tools work. You know, you get more
10:24
out of a tool when you know how to use
10:25
it properly. That's why an expert using
10:27
AI gets better results than an amateur
10:29
using AI. But this becomes really
10:31
problematic when we're talking about
10:32
studying and learning because generally
10:34
people that are learning something for
10:35
the first time are not experts in that
10:38
thing. And so one of the things that a
10:39
great teacher is able to do is they're
10:41
able to teach the student how to think
10:44
about this subject so they can navigate
10:46
their own path and when it's too
10:48
confusing show them the path to walk in
10:51
the first place. Now, the way that
10:52
chatbt tries to combat that is at the
10:56
end of each response, it gives you like
10:58
a suggestion. Like it'll explain
10:59
something and it'll say, "Hey, do you
11:01
want me to break that down in this way
11:02
for you instead to make it easier to
11:05
understand?" Which is a great attempt,
11:07
but in my experience, I actually found
11:08
that it didn't really help very much
11:11
because it didn't target the reason I
11:14
was actually confused. And this is
11:16
actually about a very core principle of
11:19
learning which is that learning is a
11:21
messy chaotic uh very omnidirectional
11:25
process and understanding a single
11:27
concept correctly. There's actually
11:29
hundreds maybe thousands of tiny tiny
11:31
subconcepts that you have to understand.
11:33
So, if you're talking about learning an
11:34
entire topic, it's very hard to read a
11:37
big explanation and then point
11:39
specifically to which sub subconcept you
11:43
didn't quite understand or you may have
11:45
misinterpreted that is leading you to
11:47
not understand the the big picture. And
11:49
what that means is that your experience
11:51
is that you are learning stuff. You are
11:54
understanding what it's explaining to
11:55
you, but it still isn't clicking. The
11:58
big picture of it isn't coming together,
12:00
and we're remaining sort of confused
12:02
about it. The natural consequence of not
12:04
having it integrated into a big picture
12:06
network is also that our memory of it is
12:08
going to be very short. That's one of
12:09
the great illusions of learning is that
12:12
you feel like you are learning because
12:13
you understand what you're reading at
12:15
that point in time thinking that that's
12:18
going to translate into long-term
12:19
memory, but that's just not how memory
12:20
works. And then after you've read the
12:22
explanation, you feel confused and then
12:24
it will say, "Hey, uh, do you want me to
12:27
explain it to you in this way instead or
12:29
break it down into this kind of table?"
12:30
And you say, "Yeah, okay. that might
12:32
help me. So you say, "Yes, do that for
12:33
me." And it will generate that
12:35
explanation. And again, you'll read it
12:36
and it's much the same. You're still
12:38
kind of confused. And actually, you can
12:40
become even more confused because you're
12:42
being taught more and more information,
12:44
which means there are more and more
12:46
possible places where that confusion is
12:48
arising from. And it gets harder and
12:50
harder to pinpoint where it's really
12:52
stemming from. Now, normally if it's a
12:54
human to human interaction, if someone
12:56
is not understanding something like
12:57
three to four times in a row, you
12:59
wouldn't just keep trying to explain it
13:01
to them like reframing it each time. You
13:03
would have more of a conversation to
13:04
figure out why they are confused and
13:06
then tailor your explanation to
13:08
specifically the part that they're
13:10
misinterpreting. But I noticed wasn't
13:12
doing that after even five or six
13:13
iterations of confusion. So to combat
13:15
this, I tried to get it to test me. I
13:18
instructed Chatbt to test me on my
13:19
knowledge to see where my confusion
13:21
stems from. It generated a pretty good
13:24
question and I answered it in the way
13:25
that a genuinely confused student would.
13:27
Uh, and it was much better at figuring
13:30
out why I was confused, but it was a
13:33
little too
13:35
obedient. If I'm teaching someone and
13:37
they clearly don't understand an
13:39
important core concept, it's really
13:42
important for that person's learning
13:44
that I explore and probe the way that
13:46
they have tried to understand this
13:48
topic. The idea is that most of the
13:50
benefit you get from learning a topic.
13:53
And the way that that topic becomes
13:54
easier for you is not in understanding
13:57
more facts and concepts. It's about
13:59
getting a way to think about that topic.
14:01
You have to create a mental model. And
14:04
it takes a bit of time and effort to
14:05
create that mental model. A great
14:07
teacher is able to facilitate that
14:08
process. They're able to see how they
14:10
are trying to think about a topic and
14:13
then compare that with the way that
14:14
they, the expert, thinks about the
14:16
topic. And I continually responded in a
14:18
way that shows I do not have a basic big
14:22
picture understanding about how to think
14:24
about this topic. But it took a lot of
14:27
very active work to get it to the point
14:30
where it took a step back realize that
14:33
was the issue and then addressed it
14:34
explicitly. If this was a human
14:36
teaching, they would put a mirror up to
14:38
that student and make them reflect on
14:41
the way that they are thinking about it
14:42
and it would be challenging and it would
14:44
be effortful. But after going through
14:46
that process, things would start
14:47
clicking a lot more easily. And at this
14:49
point, I wonder if this is a conflict
14:52
that OpenAI has because CHBT at the end
14:54
of the day is a commercial product. And
14:58
if engaging with Chat GBT makes learning
15:01
feel challenging because you're having
15:03
to do this difficult thinking and it's
15:06
actually making you think about it in a
15:08
certain way, then that could actually
15:11
reduce the amount that people use this
15:12
as a tool. This is what we see
15:14
universally across basically all
15:16
learning which is that effective
15:17
learning is active involves lots of
15:19
thinking and is mentally difficult. Like
15:22
it takes mental effort to do effective
15:24
learning. You cannot bypass that
15:27
biologically. It's actually impossible.
15:29
But the great misinterpretation is that
15:30
people feel that learning is meant to be
15:32
easy. But actually when learning becomes
15:34
easier it's becoming easier because your
15:36
brain is doing less work which means
15:38
it's doing less learning. And so I find
15:40
this really interesting because for
15:41
chatbt to really crack the code on
15:44
making learning more effective and
15:46
efficient for the user, they need to
15:49
navigate that expectation of the learner
15:52
themselves understanding that in order
15:54
to learn effectively, they're going to
15:55
have to do some of the heavy lifting
15:56
inside their own brain. This as a
15:58
self-regulated learning coach, like this
16:00
is one of the big lessons that I make
16:03
sure to teach all of my students, and
16:04
it's a it's a real challenge. So I'm
16:06
interested to know how OpenAI is going
16:07
to handle that. to give you a comparison
16:09
of the difference it makes being a more
16:14
passive learner versus being a more
16:16
engaged active learner that's more
16:18
metacognitive. I spent 30 minutes being
16:21
confused with a specific concept,
16:23
letting chat kind of guide me. When I
16:25
did the same test again with the same
16:28
concept, uh, and instead of just being
16:30
more passive, I ignored its
16:33
recommendations and suggestions, I
16:35
looked inside my own set of knowledge to
16:37
understand why I'm being confused and I
16:39
asked very specific targeted questions
16:41
and I was able to break through that
16:42
confusion in 2 minutes. So the first
16:44
instance 30 minutes of confusion
16:47
constantly learning and reading and
16:49
going back and forth with chatbt. The
16:51
second case 2 minutes basically two
16:54
rounds of back and forth to clarify it.
16:56
And FYI I actually feel that it would
17:00
have been faster for me to learn that
17:02
concept through like a 15minute YouTube
17:04
video instead of that 30 minute back and
17:06
forth interaction with chatbt. So in
17:08
that case it was actually slower than a
17:11
more conventional style of learning. On
17:12
the other hand, uh using a higher order
17:15
thinking framework, it was much faster.
17:17
The answers I was getting were very
17:18
targeted because questions I was asking
17:20
were exactly the questions that I needed
17:23
to break through my own knowledge
17:24
plateaus. And so it's interesting
17:25
because the type of learner I was when I
17:28
was interacting with CHBT made the
17:31
biggest difference than anything. In
17:33
fact, I would actually say that using
17:35
ChachiBT as a active, engaged,
17:39
metacognitive, reflective, higher order
17:41
learner without study mode is vastly
17:45
more effective than using it with study
17:48
mode while being a passive disengaged
17:50
learner. Being a higher order learner
17:52
that knows the right question to ask,
17:54
that knows how to navigate their own
17:55
confusion led to much faster learning. I
17:58
mentioned that I spent about four to 5
18:00
hours on this. probably 80% of that time
18:03
was being the passive learner because it
18:06
was just so slow being stuck in that rut
18:09
of confusion because I just didn't know
18:11
what questions to ask that would get me
18:13
out of that rut and chach wasn't
18:15
figuring it out for me as well. So with
18:17
all that being said, here are my tips on
18:19
getting the most out of study mode.
18:20
First of all, and probably the most
18:22
important tip, I would use it for more
18:23
targeted study. Get to a point in your
18:25
studying where you have specific
18:28
questions or specific areas of confusion
18:30
that you want to work through, then go
18:32
to study mode. It will do an excellent
18:34
job at helping you through that,
18:36
probably faster than a Google search or,
18:38
you know, like reading through the
18:39
textbook and figuring it out the
18:40
oldfashioned way. Second, try to avoid
18:42
getting sucked into the rabbit hole of
18:45
suggestions. If you read a response uh
18:47
and at the end it suggests a way to make
18:49
it easier for you, before you just say
18:51
yes and then commit some more time to
18:53
reading the next response, take a step
18:55
back, evaluate what you've just read,
18:57
and if you're still confused, try to
18:59
understand what specifically about this
19:02
is making it confusing for you and
19:03
respond with detail. The specific points
19:06
of confusion, try to explain and
19:08
articulate in as much detail as you can.
19:10
I found that when I did that, it was
19:12
much better at correcting itself and
19:14
then creating a much more targeted
19:15
explanation. And the third tip, which is
19:17
kind of like the general learning tip
19:18
for all learning that you're ever going
19:20
to do for the rest of your life, don't
19:22
try to make learning easy. Your goal is
19:24
not to make learning easy. In fact, your
19:26
goal is not like the learning at all.
19:27
Your goal is just to have knowledge.
19:30
That's the that's the reason you're
19:31
learning in the first place. You want to
19:33
go from not knowing stuff to just
19:34
knowing stuff. If you could click a
19:36
button for that to happen, that would be
19:37
the ideal scenario. But we can't do
19:39
that. So unfortunately we have to do all
19:41
of this like studying and learning to
19:43
make that happen. When you make learning
19:45
easier for yourself, most of the time
19:48
you are shortcutting the process that
19:51
turns external information into internal
19:54
knowledge. And there are lots of things
19:55
that you can do externally to make it
19:57
easier for yourself. But when it comes
19:59
time to do the hard work of thinking and
20:02
pulling it apart and exploring the
20:04
confusion and making sense of it, these
20:06
are all skills that you need to have.
20:09
You cannot expect any more in the
20:11
external space to make it any easier for
20:14
you. Like it's already so so insanely
20:19
easy to get information. The bottleneck
20:21
is it has been for a long time but
20:23
especially now the bottleneck is whether
20:26
you can take that information and
20:28
actually turn that into viable useful
20:31
connected knowledge. But if you build
20:33
your higher order learning skills and
20:35
you know how to think and learn like I
20:38
teach in all my other videos, then chatd
20:41
study mode is a great upgrade. And I'd
20:43
recommend it for learning anything and
20:44
it's only going to get better over time.
20:46
And if you give it a go and then you
20:48
find that you're still really struggling
20:50
to learn a certain topic, I think that
20:54
that may be a sign to realize that
20:58
uh the issue is probably not with
21:02
chatbt.
21:03
But anyway, it's an exciting space to be
21:05
in and I'd love to hear what your own
21:06
thoughts and experience have been with
21:08
using chatbt study mode. Do you agree or
21:10
disagree with things that I've said?
21:11
Let's have a discussion. Leave your
21:12
comments below. Thanks for watching and
21:14
I'll see you next time.
