I am somewhat skeptical of this.<p>First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement<p>Second, trying to incorporate past grades into their modelling is not a substitute for a randomized trial.<p>Third, the headline engagement number of 90% is for "engaging with the platform, via Module Review or Lesson Quizzes, at least once". I don't know why much of that couldn't just be attributed to novelty. Or even partly a professor with all sorts of enthusiasm for the platform.<p>Fourth, the "full dosage" effectiveness is measured based the final exam scores. Were these exam questions produced independently from the "Phosphor" materials? (e.g. by blinding?) Were they checked for direct overlap with those materials? The 0.7 sigma shift is 3 points on a 24 point exam; if even a few of the questions on that exam were very similar to those materials it could account for almost all of it. This is not clear to me from the manuscript.<p>If this was the case, then it's a question less of "is AI effective" vs. "did the students look at the materials". You could still argue that the AI platform got them to read, but that is a somewhat different statement than the AI helped them learn.
Education research is hard hard hard. Getting clean studies, especially at large sample sizes, is extremely difficult, leading to a lot of ambiguity in results.
Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but <i>after analyzing the exam results</i> they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again.<p>That means their experiment design is partially caused by their results instead of the other way around, which is a bad situation to be in. Their statistical analysis is completely inadequate for dealing with this.<p>And the change in engagement suggests that there's strong selection involved. Their attempt to use midterm scores to control for selection effects is unconvincing. Why not control for whether students used the platform more when there were only multiple-choice questions? Those are the ones who self-selected out of using the AI grader.
It feels to me that the venn diagram between "students that fully engaged with the material" and "students that learned well from the material" is going to basically be a circle for any teaching method.
The tricky question at the forefront of education research is, at least in my mind, trying to thread the needle between effective techniques that students don’t like, ineffective techniques that students do like, and poorly defined techniques that administrators like. And on top of all that student self-reports aren’t actually very reliable indicators of learning progress at all!
Yeah you'd need to compare this method with some classic method as a control. Otherwise "engagement" is just an indirect measure of how much the students studied.
Yeah, calling this an "effect size" is just nonsense, and it is alarming that educational software can get away with such poor statistical practice. I'm hoping this was just a student project.
This is a helpful explanation - am not a researcher so I have little idea how to run an unbiased, meaningful experiment (except that it takes a lot of effort and thought to run one). Useful analysis
I'm not an expert, but how much of this is down to novelty, ie <a href="https://en.wikipedia.org/wiki/Hawthorne_effect" rel="nofollow">https://en.wikipedia.org/wiki/Hawthorne_effect</a> ?<p>(ie changing the environment can lead to short term productivity gains because either participants are aware they are being watch, or it breaks up the monotony and makes people work a bit harder. )
I'm on record saying that a system like this with some extra hardware (i.e. a way for the LLM to have live understanding of the student's paper notebook or handout which are being written in with a plain old pencil) combines the best of both worlds - individual tutoring with approximately zero screen time which scales linearly with the number of students. The role of the teacher or professor then becomes a manager of the student - agentic tutor pairs, a referee when the student and model disagree, etc. and most importantly still being the human teacher you can just talk to in the human education process.<p>I'm convinced this is the future of education - models are there, we need the classroom tech to catch up. The alternative is obvious and quantified in the paper - students just use models to do their work for them and learn nothing.
I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking.<p>Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad.<p><a href="https://remarkable.com/" rel="nofollow">https://remarkable.com/</a>
> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad<p>What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer.<p>What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significant to any kind of development of anything.<p>Are you talking about running a local model??
I feel like an eink b/w device will work better because of the display and lack of distractions like on a full blown iPad. Seems like sink would be similar to a sheet of paper but digital so it could be sent to an LLM or some
Other API. Just my two cents though, I don’t have a super strong opinion about it.
I assume that the reMarkable here would mainly just be used to capture the writing. It could then be examined by an LLM asynchronously.
A 'smart pen' that records the student's writing in some way, maybe? My first thought was a tablet that boots straight into a writing software but students should not be subjected to any amount of latency in their writing.<p>Practically, I think if you want the AI system to have a live view of what the student's doing you're going to have to replace one of either the tablet or the writing instrument. A wearable camera could work as well but there are issues with that.
there was a pen that used special paper to directly record your notes (15-20years ago)... should be possible nowadays to directly transfer this to a connected device and have it feed it to an llm.<p>and after looking it up, it appears they are still available: <a href="https://www.livescribe.com/landingpage/ls3_onenote/" rel="nofollow">https://www.livescribe.com/landingpage/ls3_onenote/</a>
They still exist, along with a bunch of different ones. I don't think it's going to be all that different compared to just writing notes in an e-book or on an iPad though. And for many people who learn in other ways, the iPad or similar is superior because you can copy in pictures, make diagrams, and use other ways of learning all in one spot. For me, honestly, something like OneNote (or especially Obsidian) is awesome, because it's super easy to tie in AI into mark down text.
That was likely what I was thinking of - I have vague memories of seeing an ad for this in Popular Mechanics or Popular Science in the 2000s.
I would add that somewhere in there should be a spaced repetition algorithm.<p>Spaced repetition is very effective, but it's really really clunky to use. My unpopular opinion is that we all have Stockholm syndrome when it comes to creating "cards", and people talk about how valuable creating cards is; but I think it stucks, it takes a lot of time.<p>If AI is already teaching me math (let's say), it would be nice to tell the AI/app "quiz me on this periodically", and then the AI makes up a fresh polynomial to factor (or whatever) and presents that to you according to a spaced repetition algorithm.<p>Behind the scenes, the AI should have access to what has happened the last several times a specific topic has been quized, so the AI can watch to see that certain mistakes are resolved, and the AI might also know better how to correct the user if it has context about previous quizzes of that topic.
But the very act of making and organizing your card deck is part of the SRS! It “sucks” because you get no dopamine hit from a fresh desk, as the reward system is not yet in place.
Again, I really think this is a viewpoint we've talked ourselves into to help us feel better about how cumbersome creating the cards are.<p>I'm willing to grant that there is some value in choosing what to put in the cards, but most of the awkwardness around making cards is UI related. Nobody creates cards on their phone, or while they're walking (AI could do both of these) - people create cards sitting at their computer (like cavemen!) usually clicking through a clunky UI and managing thousands of cards with thousands of clicks. That sucks, and people probably wont realize it sucks until something better comes along.
> Nobody creates cards on their phone, or while they're walking<p>Wait, when are you doing it then? No wonder you think it sucks! Adopt some modern tools, yo. Use Anki, or vibe code your own app.
> Use Anki<p>Anki is great for studying, but the card creation experience sucks. To be specific: I found creating any custom card type immediately dropped me into the bowels of CSS. It felt like writing HTML by hand.<p>Is there any facility for re-using shared pieces?<p>I felt like it needed a static site generator type tool to move up a layer of abstraction and reduce the copying of chunks into my card. Is there one? Please mention if so.
I really never thought it would come up in a HN thread but I’m actually working on a modified version of Anki as a personal project (not quite ready yet though, but will open source probably in a few weeks) where improving the editing/curating/creating experience is a big focus. I’m trying to make it markdown-based too.<p>Just to pick your brain real fast, when creating new cards (from scratch?) what would a better experience look like for you?
Write a tool to reduce the friction.<p>I use an Emacs based SRS tool. I have a capture template to quickly make a card.<p>For anything tedious, it's critical to reduce the friction!
> I found creating any custom card type immediately dropped me into the bowels of CSS.<p>Ah, LaTeX syndrome.
It's rather nerdy, but QuickTurtle9 released mdfc on GitHub, <a href="https://github.com/bttger/markdown-flashcards" rel="nofollow">https://github.com/bttger/markdown-flashcards</a> a couple of years ago<p>Using that as the input file format standard, AI can generate what you're looking for, Android app, Webapp, iOS app, pdf.
> Spaced repetition is very effective, but it's really really clunky to use<p>Spaced repetition is just reviewing the same material periodically. It doesn't have to be a complicated system.<p>We already know how to learn and educate: spaced repetition (periodic review), and retrieval practice (frequent testing). This is how school used to be fore centuries; it's not sexy but it's effective.
Today I saw a demo of Remarkable turned into Voldemort's diary from Harry Potter - you write to it, and it writes back, in handwriting.
The title is misleading. This isn't an AI tutor so much as a practice quiz platform with an AI autograder.<p>> constructed-response questions (CRQ) are graded by Claude Sonnet 4.6 against
instructor-defined, question-specific rubric criteria<p>> Crucially, LLMs make it feasible to grade formative CRQ against rubric criteria at scale, a capability that appears pedagogically significant rather than merely convenient.<p>They specifically call out that the "RAG chat assistant" part of Phosphor (the platform) wasn't used much.<p>I commend the effort here, but I don't think these results are particularly noteworthy. The conclusion is essentially that people who do practice quizzes will do better on exams.
This is exciting because the effect size is so large. But as the author's acknowledged, selection bias is nearly impossible to control for in this non-randomized study:<p>> and lacks randomized controls. Self-selection is the central threat: students who complete more quizzes
may be more motivated or higher-performing generally<p>But this is still a strong result. I'm excited to see more in this space.
Yes! Very exciting to see this.<p>Bloom's Two Sigma Opportunity suggests that there's another SD improvement available: <a href="https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem" rel="nofollow">https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem</a>
I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine.<p>Earlier I used Claude by giving it the course material and asking it to generate me exercises (our cpurse work went way over my head) and yeah i learned to differentiate a gradient or Jacobian, but it was very shallow - I knew the formulas, but not what they meant or how to apple them correctly. After I just filled glaring holes I had in Univariate Calculus by readong and doing, I actually started to understand something.<p>Lon story short, in my experience Learning with LLM’s is ok with very unfamiliar material that is not too complex (there’s obvious problems of LLM’s themselves being pretty ghastly with maths sometimes), but at least it os not better than the traditional method of just putting your nose on the grimd stone.
The article explicitly calls out selection bias (this is entirely based on 90% that opted into using the tutor, there was no control group), I wish the headline did as well. "Engaged students score 0.71 - 1.30 SD better in tests" sounds like a much simpler explanation.
I used to TA a graduate level CS math class at Georgia Tech. We regularly saw that the students who self-organized study groups did <i>dramatically</i> better in the course than average. One semester they told us to put everyone in study groups to see if it helped. The effect disappeared. Turns out that it was the self-selection of the most engaged students into a small group that mattered, not the study group itself.
"Full dosage of the Phosphor material is associated with an increase in final exam performance."<p>This sentence is accurate, but inevitably leads to the confusion you see in these comments.
Conflicted about this study. On one hand, LLMs have been incredible for my personal learnings of new concepts.<p>On the other, I'm sceptical of that it'll have "strong benefits" at scale; I'd be more in favor if the wording was "some"/"moderate". I reckon self-selection plays a huge part, as mentioned in the "Limitations" section of the paper.<p>I'd also caution against attaching the tool to grading. That means students have to put more effort into the course, which increases the chances that they will use LLMs to save time rather than make the investment.
> LLMs have been incredible for my personal learnings of new concepts.<p>Mind if I ask what did you learn and how you're using it?<p>The reason I'm asking is that I repeatedly felt excitement only to realize down the line that the explanations didn't actually translate into practical skills. I'm not sure it's even an AI problem, it's a "doing versus reading" problem. Same as with reading a pop-science article and thinking to myself that I learned something about physics or medicine or mathematics.
Wow, putting effort into training material, thoughtfully designing it, and relating the material to the final exam, will increase performance on said exam.
So much AI so much wow.
Like seriously. Most university statistics courses suck big time, so literally any effort put into them will Improve the field.
I’m happy the authors want to improve education but they don’t seem to understand that preparing questionare style material is a confounding factor which could very well explain the better performance too… instead of cramming AI into the next thing. I’m generally opposed to AI on basic textbooks. You don’t want hallucinations imprinted on students who have no idea and can’t judge the quality of the generated text. Some things require effort, reading intro to statistics is one of them, and it’s for a reason, the effort IS the learning
Do you have a larger study planned for the Fall? It definitely seems promising.<p>I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature.<p>PS - I'd say this qualifies for Show HN, too!<p>Do you
Interesting article, wonder where we're going with this though, I find it's very difficult to keep LLMs on track and critical enough to be useful.<p>Just want to say that:<p>>In our deployment, student-reported reading completion
baselines for MATH 010 were approximately 15%, with instructors estimating 10%. Individual student
reports of reading compliance ranged from "literally no one does that" to "is this being recorded?"<p>is hilarious
> the platform was adopted by 90.2% of enrolled students<p>Then it is "effectiveness", not "efficacy". Prefer simpler and more specific words when possible, to reduce effort for the reader.
Even if the research is flawed I'm happy they are trying this. They are taking advantage of LLMs to have less rigid tests and also give feedback.<p>I think there is more potential applications possible with combining LLMs with reference/text books.
Like how about an assistant that points you to the correct books/chapter/paragraph for the concept you need to understand better for a project you are working on? Or clarify any confusion you are having?<p>Like a human tutor but infinitely patient and non-judgy + search engine.
While there's some skepticism in the thread, I'm not particularly surprised if this is true. Children who can get <i>human</i> tutoring do a lot better. An LLM that can answer questions and patiently explain likely offers some benefit.<p>What creeps me out about bringing LLM into early education is that it's a period where kids learn to socialize and cope with problems, and I do worry about forming substitute relationships with chatbots that are engineered for sycophancy / enablement. But I guess that's a problem either way, because almost every student will try an LLM at some point.
Shocking that a well executed AI tutor improves outcomes.<p>Hasn't computer assisted interactive learning already been proven for years? Why does there seem to be so much skepticism about enhancing it with AI?<p>Is this just something like, astoundingly slow adoption or poor execution? Being held back by paper textbook makers? Teachers unions dragging their feet?<p>How can interactive AI driven individually paced learning _not_ be obviously dramatically more effective?
Selection effects are extremely important in education. Dartmouth students have already had a large selection effect. If you try to apply this more broadly then it might not work.<p>Motivation is also a huge part of the problem. I'm wondering if the novelty of the AI tutoring gets more people to try it and whether it would wear off?<p>It's surprising to me that many students at Dartmouth don't read the textbook. You'd think college admissions would select for that?<p>It seems promising but, as they say, more research needed.
Lots of people in education will happily tell you how the past 15 years of tech integration has been a net negative.<p>There ARE technologies that have improved things, but so much high-cost useless tech has been shoved into every level of education that many educators are incredibly leery of new tech.<p>The issue is that while the underlying technology is useful, the way it gets integrated is frequently not. An administrator cuts a deal for a product they never have to use to an ed-tech giant for a huge amount. Because the ink is dry and a huge sum of money has been spent admins pressure educators to use the technology as much as possible regardless of outcome.<p>In that context it makes a lot more sense why there is pushback and FUD among educators.
I had a chance to use Google Classroom for a non-profit I was volunteering with, and wow it sucks. If that is what teachers and students have to contend with, yeah, I'd push back against any and all tech forced on me as well. It's all well intentioned, but the road to hell is paved with them.
its like anything else. benifits students that are already motivated to learn.<p>very few are actually motivated to learn and are just there to get a job or its just next thing that they have to do in life.
This is super, but students will have access to AI during the test in real life, so it's ironically less realistic to remove it (thinking of the "... GPT-4 actually harmed subsequent performance by 17% when the tool was removed ..." part).<p>I'm more curious how students perform on the test with vs. without AI.
Interesting, congrats.<p>Are you planning on opening access to Phosphor?
In mice!<p>Jk, but the skepticism is inevitable. I think we can be dubious about how AI mobilizes global capital while also appreciating tutoring as one of its best targeted use cases.
Honestly whether or not this was effective seems less important to me than the adoption numbers.<p>Text book reading in this course was 10-15% at baseline ... but this AI thing got 90% voluntary usage ungraded.<p>Even if its worse per-hour than a textbook, you're now teaching 6x as many students _something_ instead of teaching a small minority everything.<p>So really it just becomes an optimization problem at that point because most students are at least in the funnel/in the running to learn something.<p>The paper kind of proves this itself ... they tweaked the quize formats mid-semester and where able to iterate which you can't do on a textbook that nobody opens in the first place
Very nicely typeset.
A lot of pessimism in the comments, but I am just happy that we are seeing some work towards bridging the 2 Sigma gap for regular education vs. elite private tutoring. I can't imagine that people assume it's the physical presence of the tutor that is making the difference, it has to come down to the personalisation and expertise which is exactly what AI can provide in a form. And yea it might not be "there" yet. But if we don't start trying and studying then it'll never get there.
Tell me if I am oversimplifying, but I never understood the noise about the two sigma problem. Like, of course if you have a private tutor to immediately answer any question that pops into your head at the immediate moment you get confused, you are going to learn vastly more efficiently than in a large classroom where once you get confused you are likely to stay confused. To say nothing of how the pace will likely either drag way behind what you'd like, or accelerate too fast ahead of it.<p>The environment is just obviously two sigma better. This just... seems obvious to me? In the same way that I will get stronger much faster if I have a physical trainer to tell me exactly what I am doing wrong when I do it? And it seems obviously unsolvable other than by getting everyone a private tutor (or AI..?).<p>Asking from a place of curiosity.
Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
Wikipedia doesn’t make much money but is still helpful. LLMs don’t need to make a whole bunch of money to be helpful.
I don't want to learn from hallucinations where it will change its answers based on me questioning their teachings. I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions for example is not a wise decision.
Curious whether you were just bare asking it questions, or whether you provided it with lessons one by one with instruction that the lesson is the baseline truth etc
Are we talking about human teachers or LLMs here?
Not sure if its education, but there is huge money in the college admissions process, e.g., SAT prep.
[flagged]
[dead]
[flagged]