In the AI era, tests are going to have to be in person and hand written.<p>I have written an article on how I have adjusted my classes to the situation:<p><a href="https://htmx.org/essays/universities-and-ai/" rel="nofollow">https://htmx.org/essays/universities-and-ai/</a><p>Ironically, I think the AI era may make university degrees a better signal of the intellectual abilities of students due to the presence of pre-computer infrastructure like large lecture halls, industrial-scale copiers, etc.
> In the AI era, tests are going to have to be in person and hand written.<p>Forcing hand written should really not be necessary. It would be very cheap in terms of old computing hardware to set up a test room with old desktop PCs that have wired only NICs (with a network connection that goes to a switch in the same room with no uplink, connected to a decent size laser printer only), running something like lubuntu and libreoffice writer as a basic word processor.<p>Let people at least type their essay with the standard features of a word processor as usable as MS Word 2000 or better.
I’m 30 and “we can’t do tests in paper” seems _insane_. Just how metastatic has ed tech been in what, 9 years since my undergrad?
More expensive than you'd think, but I am pushing for something like that.
Can you fit a decent LLM on a thumbdrive?
Nobody is going to do that.
There are limits to what you can assess on timed assessments, and there are students whose performance on such assessments is not a good signal of their intellectual ability.<p>In addition, at many institutions such tests are given infrequently and can be worth a significant component of a student's overall grade, increasing both student stress levels and the tendency for such assessments to measure short term knowledge students have obtained by cramming, not more meaningful longer-term knowledge gains.<p>I see you're giving quizzes every three weeks, which is better than twice a semester, but still not what I would consider an ideal cadence. In my course weekly computer-based quizzes comprise 70% of a student's grade, but that's supported by a significant institutional investment in high-frequency computer-based testing: <a href="https://cbtf.illinois.edu/" rel="nofollow">https://cbtf.illinois.edu/</a>.
> there are students whose performance on such assessments is not a good signal of their intellectual ability<p>Is there a form of assessment that is a good signal of the intellectual ability of <i>all</i> students?
I agree and would like to move towards a customized computer setup like you mention. A friend at Berkeley manages a similar setup. Unfortunately Montana State is too small to have set one up yet.
so let say you give the students a pop-quiz. is that not acceptable anymore because some students don't do well when surprised?
<i>take-home, closed-book type</i><p>What an oxymoron. I agree with the others here that AI isn't the problem.
My favorite exams (as a ugrad for classics, and in grad school to advance to candidacy in CS) were in person, hand written, open book.<p>We had lots of time, and a fair idea of the range of questions. It rewards actuality mastering the material vs memorizing it.<p>For the CS exam some people brought more books than they could physically carry, I don’t think it helped them much.
I've had such exams. It was the honor system. The idea is that a typical exam is too short to evaluate the student's knowledge and a belief that fast students shouldn't have an advantage.
How long is too short? Each exam in my BSc Applied Physics final (1977, Exeter Uni.) was three hours and we had similar exams in each of the preceding years to weed out those who weren't keeping up. I'm pretty sure that having more time would not have helped the weaker students get through the Quantum Mechanics exam.<p>In addition I had to defend the report (120 pages of typescript and charts) of my final year project to my supervisor and another senior academic. And it was clear that they had actually read it.<p>All those exams were open note; anything in your own hand or a copy of a lecture handout was permitted. Again the weaker students would not have been helped by more time because they hadn't understood that you have to have enough familiarity with your notes to be able find the right information. Some brought in 50 litre rucksacks stuffed with ring binders and the noise of them furiously leafing through was enough for the invigilators to warn them to make less noise or risk being ejected.<p>In Norway it is typical that an exam of similar standard allows five hours.
I really don't see how it would cost too much to pay TAs some more proctor hours.
Fast students are smarter. Why avoid grading on that?
Define “smarter” —- already a vague and overloaded term.<p>And then consider whether the point of the class is to test smarts, or something else.<p>I’d expect that’s not the intent of most undergraduate degrees.
Are they? Or do they just have superior recall? Or maybe lack test-taking anxiety? Or write or type quicker or...?<p>Lots of reasons a slow student can be just as smart or smarter than a fast one.
Those are all proxies for "is smarter". They have better memories, perform better under pressure, etc. Universities are meant to prepare students for the real world where these things matter.
Care to explain?
See my other comment. I want my doctor/plumber/etc to be able to recall faster, type faster, work better under pressure. If you're better at those things you should get better grades and be paid more.
There's a way. But if your professor is confident AI won't help you too much then it's a very hard test
The craziest part is that a game theory expert can't see the problem here!
The other day some sucker told me, "don't throw that trash over there it's littering" and I told that sucker "there's no way anyone could enforce it" lol.<p>Some people get up in their jimmies about this but if they don't want me doing something they should make it impossible to do it<p></bait>
His research is in Game Theory. He should have realized that, in a situation where all competitors are (possibly) using LLMs, the game theoretic optimal choice is to use LLMs.
Game Theory seems sort of useless in the real world because people are not rational players, and the real challenge is in getting an accurate model of their behavior. The honor system would work probably fine in a tiny close-knit liberal arts college, while it would obviously wouldn't in a place where the degree itself is the target.
That depends on the reward function. Should society reward credentials or skill?
Credentials depend on the institution not just the final degree.<p>As such a public hard line stance is beneficial here.
"Should" is one thing, reality is another
"take home, closed book"<p>This is a trap. I understand they've done this in the past, but profs are paranoid now.<p>I don't believe he's 100% correct on each incident of fraud and he's going to ruin students lives because of it.
> He has conclusive evidence that at least 50 students cheated on the March midterm exam, making it the biggest known scandal at Brown and in the entire Ivy League<p>I'd like a citation for this being the "biggest known scandal" in the "entire Ivy League". Frequently such situations are kept somewhat quiet, for a variety of reasons. But fifty students is not a large number in courses that can enroll hundreds or up to a thousand students.
Ivy League classes usually aren't as large as their state school counterparts. At <i>most</i> you might get around a hundred students in a large lecture course; this is rare by design.<p>You can crunch the numbers on this to verify for yourself – most of these have a population of 6000 undergrads or so. 1/6 of all of the undergraduates need to take a single course for your "up to a thousand" to be true.<p>I agree that a citation would be nice, but the number of students is indeed large for Ivy League courses (as per the article, in this econ course, he had 86 students for a class that usually only has 30).
> “…if we want to preserve the future of higher education”<p>He understands the stakes here. If a university degree becomes useless, then what?
When you're a student in a competitive program at a top university, graded on a curve, and you know your fellow classmates are cheating with AI, you have little choice but to do the same. Especially when jobs for new grads are harder to come by and there's more pressure to also go above and beyond with internships and side projects during your time in school. There's no way to compete without cheating.
After retiring at 65 from a university teaching and research science career (all pre-AI), I went back to teaching, but this time teaching high school science, mostly AP STEM courses at an A-ranked public high school. The cheating/AI problem is now a crisis greater than COVID. My experience: very few students in advanced and AP classes do not cheat — largely for the reasons given above — and it takes enormous resourcefulness on the teacher's part to design coursework and examinations in which cheating through AI is not an issue. Many teachers I know have all but given up — the cost and effort required to circumvent cheating are simply too great given the already sky-high demands on teachers' time and energy. And school administrations are little help, due to thoughtless and enthusiastic reliance on software at every level. In some ways they are part of the problem. I don't know what the situation is in schools outside the US. But here it had become an arms race.<p>[Edit: typos]
Personally I believe AI has made exams and high stakes testing unworkable. Even before AI I would argue teaching to the test made high stakes testing unworkable. How grades are assigned IMO will be more like how employees are evaluated in the workplace: some metrics, some oral exams, some peer feedback, but mostly on what they produced.
Yes, oral exams, content created in plain sight, project-based activities, all of these can provide a true appraisal of student understanding. But these approaches, although highly rewarding for both student and teacher, are extremely time-consuming. They also run counter to the priorities of the district, which are forever and always: student achievement on standardized tests accomplished with minimal teaching personnel. The only ones benefiting here are the corporations providing and grading the tests. To a large extent it is a sham. More importantly, IRL science is not a multiple choice test. As you state above, whether you work in industry or academia, your value is what you can produce (usually by plain hard work), what problems you can solve, your imagination, creativity, what you can create yourself or with a group. But from what I've seen thus far: AI has little place in education.
> you have little choice<p>I personally disagree with that very hard. Deontology begins at home.
The Lance Armstrong defense
In his generation, only cheating cyclists could stay in teams. He was the one who created the situation, but in fact, cyclists had two choices - stop being cyclist or cheat.
Damn that's crazy. Guess the take home test is dead now.<p>I never understood this behavior from undergrads though, you're paying so much for an education and then you just skip the education part? Why bother?
Because for many a college degree is a pure formality to land a job.<p>My first job out of college, I worked with veterans at the company who all got in with a HS diploma. Now you realistically need a masters degree to be competitive, for no other reason than that where I live (Norway) most applicants have a 5-year masters degree. It is basically academic inflation.<p>The worst part is when you start working, and indeed discover that this is a job you could have done just fine straight out of HS.
A master's in CS can teach you interesting and very useful things, like how OS kernels, distributed systems, networks, and microprocessors work. A master's in EE will teach you things like signal processing and analog circuit design as well. Knowing these things helps you to design, build, and evaluate systems that are reliable and efficient.<p>A master's in science helps you understand how the physical world works and how to reason quantitatively as well as qualitatively. A master's in humanities gives you knowledge and understanding of human culture, such as literature and the arts, and history - subjects that can be deeply enriching and can provide insights that transcend disciplines. A master's in social science will teach you about how humans behave in groups and how they interact with their environment, and about statistical analysis.<p>Writing a master's thesis will also teach you a lot and make you a better writer - if you actually write it yourself and don't rely on AI.<p>Any of these degrees will certainly qualify you to be a more interesting, knowledgeable, and insightful barista or Uber driver.
Don't assume that TrackerFF is doing the kind of job that requires higher education.<p>It's also dangerous to assume that higher education is for everyone. (Although I agree the opportunity needs to be there for anyone who wants to try it.) Some people just want to get on with their life after high school. (I raise my children assuming that they will go to college, and if they want to seek an alternate way through life, I will support it.)
There should be no reason you have to jump to a master’s for that. A bachelor’s in CS or EE would be a joke if it didn’t (doesn’t) cover those things. Arguably, even the current 4 year bachelor’s is a waste compared to a focused 2 year program: looking at my college’s requirements, many classes are wasted. Business majors taking a physical science with lab component, entry level English classes being taught by a TA that doesn’t speak English natively, etc.
I guess it depends on the program, but at my university an undergrad EE major, even though it had more units than any other major, didn't get to the best and most interesting stuff (perhaps because engineering majors also had to learn about things other than engineering, which seems like a pretty good idea.) Personally I wish more CS grads (including many people I worked with) had a better understanding of compilers, programming languages, databases, operating systems, distributed systems, networks, and computer architecture, as well as applications programming and interaction design. It's hard to get all of that while working at a single job, but readily achievable at a university, and an extra year of coursework really helps.<p>Business majors <i>should</i> take physical science courses with a lab component! How else are they going to learn anything about reality?<p>But there is no excuse for bad teaching, anywhere (especially given how insanely competitive faculty positions are - even crummy adjunct and lecturer positions.)<p>Unfortunately research universities prioritize fundraising > research > grad students > undergraduate teaching. And sometimes grad students are selected to teach based on financial need rather than interest or ability.
[dead]
>you're paying so much for an education and then you just skip the education part? Why bother?<p>Because you are viewing the motivation of college wrong for most people. For most people, the purpose of college is to get piece of paper that will open up higher salary opportunities. Ergo, they are just doing whatever required to get said piece of paper with least amount of effort.<p>Until degrees, in particular, degrees from well-regarded universities stop being that method, this behavior will continue.
> For most people, the purpose of college is to get piece of paper that will open up higher salary opportunities<p>More true at an Ivy than anywhere else.<p>> This year, the economist decided that both the midterm and the final exams for his course would be of the take-home, closed-book type (there is a certain tradition of this at Ivy League schools).<p>If this guy thinks AI is motivating his previously guiless student body to start cheating on these tests, rather than simply changing the way they are cheating, he's been sniffing too many of his own farts.
<i>Until degrees, in particular, degrees from well-regarded universities stop being that method, this behavior will continue.</i><p>This is already starting to happen, at least for software engineering positions. There have been plenty of stories of candidates with degrees from prestigious institutions failing to answer the simplest of questions correctly. FizzBuzz is a famous example, but there are many others.
Sadly this is true. Another take is that if you don’t use AI but everybody outperforms you on exams using AI at some point you’re forced into it as well.
That motivation isn't necessarily inherent in the attendees though. That has been formed by corporations increasingly placing pressure on universities to be their personal training grounds, without any actual investment. Corporations don't want to train anymore. They want universities and other companies to do their training for them.<p>It's why we're seeing the death of the liberal arts majors. It's sad, because usually the smartest and most creative people I've worked with in the field of engineering and software have been liberal arts majors. But corporations don't want intelligent people. They want people who have been molded to whatever the soup du jour is.
> That [motivation] has been formed by corporations increasingly placing pressure on universities to be their personal training grounds, without any actual investment. Corporations don't want to train anymore. They want universities and other companies to do their training for them.<p>I don't think so.<p>The problem rather is that corporations very often want some very different knowledge of employees than what universities teach to the students.<p>If what the universities teach was very important for the job, applicants who have not invested serious effort into getting a deep understanding of the topics of the courses would nearly all fail in the job interviews.<p>The problem rather is that for many jobs the knowledge that you could have gotten from the university typically does not matter, and thus investing minimal effort into the courses <i>does not</i> get you rejected in a job interview.
If only they could communicate with each other and explain their reasoning.<p>Corporation A "Hi University, here is what we hire highly paid people to do, and what we need to improve as a company."<p>University B, "Hi Corp A, here is our educational mandate to create well-rounded, highly educated people, we can probably fit your needs into the curriculum in the last couple of semesters, let's work together to make sure you have good employees and we have people who aren't struggling to pay back loans because they have an engineering degree but can't make more than 50K at a dead end job."
* I am in the state of not knowing about something<p>* This is brought to my attention by an exam question<p>* I have an oracle in the form of a textbook, an LLM, the internet, or all of the above<p>Which action is skipping the education: looking up the answer, or not looking up the answer?
Apparently some students aren't actually interested in learning and view the diploma as a meal ticket rather than a meaningful credential. Or perhaps the university is just seen as a networking opportunity.<p>If students don't want to be there in the first place and/or don't see any value in learning, it is unsurprising that they'd take the easy way out. Or maybe they cheated their way into Brown and are just continuing.<p>But I was always interested in learning, and understood that cheating was a method of learning avoidance. Why waste the amazing learning resources - faculty, teaching assistants, courses, labs, libraries, studios, rehearsal spaces, interesting speakers, arts and culture events, computing facilities, maker spaces, etc. - that are available at a place like Brown?
When it costs a lot of money, the failure itself costs a lot of money. And you cant afford it. Because failure means you paid a lot of money for nothing.<p>An expensive education comes with higher temptation to cheat.
i mean, even if you are truly there for learning, doesn't it make sense in a low risk setting to try and boost your grade? it's different if you're cheating on your homework or other learning, but there isn't much learning left to do on the exam, that's for the grade
> then you just skip the education part? Why bother?<p>You've never put anything off in your life or taken the easy short term route? Come on this is not difficult to understand. You aren't different either.
Because it's an important aid to getting to a high-paying job in the US, not just a means to learn.<p>One need only look at the resume filtering process, a once manual bias that has now been codified into algorithmic bias with AI. A degree from a good school boosts your chances immensely, and other facets such as coursework don't matter much.<p>If you have ever seen someone filter applicant resumes, you will understand instantly. There are too many, you have to filter them <i>somehow</i> and the allure is irresistible.
They’ve been taught that <i>not</i> having the piece of paper will keep them from having even a menial job,<p>so you have a huge population of people bullshitting their way through to the piece of paper.
These articles consistently fail to acknowledge students were cheating in large numbers prior to these AI tools being available.<p>It was certainly not difficult to cheat at a "closed book" take home exam before.
Different magnitude of cheating altogether
The whole discussion consistently fails to acknowledge that, in a day where we have a Supreme Court Justice who cannot define "woman", education devolves into anarchy.<p>Surf the chaos, bro.
Administration needs to eschew "technology" and demand analog solutions: hand written exams in proctored rooms, no devices out in the classroom, no take home work, etc.
Ensuring integrity definitely requires in person proctored exam centers. It does not require hand written exams.<p>Hand written exams are either very labor intensive to grade or are confined to multiple choice, so either inflationary to cost of education or inauthentic / inaccurate representation of most knowledge and skills.<p>The best answer, which enables authentic meaningful high integrity assessment that is also unit cost efficient is to have testing center facilities with institution supplied devices and well trained proctors.<p>This way instructors can assess students in ways that are relevant and authentic to the subject matter while ensuring the assessments are accurate, consistent, fair and actually reflect the students abilities.
Time for hand written essays again. That way, at least if they do use AI, they will have had to process some of the content a bit more.
The problem isn't AI, it's that you gave a take-home exam expected no one to cheat.
> This year, the economist decided that both the midterm and the final exams for his course would be of the take-home, closed-book type (there is a certain tradition of this at Ivy League schools).<p>These news articles are just tiresome at this point. Obviously folks cheated previously, obviously it's easier now, obviously the answer has been to not have take homes all along.
As of now chatgpt subsidies its consumer subscription-I wonder if cheating on exams will be still promiment once students are forced to pay $30 a month<p>Since students are notorious for being cheap
the professor has all the power in the classroom. If you don't want cheating, define better conditions for the exam. You allowed a take-home exam which means students are able to use any and all resources.
> You allowed a take-home exam which means students are able to use any and all resources.<p>It was a closed-book exam. The professor shouldn’t have to hold students’ hands for them to act with integrity, they are all adults.<p>In this particular class, the professor made the final exam in-person, and didn’t count the take-home midterm because the score distribution wasn’t consistent between the two exams. I think that’s a reasonable approach, but it’s kind of sad that it was necessary
The thought of a closed book take home exam really made me laugh. They also mentioned Princeton hasn’t had professors in the room for exams since the 1890s… They just have a code of honor and rely on other students to report cheaters??<p>Ivy league is such a scam, in so many different ways.
They're going to have change everything so use of an AI assistant doesn't matter because once they graduate they're just going to continue using it anyway.<p>If it's a math for finance course then some kind of model building for the midterm and being marked on the quality of the model or something. If AI becomes so good that it always chooses the best fitting model and requires no numerical optimization then they will have to change the courses to be more like UChicago where it's primarily undergrad directed research but AI assisted.
Why teach kids how to read, they can just take a picture of whatever text they want and have AI say it aloud.
The challenge I think is that students then struggle because they used AI throughout the semester and didn't actually learn. The proper response would be to be strict and fail students that don't perform to a satisfactory level, but this messes with the funding incentives.<p>You can only lead a horse to water, you can't make it drink. Maybe a student's sincerity should play a larger role in the admission process, maybe with a sharp expense curve such that students judged to be more sincere have to pay less tuition. It is an inherently subjective evaluation though.<p>Edit: I completely misread your comment. Asking students to build a model is not a finance class anymore.
This is a dumb take. It's like not teaching kids 1 + 1 because a calculator can do it for them.