It just goes to show that when AIs are relied upon to recall information, they may hallucinate.<p>It lends credence to the theory that a <i>smaller</i> model <i>without</i> the ghost of information past, could do better with ability to search the web, although this is heavily reliant on the effectiveness of the search engine itself.
I can't decide if the report's writing ("One number, followed" & "Quoted by all five. Sourced by none.") is being lazy or ironic?
I know Perplexity isn't often talked about anymore but their design of every sentence needing to be able to be referenced back to an actual source (like an academic peer reviewed paper) seems like a super power for trust.
They're referenced here and don't appear to score much better, assuming this page isn't itself a parody and just zombie stats themselves.
My initial experience with Perplexity was it citing AI blog spam as a primary source so consistently I immediately dropped my subscription. (this was over a year ago, I haven't re-evaluated them since)
I’ve added to my system prompt in perplexity “ignore unreliable sources” and “make sure claims are supported by multiple sources” and that seems to help.<p>Of course for anything of consequence I’d go and actually read the sources (but for plenty of every day stuff it’s fine). On the free plan btw
Is it up to Perplexity to determine what is AI blog spam & what is correct. Versus what exists online?
I'm not sure what the question means.<p>Whether Perplexity is useful to someone, and how useful, will depend on how effective it is in giving answers that are (1) correct as often as possible and (2) checkable if they're sometimes wrong.<p>If Perplexity's answers often treat AI-generated blogspam as reliable then that will probably make them less often correct (because people generating AI blogspam generally don't need what they generate to be consistently correct) and harder to check (because people generating AI blogspam generally don't need what they have to be clearly sourced, and because even if they did it's an extra step of link-following).<p>So if Perplexity turns up a lot of AI-generated blogspam, that probably makes it less useful. Is it "up to Perplexity"? Again, I don't know what that means, but I don't see that it matters. What matters is how much value it provides, at what cost in time and money and effort and attention. If its output is mostly repeating claims from AI-generated blogspam and that makes the answer "not much, and at too much", then so much the worse for Perplexity even if there's some sense in which it's under no <i>obligation</i> to do better. If its output is reliable and checkable, then so much the better for Perplexity even if some of its deficiencies are in some sense moral failures.
How do they ensure that?
They fine tune for that, they don't ensure anything.<p>(Or at least they used to, I think they changed everything recently, but I've mostly stopped using them.)
No one ensures what it does but it’s proactive, it gets the people buying
This article is complete slop and not worth looking at, but it's an issue worth discussing.<p>One of the original php2shell announcement posts (which was clearly AI written) claimed that 70% of websites use PHP. This number has now been repeated by every single article about php2shell, including in the most trusted tech news outlets.<p>I think it could have been approximately true in 2008, but not in 2026. The number is too round anyway. Trying to search for a source to the claim is now impossible.<p>If the basis of a reliable fact is consensus, then every AI model should now consider this fact to be true. In reality consensus is not useful of course, offhand facts are frequently generated by citogenesis. An article mentioning a stat offhand is <i>not</i> a source. A source should take some amount of epistemic responsibility for a claim.
There is a human/user element here as well - will the users check the citations, if they are provided inline? I recall in the early ChatGPT web search and Perplexity days, most of the citations didn't work (made up, or 404). You get a false sense of security by having a "[1]" next to the stat or a claim. Maybe labs companies should have some additional quality ranking/checks on these, if it's in their interest of course.
A full 40% of AI stats actually have a specific study behind them? Knowing how the human baseline is that 73.6%* of all statistics are made up on the spot, this is really encouraging.<p>*That particular stat was of course made up, but not by me.
Just an thinly veiled ad for something called trakkr. Not to mention that none of these "stats" actually matter aside from being the type of thing that a consultant would put in a pitch deck, at which point their accuracy doesn't matter anyways
Never trust a statistic you haven't sneaked into the training corpus yourself.
So no different, than humans repeating facts we hear every day as 1 in 5 dentists tells us
Better to be on the 40% or the 60%? Seems like both sides will bite you in the ass to an extent
Ah yes. The AI slop blogpost calling out AI slop statistics. We must go deeper
when people say "there should be no AI taught in schools ever. prompting? what skill is there in talking to a liar bot?" this is where they are dead wrong. Being correctly critical of LLM output is a huge skill and news flash it applies to people and human-written things too.
[delayed]
Being skeptical/critical isn't a skill, it's an attitude, a state of mind. An education that requires you to find your own information and sources manually breeds that state of mind. Giving LLMs to kids (or even adults) nurtures the opposite. It's like taking them to McDonald's every day and telling them to eat healthy. Yes there's a salad on the menu, but it's rarely going to be chosen.<p>Of course the reality that adults do trust AI uncritically because it's the path of least resistance means the traditional education I mentioned above wasn't sufficient. But that's all the more reason not to introduce a variable that makes things worse.
[flagged]
[dead]
[flagged]