12 comments

  • maxgee2 hours ago
    Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.<p>We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.<p>OpenAI does not care about fighting child sexual abuse, this is completely performative.<p>EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We&#x27;ve received no notification of any problems from Cloudflare. Something on OpenAI&#x27;s system thinks that an image we sent to them for classification is CSAM, and they aren&#x27;t willing to elaborate or communicate about it at all. For reference, our website is <a href="https:&#x2F;&#x2F;ytmnd.com" rel="nofollow">https:&#x2F;&#x2F;ytmnd.com</a>
    • uptrails1 hour ago
      I&#x27;ve done some work in this space and it&#x27;s mostly that Big Tech has regulators and parent advocacy groups <i>already very angry with them</i> and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.<p>Another complicating factor is that it <i>really</i> blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.<p>I don&#x27;t think there&#x27;s an elegant solution to this, other than &quot;don&#x27;t allow tech companies to get so big that they can afford not to care about individual customers&quot;.
    • icantevenhold1 hour ago
      I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.
      • hingler361 hour ago
        I would posit the exact opposite: it&#x27;s really important to fight child sex abuse, which is why it&#x27;s so frustrating that people are co-opting that cause to promote surveillance and data collection.
        • icantevenhold13 minutes ago
          Oh you misunderstood me; it’s important for sure.<p>I don’t think this is something that can be solved by more surveillance though.<p>Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)<p>Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.
        • exolymph1 hour ago
          Saying it&#x27;s important is not the opposite of saying that it&#x27;s rare to actually care.
        • demibabs1 hour ago
          How else do you prevent it tho?
          • joe_the_user40 minutes ago
            One key means of preventing the exploitation of children is placing children in situation where they aren&#x27;t powerless, where their caregivers aren&#x27;t in such a marginal position that they can&#x27;t prevent predators from exploiting them.<p>Basically, keeping children from being miserably. Poverty isn&#x27;t the only source of the exploitation of children but it&#x27;s huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.<p>But in our society, even in a much better society than this, bad things are going to happen <i>at scale</i>. At a certain point, you have abandon the &quot;we&#x27;ll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done&quot;.
          • GuB-4230 minutes ago
            [dead]
        • DrammBA16 minutes ago
          I&#x27;m not sure your definitions of &quot;posit&quot; and &quot;exact opposite&quot; align with common usage.
      • joe_the_user1 hour ago
        I&#x27;d agree but I&#x27;d add a lot of things are done to get the child sex crusaders off a given corporation&#x27;s back. Moreover, the crusaders will use every victory to widen their definition of bad things (which isn&#x27;t me saying anything &quot;good&quot; is here) and number of areas they can trawl for this bad stuff. Enlarged enough, the agenda essentially would include criminalizing pornography and any &quot;deviant&quot; sexuality.<p>But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
      • nonethewiser1 hour ago
        [flagged]
        • graemep1 hour ago
          I think the good faith interpretation of GP&#x27;s comment is that almost no one among big tech companies (and maybe governments) cares.
          • nonethewiser1 hour ago
            I think the good faith interpretation of his comment is that most people and companies dont care, which Im responding to. If the commenter doesn’t agree with his own words he should clarify.<p>I think virtually every person and company does care. In fact I think its exceedingly obvious.<p>Even if OpenAI, Facebook, etc dont care be cause they are evil, they do care because pf liability.<p>None of this even excludes the possibility that CSAM controls are co-opted for power sometimes. Which is why it’s absurd to suggest these companies don’t care - you can concede that point and still argue its being co opted.
            • zelphirkalt55 minutes ago
              So they don&#x27;t actually care about CSAM existing, they only care about not losing money due to it being on their platforms. Did I get that right? In that case you are confirming what icantevenhold is expressing.<p>To actually care about something is not the same as doing arbitrary actions due to financial reasons. That&#x27;s caring about money, and nothing else.
              • nonethewiser20 minutes ago
                Im not confirming what icantevenhold said because what he said is broader than the narrow point of liability=caring. Executing on operational requirements to remove CSAM looks a lot more like caring than actively fearmongering people about CSAM to justify new powers.<p>What do you think? Do you think Facebook, OpenAI, etc. are fine with child porn on their platforms save for liability? A liability which of course exists because virtually everyone agrees that it&#x27;s a real problem and not just an excuse to grab power.
      • grvdrm1 hour ago
        I agree. Because if they do they hold some sort of view on what is or isn’t in an image&#x2F;video. I think it is simpler and easier to put your hands over your eyes and look the other way.
    • chinathrow8 minutes ago
      If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?
    • mattmaroon1 hour ago
      I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.<p>The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
      • zelphirkalt52 minutes ago
        Though of course fastest growing is not any excuse to not be prepared. Maybe the growing should happen in other directions too, than only investor money and evaluation. There are other axes of growth than the financial one.
    • nonethewiser1 hour ago
      &gt; OpenAI does not care about fighting child sexual abuse, this is completely performative.<p>That doesn’t really follow from your experience.<p>OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.
      • ezfe1 hour ago
        But part of that was trying to identify which content was flagged so that the uploader could be identified, but since it sounds like they didn&#x27;t identify the content now the OP cannot report it to the authorities.
      • rpdillon16 minutes ago
        OpenAI was given a chance to help someone fight child sexual abuse. They declined to do so, potentially violating the law in the process. It&#x27;s a pretty straight line, given the facts of the post.
      • maxgee1 hour ago
        If they were serious about stopping CSAM they&#x27;d spend 10 seconds looking at the prompt we used, the emails we&#x27;ve sent and the fact we&#x27;re trying to get an NCMEC case number and report the uploader, but we can&#x27;t because we literally don&#x27;t know what image they&#x27;re referencing. If a human was involved in any part of this process or decision, it isn&#x27;t apparent because all we&#x27;ve received is automated responses.<p>Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
      • charcircuit1 hour ago
        His point is that even if they do care, they are being performative in going completely silent, doing &quot;nothing&quot; as opposed to working together with others to remove content and report bad actors.
    • jstummbillig8 minutes ago
      Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.<p>How much patience would you have for some entity maybe &quot;accidentally&quot; processing something that (apparently) looks like child porn against your policies?
    • delichon1 hour ago
      Classifying unreviewed user-uploaded content with a service that does not accept CSAM is a footgun, even if they misclassified something this time.
    • antiloper1 hour ago
      I mean, you&#x27;re holding it wrong. You uploaded CSAM to OpenAI which is presumably against their TOS.
      • user439280 minutes ago
        Allegedly. Or the discrepancy in Cloudflare&#x27;s and OpenAI&#x27;s detection is caused by a false positive on OpenAI&#x27;s side. Who knows?
    • surgical_fire1 hour ago
      Oh my, ytmnd!<p>I used to view shit there a lot sometime in the mid-2000s.
    • teiferer1 hour ago
      Seems like they want to play in the big boys club (Google, Facebook, ... ) where this kind of treatment is standard.
    • gabucino22 minutes ago
      [dead]
    • ajjahs1 hour ago
      [dead]
  • Donald2 hours ago
    It is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it&#x27;s detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user&#x27;s local law enforcement jurisdiction, and they&#x27;re arrested within a short time period, like one or two weeks.
    • itake1 hour ago
      &gt; A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.<p><a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;2022&#x2F;08&#x2F;21&#x2F;technology&#x2F;google-surveillance-toddler-photo.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;2022&#x2F;08&#x2F;21&#x2F;technology&#x2F;google-surveil...</a>
      • jauer23 minutes ago
        This is addressed in the model card: <a href="https:&#x2F;&#x2F;deploymentsafety.openai.com&#x2F;csam-image-classifier&#x2F;definitions" rel="nofollow">https:&#x2F;&#x2F;deploymentsafety.openai.com&#x2F;csam-image-classifier&#x2F;de...</a>
      • graemep1 hour ago
        This is the problem. Unless an image is unambiguously a sexual image of a child it needs context, knowledge of intent, and common sense to classify correctly.
        • itake24 minutes ago
          Companies declare that human moderators are augmented by AI, but humans make mistakes and even a photo lacks the context.
      • teiferer1 hour ago
        There will only be very few generations of people who can see photos of them running around at the beach as small kids. That window closes faster than it opened just a few decades earlier.
    • abalashov1 hour ago
      I can see how this the average, unthinking member of the general population thinks this level of tech surveillance is a good idea, but those people haven&#x27;t paused to consider the implications in literally any other context.
  • nolist_policy1 hour ago
    Relatedly, Openrouter now does CSAM scanning and retention on so called &quot;Zero Data Retention&quot; endpoints as well.<p><a href="https:&#x2F;&#x2F;openrouter.ai&#x2F;blog&#x2F;announcements&#x2F;csam-screening&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openrouter.ai&#x2F;blog&#x2F;announcements&#x2F;csam-screening&#x2F;</a>
    • AlanYx1 hour ago
      Every mainstream vendor with a US presence does this on their ZDR&#x2F;MAM AI endpoints. This also includes AWS and Azure. It&#x27;s explained directly in the AWS ZDR docs, and it falls under the &quot;synchronous&quot; scanning pipeline in Azure with full MAM.<p>At least with AWS they&#x27;re clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it&#x27;s actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.<p>The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes &quot;actual knowledge&quot; of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there&#x27;s also a public optics element too I&#x27;m sure.
      • nwlieb8 minutes ago
        I&#x27;m personally hopeful that as an industry we can move towards E2EE and attested secure enclave pipelines to remove the &quot;actual knowledge&quot; argument. NEAR AI and Tinfoil are the only publicly available providers in this area but I would be excited to hear about more advancements in this area because I personally think that these NCMEC integrated &quot;ZDR&quot; systems are major privacy violations and deceptive marketing.
    • itake1 hour ago
      How are these companies able to legally train these models? Are the employees citizens of countries that this material is illegal to download or possess?
      • yh115 minutes ago
        i&#x27;m pretty sure certain organizations are given authorization
  • eutropia1 hour ago
    This is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.<p>When there&#x27;s an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the &#x27;creative freedom&#x27; of AI-generated material)<p>A model capable of shielding investigators from additional trauma is an excellent thing.
    • paulryanrogers1 hour ago
      &gt; diffusion models dramatically multiplying the supply of CSAM.<p>Isn&#x27;t CSAM only material based on abuse of real humans? How could computer models multiply CSAM by that definition?
      • schleck81 hour ago
        Depends on the definition. It&#x27;s become more frequent for predators to download images of real children from their parents&#x27; social media, then edit them with diffusion models to be nudes. This should still be illegal, and it is in the US.
        • paulryanrogers51 minutes ago
          So deep fakes? That does sound horrific and challenging to reliably detect and stop.
      • NoahZuniga1 hour ago
        Legally, if the computer generated content looks too real, its still criminal.
      • xp841 hour ago
        Yup. This is such hysteria. Personally I’d argue that the existence of diffusion models “multiplying the supply of this material” means that the value of it goes way down, meaning it’s harder for the truly monstrous parties who have for decades harmed actual human beings to create that “content” now face collapsing value of what they make, which makes it far less lucrative to be in that business and should make it less attractive to get into it, and remove some incentive to make more and more of it.<p>This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.<p>If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
        • nozzlegear0 minutes ago
          [delayed]
        • abalashov1 hour ago
          Some have argued the opposite, that &quot;multiplying the supply of this material&quot; further normalises sexual abuse of children, whereas before, it lurked in the shadows of the dark web and was treated like serious contraband, which imbued it with a certain gravity.<p>Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
      • nonethewiser1 hour ago
        This is a real moral dilemma for people who think consent and harm are all that matters. Vitriolic disgust at the idea of generating child porn using AI, but no good way to explain why its wrong.
      • simoncion1 hour ago
        &gt; Isn&#x27;t CSAM only material based on abuse of real humans?<p>If <i>only</i>.<p>The same &quot;hard on crime&quot; ratchet that happens to most every thing that&#x27;s popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
        • nonethewiser1 hour ago
          Huh. So you don’t think it’s morally wrong to generate child porn using AI?
          • abalashov39 minutes ago
            I don&#x27;t think OP thinks that. I hear him as pointing out that this question can&#x27;t be considered in acontextual, ahistorical vacuum. It&#x27;s a set of simultaneous equations, where the other equations and terms are of a more general form:<p>&quot;So you don&#x27;t think it&#x27;s morally wrong to generate _______ using AI?&quot;<p>&quot;So you don&#x27;t think it&#x27;s morally wrong to _______ using AI?&quot;<p>The matter can&#x27;t be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as &quot;pro-CSAM&quot; or &quot;anti-children&quot; at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.<p>As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That&#x27;s just the cost of freedom.
            • nonethewiser16 minutes ago
              I don&#x27;t really understand what you are saying.<p>To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn&#x27;t actually involve a human.<p>What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?<p>I guess calling it a cost recognizes there is something wrong with it at least.
              • abalashov8 minutes ago
                &gt; What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?<p>I don&#x27;t have a great answer on whether it should be &quot;allowed&quot;, but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for &quot;disallowing&quot; it.
        • abalashov1 hour ago
          It is indeed a very dangerous ratchet. Who in their right mind would take an extremely unpopular position that is affront to the SafeTy of children?<p>And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won&#x27;t be applied to other &quot;objectionable&quot; content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.<p>Automated technological surveillance and censorship is really, really having a moment right now. I&#x27;m not holding my breath that the pendulum swings the other way, but I sure hope it does.
  • itake1 hour ago
    My colleagues in Singapore trained pornography detection model.<p>Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
    • layer825 minutes ago
      Maybe they trained with live camera feeds. Data doesn’t need to be saved or downloaded in order to exist. :)
      • itake23 minutes ago
        Streaming is also banned.
        • layer821 minutes ago
          Camera feeds aren’t automatically streaming.
    • zelphirkalt48 minutes ago
      Maybe they got it sent from outside the country, on fast disks, so that they neither need to download it, nor save it (again).
    • dmitrig011 hour ago
      They must have amassed a large data set of things that are not pornography. Then trained the model on that. Anything that doesn’t match the model <i>is</i> pornography.<p>&#x2F;s
      • agilob24 minutes ago
        A reverse bloom filter<p>&gt;A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
      • trekz1 hour ago
        Funniest thing I read all week lol
      • nonethewiser1 hour ago
        Not hotdog
  • AmazingTurtle2 hours ago
    reminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)
  • nrmitchi15 minutes ago
    Frankly, and I don&#x27;t mean this in a bad way about the <i>problem</i>, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. &quot;Scan all your images through our CSAM detector&quot; means &quot;send all images to us, including the 99.99% that are not CSAM!&quot;<p>Don&#x27;t get me wrong: these are very necessary tools for the modern internet to function, and I&#x27;m sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
  • callc1 hour ago
    Curious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.<p>I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.<p>How should society&#x2F;law change in response? Is it purely a give and take between surveillance&#x2F;right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
    • nonethewiser1 hour ago
      &gt;I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.<p>Privacy, unfortunately, is largely the result of it being impractical to violate it.<p>I see zero reason to believe that states wont read your thoughts if they can easily manage to.
    • bob10291 hour ago
      There is a popular movie starring Tom Cruise that explores the consequences of this question.
      • VariousPrograms1 hour ago
        In Minority Report (spoilers), the precrime department is shut down because of a single error proving the system is fallible, thus making the plot preposterous in the age of AI.
        • mitthrowaway225 minutes ago
          <i>Psycho Pass</i> is perhaps a better depiction of what could happen to a society where the system is broadly accepted.
  • eastbound2 hours ago
    Finally Sam Altman found something good to do in terms of my security, I&#x27;m relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).
  • bearjaws2 hours ago
    If only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.<p>Could be a great start for &quot;rogue AI&quot;<p>Jokes aside, I&#x27;m glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
  • haickernews1 hour ago
    [flagged]
    • graemep1 hour ago
      Are you saying you are a paedo? I do not think you are going to get much sympathy for &quot;discrimination&quot;.
      • haickernews1 hour ago
        [flagged]
        • mcmcmc1 hour ago
          If you’re non-offending then why would you care about CSAM detection? Possessing CSAM <i>is</i> offending. You are not being discriminated against. That’s like saying murder laws are discriminatory against schizophrenics because it’s not their fault if they kill someone. It’s also incredibly damaging to the LGBTQ community to suggest that paedophilia is a sexual orientation when in reality it is a fetish&#x2F;mental illness.
          • GalaxyNova51 minutes ago
            While this is true it is difficult to argue that AI generated content or text hurts real children in the same way as CSAM.
          • haickernews1 hour ago
            From the article:<p>&gt;In SafetyKit’s evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases<p>&gt;This is the first CSAM classifier with broad availability that has a specific signal on indicating if CSAM may be AI generated.<p>What children are being abused by drawings or AI generated images, stories, etc?<p>I understand the cases where it&#x27;s problematic to make such content based on real children and share it, and I think it&#x27;s probably a bad idea to allow photorealistic child porn to be publicly posted on the internet for the sake of not confusing law enforcement (in the same way posting misinformation is harmful), but if these images are not realistic, or if it&#x27;s just text, or if they are realistic but they aren&#x27;t being shared on the internet, who is being harmed?
        • graemep1 hour ago
          You are human, but you are not just talking about a sexual preference, you want to be allowed to act on it to at least the extent of being allowed child porn.<p>Whatever you say about &quot;non-offending&quot; (and possessing child porn IS an offence in most places). I would definitely not have wanted you anywhere near my kids.
          • haickernews47 minutes ago
            As explained in another comment, this article includes examples of detecting drawings and AI generated images.<p>I believe AI is one of the only ways to engage with these desires in an ethical way.<p>Of course, you can always recommend abstinence, but unless you&#x27;re rather conservative in your views, you probably don&#x27;t think that&#x27;s a good way to deal with teenage pregnancy, homosexuality, etc. So why would this be the exception?
        • Computer01 hour ago
          For anyone else who doesn’t know what that is: NOMAP stands for &quot;non-offending minor-attracted person.&quot;
          • nonethewiser1 hour ago
            Aka pedophile<p>I understand the distinction between offending and non offending.<p>But pedophile is inclusive of both.