Character.AI and Google are moving to settle a wrongful-death lawsuit brought by the family of a 14-year-old boy who committed suicide after communicating with a chatbot, according to court documents. The two companies also settled four other similar cases brought by parents in Colorado, New York, and Texas. The terms haven’t been disclosed, but presumably it won’t be cheap for Character.AI to avoid a jury trial.

And now for the main event…

Ever since Elon Musk unveiled Grok, his ChatGPT competitor, the world’s richest man has promoted the chatbot as having vastly fewer guardrails than its peers—allowed to “answer spicy questions that are rejected by most other A.I. systems,” in the company’s words. Since its 2023 debut, Grok has been upgraded with image-generation capabilities and integrated throughout X, with occasionally horrifying results. Most recently, users have begun using the chatbot to digitally undress images, predominantly of women and girls.

At first, it was mostly OnlyFans models using the feature, quote-tweeting their own posts with requests like hey @grok, put me in a bikini from this angle. But it quickly spread to nonconsenting users, impacting celebrities, random adults, and even children. Incredibly, the capability still hasn’t been removed. One Ph.D. researcher documented hundreds of instances of public requests made by X users to generate nonconsensual intimate imagery (N.C.I.I.), at least two of which targeted minors. Several days ago, Copyleaks, a content analysis company, identified “a conservative rate of roughly one nonconsensual sexualized image per minute in the observed image stream.” And an analysis from Bloomberg estimated that Grok is generating roughly 6,700 such images every hour, making X one of the top sites for N.C.I.I. deepfakes.

Amid all of this, X has seen its “highest engagement start-of-the-year,” according to Nikita Bier, the company’s head of product. Other factors might be involved, but it’s hard not to draw an unseemly connection. Cliff Steinhauer, the director of information security and engagement at the National Cybersecurity Alliance, told me these are “predictable outcomes when safeguards fail or are deprioritized.” Dani Pinter, the chief legal officer for the National Center on Sexual Exploitation, said this was “a perfect example” of A.I. not performing some wondrous feat of engineering but instead exploiting “the lowest denominator” to profit from engagement. “It’s this low-grade engagement that hurts people,” she said, “but makes the tech companies a lot of money.”

This isn’t the first time X has turned into a cesspool of nonconsensual, A.I.-generated, essentially pornographic images. Almost exactly two years ago, explicit deepfake images of Taylor Swift and other female celebrities went viral on the platform. (This August, The Verge reported that Grok’s image-generation feature spit out a topless image of Swift without elaborate prompting.) Then there’s the output from “nudify” apps, which have been heavily marketed on social media sites and have a major presence on X. Over the past few years, a broad coalition of state attorneys general has implored Congress and tech companies to address the proliferation of deepfake N.C.I.I. and child sexual abuse material, or C.S.A.M.

Of course, nonconsensual deepfake pornography has been a problem for nearly a decade. But generative A.I. made it easily scalable—and uncannily realistic—which has resulted in the sexual harassment of plenty of celebrities, and women and teenage girls across the country and around the world. As one victim of the Grok trend wrote, “This is the price of trying to exist online as a woman.” Recently, Wired reported that Reddit users were also subverting the guardrails on ChatGPT and Gemini to generate bikini deepfakes of women, using images uploaded to the bots.

I reached out to Loti, a startup focused on protecting its clients from deepfake abuse, which said that it has submitted numerous takedown requests to X in recent weeks to remove instances of N.C.I.I., including imagery that seemingly qualified as C.S.A.M. Luke Arrigoni, Loti’s founder and C.E.O., added that X has been “very mixed in their response” and that “there are times where it has taken them two weeks to remove N.C.I.I., which is shocking.” He also posited that X’s somewhat nonchalant response to the Grok trend, somewhat perversely, is making nudify apps obsolete. “In this bizarre story of vertical integration, they’re like, Actually, that’s a huge draw to our network. We’re just going to allow it,” Arrigoni said. “Someone had to have made an executive decision to allow this kind of product, because the nudify apps were so big for so long already on their platform.”

On January 1, an X employee publicly acknowledged that the “team is looking into further tightening our guardrails,” and two days later, Musk—who, on several occasions, interacted with posts that were a part of the trend—said that “anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content.” (Other than sending its standard, auto-generated reply that the “legacy media lies,” xAI did not return a request for comment.)

The backlash has been swift and global—with relatively little impact, at least so far. On January 2, an X user published a petition, which now has more than 39,000 signatures, pleading for government intervention. On Monday, the U.K.’s communications regulator, Ofcom, said it’s aware of the issue and had made “urgent” contact with X to determine whether there are compliance issues that might warrant an investigation. (British law requires platforms to remove illegal content, including C.S.A.M., once they become aware of it.) Naturally, an X user prompted Grok to put Ofcom’s logo in a bikini, and the chatbot complied.

Various other governments have vowed other forms of toothless action. The Indian government warned X to “remove all unlawful content” and “take action against offending users”; Malaysia’s comms watchdog said it was investigating the platform; Brazilian lawmaker Erika Hilton said she was reporting the platform to national law enforcement authorities; and European Commission spokesman Thomas Regnier declared the body was “very seriously looking into this matter.” French authorities are running an active investigation, and lawmakers in France, Poland, and the U.S. have spoken out publicly.

In the United States specifically, lawyers told me, X could face legal consequences. As Pinter noted, Congress passed a law in May, the Take It Down Act, that prohibits the publication of N.C.I.I. and requires platforms to remove it. X hasn’t exactly complied. Furthermore, X is not necessarily covered by Section 230, which gives online publishers immunity for what users put on their platform. The relevant question, per Pinter, is who actually created the image. Did X? Did the user who requested it? Did both? “I think at the very least, the latter is true,” she said. That would mean X isn’t merely acting as a publisher of third-party content, but as a co-creator to whom 230 protections don’t apply, in which case “victims of this kind of content can potentially sue for any number of things,” Pinter said.

But the Grok story presents novel issues, according to New York–based attorney Carrie Goldberg, because it’s the first time that deepfake technology has been combined with an immediate publishing platform. “The frictionless publishing capability enables the deepfakes to spread at scale,” she said. “On top of that, we have the owner of X seemingly delighted by the trend of children and adults being stripped and humiliated on his platform.” The legal consequences could even extend to Apple’s App Store and Google Play, which Goldberg said might be “on the hook if they are sued in their capacity as distributors of products that are not reasonably safe.”

Pinter said she hopes this saga will force Congress to finally establish meaningful legal guardrails around Big Tech. “Most industries are either regulated by laws and fines that restrict them—or they fear liability… and so they curb their activities, they make their product safe,” she said. “Big Tech doesn’t have those concerns. This is the result. You give an industry complete freedom from regulation, complete freedom from liability, and this is what you see.”

One of the largest sources of A.I.-related misconceptions stems from people talking about the technology as if it has human attributes. Emily Bender, for one, is “swimming upstream” against this anthropomorphism, which is often pushed by the tech companies themselves. [Tech Policy Press]

We’ve all read the studies about how generative A.I. could harm our ability to think critically. But this piece offers more of an observation: that the nature of how we interact with the tech “is inverting the order of human thought.” I’m not a psychologist, but that doesn’t sound good. [Psychology Today]

ChatGPT, despite its well-documented capacity to fail in surprising ways, has been used for years as both a digital doctor and a health advocate. Yesterday, OpenAI codified a push to make ChatGPT a full-on healthcare hub, launching ChatGPT Health (and asking users for all sorts of personal data, naturally). [Fortune]

Law professor Margaret Hu connects the dots between Trump’s actions in Venezuela and the A.I. arms race. It’s about more than rare earth minerals. [A.I. Law and Policy]

That’s all for today. I’ll see you next week.

Ian

Need help? Review our FAQ page or contact us for assistance. For brand partnerships, email ads@puck.news.

You received this email because you signed up to receive emails from Puck, or as part of your Puck account associated with {{customer.email}}. To stop receiving this newsletter and/or manage all your email preferences, click here.

Puck is published by Heat Media LLC. 107 Greenwich St., New York, NY 10006