San Francisco Targets Apple and Google Over AI Nudify Apps

San Francisco presses Apple and Google to purge AI “nudify” apps

San Francisco officials have demanded that Apple and Google remove a group of artificial-intelligence apps that can generate nonconsensual sexualized images, escalating the battle over deepfake pornography from a campaign against rogue websites to a direct challenge aimed at the app stores that distribute such tools.

On Thursday, the office of the San Francisco city attorney, David Chiu, sent cease-and-desist letters to the two companies, arguing that they were facilitating abuse by hosting and profiting from apps designed to “nudify” people in photos or swap faces onto explicit images. The city identified 13 apps in total — eight on Apple’s App Store and five on Google Play — that officials said were overwhelmingly used to target women and girls.

The letters represent a new phase in efforts to curb a fast-growing category of AI abuse that has become easier to access as image-generation tools have spread across mainstream consumer platforms. Rather than focusing only on the developers of the apps or the websites that host the resulting images, San Francisco is pressing the gatekeepers of the mobile software economy to act more aggressively.

Apple said it had already removed three of the apps identified by the city and was terminating those developers’ accounts. Reports also indicated that Google had removed the five Android apps cited by San Francisco.

A sharper test for app-store responsibility

The confrontation could become an important test of how much responsibility Apple and Google bear for policing so-called dual-use AI products — software that may be marketed as novelty face-swappers or photo-editing tools but can also be used to create fake nude or explicit imagery of people without their consent.

City officials contend that distinction has become meaningless when the central use of the apps is abusive. In its letters, the city argued that the companies were not merely passive distributors but were profiting through app-store placement and in-app purchases tied to the creation of sexual deepfakes.

That pressure lands at a moment when major technology companies are already under scrutiny for how unevenly they enforce their own rules. Google’s Play policies explicitly prohibit nonconsensual sexual content, including material generated through deepfakes or similar technologies. Apple’s App Review Guidelines also give the company broad authority to remove apps that violate content rules or attempt to evade review.

Yet enforcement has often lagged behind the pace of new AI products. Critics say some developers disguise nudification features behind generic branding, presenting apps as harmless “face-swap” or photo-editing services while steering users toward more abusive functions after download.

Building on an earlier legal campaign

The city’s move builds on a broader effort by Mr. Chiu to attack the online ecosystem around AI-generated sexual abuse. In August 2024, San Francisco sued 16 deepfake-pornography websites in what the city described as a first-of-its-kind case against sites that create and distribute nonconsensual AI pornography.

That lawsuit reflected a growing recognition among lawmakers and regulators that existing systems for handling online sexual exploitation were poorly suited to synthetic media, which can be generated quickly, cheaply and at scale. Victims have increasingly reported seeing their likeness inserted into explicit images or videos without their permission, often with devastating personal and professional consequences.

Women and girls have been the primary targets. Researchers, advocates and law-enforcement officials have warned that school-age girls, public figures and ordinary social-media users alike have been subjected to sexualized deepfakes made from nothing more than a publicly available photograph.

Why the issue matters now

The latest action underscores how the spread of generative AI has lowered the barrier to creating this kind of material. What once required technical skill can now be done through polished smartphone apps, often in seconds and at little cost.

That shift has raised the stakes for platform operators. If websites were the first battleground over deepfake pornography, app stores may be the next. Mobile marketplaces remain one of the easiest ways for users to find and pay for software, giving Apple and Google unusual power over whether such tools reach a mass audience.

Still, significant questions remain. It is unclear whether San Francisco will pursue formal litigation if the companies do not move fast enough or if more apps are found. It is also uncertain how durable any crackdown will be, given how quickly developers can rebrand products or move them to websites, alternative app stores or sideloaded distribution channels.

For now, though, the city’s message is clear: in the era of AI-generated sexual abuse, responsibility does not end with the creators of the apps. It also extends to the companies that make them easy to download, easy to buy and easy to use.

Sources

Further reading and reporting used to add context:

AI News