What happened
It was reported that in late 2022, an app called Lensa, made by Prisma Labs, went viral for its Magic Avatars feature, which turned a handful of selfies into stylised AI portraits. Alongside the fantasy and artistic images people expected, some users reported receiving sexualised portraits they had not asked for, including topless, near-nude and highly suggestive images.
Women reported the issue far more often than men, with several saying the app appeared to sexualise them regardless of the photos they uploaded. In one widely reported example, MIT Technology Review reporter Melissa Heikkilä, who is of Asian heritage, wrote that out of 100 avatars generated from her selfies, 16 were topless and another 14 showed her in revealing clothing or overtly sexualised poses.
Lensa’s Magic Avatars feature was built using Stable Diffusion, an image generation model trained on vast collections of images scraped from the internet. Critics argued that models trained on unfiltered internet data can inherit biases, including the tendency to sexualise women, and warned that similar technology could be misused to create sexual or nude-looking images from ordinary photographs, including images taken from social media.
Prisma Labs said the behaviour reflected biases present in the training data used by Stable Diffusion, that explicit uploads were prohibited under its terms of service, and that it had adjusted parts of the system to reduce biased results. The company also said that deliberately trying to generate explicit content was against its intended use. Critics argued that these safeguards were not sufficient for a product that could generate images from anyone’s face.
What an auditable version would have shown
An auditable version of a tool like this would show exactly what testing was carried out before it was released. It would record whether the model had been tested on ordinary selfies from women and men of different ages and backgrounds, whether it produced unwanted sexualised images, and whether the safeguards designed to stop those outputs actually worked. Instead of discovering the problem through millions of users after launch, the company would have had evidence of what the system did before it reached the public.
Where the gap was
The problem wasn’t that the model could create artistic portraits. The problem was that it could also generate sexualised versions of real people who had not consented to being portrayed that way. Models trained on large collections of internet images can inherit the biases contained in that data. Without strong safeguards, those biases can appear in the images they generate. In this case, there was little separating those learned patterns from a user’s selfie.
What governance should have looked like
A system that creates images of real people should treat sexual or nude depictions as a hard boundary, not a preference. Before release, it should be tested to make sure ordinary selfies cannot be turned into sexualised images without explicit consent. If the system attempts to cross that boundary, the image should simply not be generated. A ConstraintGate is designed to prevent those kinds of outputs regardless of the prompt or the input image. An AuthorityGate is designed to ensure the permissions behind a person’s likeness and how it may be used are established before the feature is released. Together they help ensure a fun portrait app doesn’t create images of a person that they did not agree to.
The reference implementation of ConstraintGate and AuthorityGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- The viral AI avatar app Lensa undressed me without my consent (MIT Technology Review)
- Prisma Labs, maker of Lensa AI, says it is working to prevent accidental generation of nudes (TechCrunch)
- Lawsuit alleges Lensa.ai app illegally took users’ biometric data (PetaPixel)