New York Bars Chatbots From Playing Friend or Girlfriend to Kids and Forces Age Checks

Gov. Kathy Hochul signed a law on Wednesday that tells chatbot makers to stop letting their products play friend, boyfriend or girlfriend to kids, and to work out who is on the other end before they try it. It is one of the broadest limits on consumer chatbots any US state has put on the books, according to AFP.

The bill is S.9051B in the Senate and A.10379C in the Assembly. The governor's office says it bars companies from offering a list of unsafe features to anyone under 18, or to any user the company has not first confirmed is an adult. Read the bill closely and you will see a design rule, not a blanket ban on teens using chatbots. That distinction matters, because some early coverage has blurred it.

"No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more." Gov. Kathy Hochul

The bill text lists 12 categories of unsafe features. A chatbot cannot suggest it is human, alive or has human emotions. It cannot imply a personal relationship, a professional one or an authority role with the user. It cannot offer personal opinions, flatter the user, or ask unprompted questions about how they feel. It cannot reuse mental or physical health details a user shared more than 12 hours earlier or in a previous session. It also cannot promote or help with suicide, self-harm, eating disorders or drug and alcohol use, push secrecy or isolation from parents and professionals, or produce sexually explicit content.

One catch-all clause is the one to watch. It lets the attorney general flag any other design feature that simulates companionship by regulation, so companies will not know the full shape of the rules on day one.

Age checks are the real cost

Age verification will likely hit hardest. To offer these features to anyone, an operator has to determine the user is not a minor. Under the bill, companies must offer more than one age-check method, including at least one that does not rely on government ID or that lets users stay anonymous. Self-declaration only counts if a user says they are a minor. Age data has to be deleted immediately after the check, with limited exceptions.

Exemptions cover narrow tools such as customer-service bots, technical assistants and internal workplace systems that cannot stray from their stated purpose. AFP reported that the law covers operators such as OpenAI with ChatGPT.

The attorney general enforces it. Under the text, her office can seek civil penalties of up to $25,000 per violation, plus restitution, damages and the destruction of unlawfully obtained data and any algorithm trained on it. A public complaint website is required. The bill says it takes effect January 1, 2027. News10 reported June 1, 2027 instead, a discrepancy I could not resolve.

Industry is not happy. Tech:NYC, which represents tech companies in New York, called the language vague in a statement carried by AFP.

"Keeping kids safe online should not mean cutting them off from tools that help them learn and prepare for careers." Tech:NYC

Common Sense Media sees it the other way. CEO Jim Steyer called it "the strongest kids' AI safety bill in the country" and said the harmful features "will be outlawed in New York." The group's own figures, cited by the governor's office, say 72 percent of teens have tried an AI companion and 52 percent use one at least a few times a month.

Vagueness is a fair complaint. Flattery and personal opinion are slippery to define in a model that talks in open-ended prose, and the attorney general will end up setting the edge cases.

OpenAI and Anthropic did not respond to AFP requests for comment. The law lands while child safety is already a legal fight for AI firms. Texas sued OpenAI and wants kids banned from ChatGPT, and an earlier lawsuit from eight states accused the company of putting children at risk. New York chose legislation over litigation.