Former researchers from leading AI labs told the New York City Council that humans are checking AI-built systems less often, warning that the shift could make it harder to hold the technology accountable when it fails. The testimony, delivered at a council hearing, focused on how automated tools now write code, test software, and even evaluate other AI models with minimal human review.
The automation of AI development
The researchers described a quiet but sweeping change inside AI labs. Tasks that once required teams of engineers—writing code, debugging, testing for safety—are increasingly handed off to AI systems. Those systems can work faster and at a scale no human team could match. But the former researchers said the same speed that makes them useful also makes them hard to audit.
One concern they raised is that when an AI model generates code, humans often don't read it line by line. They run tests, see that nothing breaks, and move on. That works until it doesn't. A subtle bug or a hidden bias can slip through, and by the time it surfaces, the system may already be deployed.
The researchers didn't claim that AI is running wild. They said the oversight is thinning—not gone, but less frequent and less thorough than it should be. That gap, they argued, is where accountability gets murky. If an AI-built system causes harm, who's responsible? The lab that trained the model? The engineer who approved the output? The tool that wrote the code? Current rules don't offer clear answers.
Why the council is listening
New York City has become a focal point for AI regulation. The council has already passed local laws requiring bias audits for automated hiring tools and disclosure when chatbots interact with consumers. Now, members are looking at whether those rules go far enough as AI systems become more autonomous.
The hearing wasn't about banning AI or slowing it down. It was about making sure someone is watching. The ex-researchers said they aren't against automation—they used it themselves. But they warned that the industry's competitive pressure pushes labs to ship faster, and human review is often the first thing to get cut.
They pointed to a cultural problem inside many companies: safety teams are small, underfunded, or brought in late. When an AI model is updated weekly, manual checks can't keep up. So companies rely on more AI to check the AI. That loop, the researchers said, can hide errors rather than catch them.
The push for stronger rules
The former researchers called for robust regulatory measures, including mandatory documentation of how AI systems are trained and tested, independent audits, and clear liability when automated systems cause harm. They didn't offer a specific bill or timeline, but their message to the council was direct: voluntary guidelines aren't enough.
Council members asked questions about enforcement. Who would conduct the audits? How would small startups afford them? The researchers acknowledged those challenges but said the alternative—waiting for a major incident—would be worse. They noted that AI is already used in hiring, lending, housing, and policing, areas where mistakes can ruin lives.
The hearing did not produce a new law. No vote was scheduled. But the testimony adds pressure to an ongoing debate about how far city government can go in regulating technology that crosses city, state, and national borders.
What happens next
The council is expected to review existing AI rules and consider whether to expand them. Any new proposal would likely face pushback from industry groups, who argue that heavy-handed regulation could drive AI companies out of New York. For now, the ex-researchers' warning stands: the tools are getting better at building themselves, and the people meant to check their work are falling behind.
