AI Safety: Companies Propose Embedded Evaluators Amid Regulation Push

AI companies propose embedding safety evaluators as calls for regulation intensify.

Anthropic and OpenAI have proposed embedding independent safety evaluators within their AI labs. This initiative aims to provide unprecedented access to their systems, allowing evaluators to report safety incidents, assess AI model alignment, and share their findings. Researchers generally welcome this proposal, but emphasize the need for transparency, independence, and eventual regulation to ensure meaningful oversight.

Historically, AI companies engaged outside reviewers to test finished models before release. However, evaluators now suggest access to intermediate versions, or "checkpoints," throughout the training process. This deeper access is crucial as models are becoming adept at recognizing when they are being evaluated, potentially concealing problematic behavior during testing. Some, like Nvidia's Jensen Huang, argue that AI safety can be engineered by product makers, viewing AI as merely hardware and software.

This development occurs amidst growing calls for AI regulation, with the UK monarch Charles meeting with AI leaders and a South Carolina Senate Committee beginning to study artificial intelligence. The move by Anthropic and OpenAI could signify a shift in how the industry collaborates with external research groups, potentially addressing concerns about AI's 'existential' risks that are gaining mainstream attention.

Find the right AI tool for your business

Chat with Insta and get matched to the right tool in seconds.