AlphaSense: mentions of open models in US earnings calls and conferences rose 6x YoY in August and September; open models hit 56% of Vercel tokens in August

Open-weight AI models are increasingly scrutinized for safety risks, prompting new development and governance frameworks.

AI safety concerns are escalating, particularly around open-weight models. Z.ai and Concordia AI have released a report proposing a framework for managing risks associated with these user-modifiable AI systems. Unlike proprietary models, open-weight models allow free downloading and modification, making safety checks challenging post-release. The report emphasizes shifting safety measures to the development phase, including rigorous data curation and pre-training to filter hazardous content. This initiative coincides with global AI safety discussions, including OpenAI's temporary suspension of its next-generation model training due to unpredictable agent behavior. Chinese companies are increasingly focused on transparency, with Xiaomi live-streaming its model training process. Z.ai is also open-sourcing its coding assistant after a data privacy incident. Beijing has introduced a national AI Safety Governance Framework, signaling a regulatory push for stricter protocols in open-weight AI development, which is becoming a key area in the US-China tech competition.

The report by Z.ai and Concordia AI offers a critical, evidence-based foundation for balancing the substantial benefits of open-weight AI against its inherent risks. By emphasizing upstream safety measures, such as training-data curation and pre-training filtering, the framework aims to mitigate potential misuse before models are released into the public domain, a crucial step given the inability to monitor or control user modifications post-deployment.

This development highlights a growing divide in AI development philosophies, with Chinese entities pushing for transparency and proactive safety measures in open-weight models, contrasting with the more secretive, proprietary approaches often seen from leading US labs. The increasing competitiveness of Chinese open-weight models also intensifies the debate in the US regarding potential restrictions on foreign open-source AI, even as major tech firms advocate for an open ecosystem to maintain American leadership.