How California’s New AI Law Supports Accountability
California's new AI law takes a "better safe than sorry" approach, creating prospective accountability for catastrophic risks from the world's most powerful "frontier" AI models.
California—home to some of the largest AI developers in the world—has a new AI law on the books. Known as the “Transparency in Frontier Artificial Intelligence Act” (i.e. Senate Bill 53, or just SB53) the law is an important example of how legislative authority can strengthen the capacity for AI accountability. It provides a series of provisions that call for the release of information that help society know about potentially risky AI behaviors.
The scope of the law is quite narrow, however, as it only applies to “catastrophic risk” and “critical safety incidents” related to “frontier foundation models”. Unlike the wider scope of something like the EU’s AI Act, SB53 is really targeted. A “frontier” model is defined as one that is trained with more than a threshold number of numerical operations. What makes something “catastrophic” according to the law is that more than 50 people are seriously harmed, or damages amount to at least $1B from a single incident. The risks in focus here are hypothetical including AIs assisting with making or releasing chemical, biological, radiological, or nuclear weapons; unsupervised AI’s that engage in conduct that you might recognize as murder, assault, extortion, or theft; or evading control of the developer or user.
Because perhaps none of these risks have ever actually materialized, it’s appropriate to see this law as an implementation of the precautionary principle, the idea that action should be taken to prevent potential harm, even when scientific proof of the risk is incomplete or uncertain. Basically, better safe than sorry. The law is a nice example of creating prospective accountability — assigning responsibilities for preventing outcomes that are in this case still quite uncertain but which we want to minimize the chance of coming about. In this case this puts additional onus on the processes implemented that should prevent such risks from materializing, and for monitoring the system for indicators of such risks.
Transparency can be defined as “the availability of information about an actor allowing other actors to monitor the workings or performance of this actor” [1] and is recognized as a enabler for accountability [2]. We can’t hold accountable what we don’t know about. The law addresses the knowledge dimension of the AI Accountability problem using three mechanisms to increase the supply of information about catastrophic risks from frontier foundation models: (1) transparency reports with a wide range of information on risk prevention processess and assessments, (2) incident reporting of critical safety incidents or risks found from internal uses, and (3) reinforcing whistleblower protections so that insiders with direct experience can raise an alarm.
Reflecting a prospective accountability perspective the law requires frontier model developers to publish on their website a “frontier AI framework” that includes a lot of details meant to create processess that prevent catastrophic risks. The framework needs to include information about standards incorporated, thresholds for assessing catastrophic risk, mitigations taken for those risks, reviews of the adequecy of those mitigations, use of 3rd party assessments, updating the framework, security of model weights, the identification and response to critical safety incidents, internal governance practices to ensure these processes are implemented, and assessment of catastrophic risks from internal use. All of this is meant to establish a sound process for preventing these risks, and creates the conditions for accountability if the developer doesn’t have an adequate process.
Beyond the many bits of information that need to be disclosed in the frontier AI framework, developers also have to post a transparency report that includes a range of information about the model including, importantly, the intended uses and restrictions or conditions on uses of the model. These bits of information are important for accountability because they help define the appropriate behavior for users of the models. This transparency report also has to include additional information about the implementation of the frontier AI framework including specific assessments of catastrophic risk, the extent to which 3rd party evaluators (e.g. red teamers) were involved, and any other steps taken to implement the framework. In other words, the developer has to say how they’re fulfilling the process they outline in their framework.
Developers also have to disclose the results of assessments of catastrophic risks resulting from internal use of its models as well as any critical safety incidents to an administrative office of the state. So not only is the developer accountable to the public by way of posting a lot of information about its framwork and assessments to its website, but it is also accountable to an administrative forum where they report additional assessments and incidents. Interestingly, the incident reporting mechanism will also be open to the public, which is important insofar as one of the risks of concern is loss of control of the model by users.
The last bit here is that the law strengthens whistleblower protections. It basically clarifies that employees at frontier AI model developers who are responsible for “assessing, managing, or addressing risk of critical safety incidents” should be allowed to and not retaliated against if they disclose information to certain actors about whether activities of the frontier developer might pose danger resulting from a catastrophic risk.
The bill attaches clear consequences for failing to meet the obligations it lays out. The Attorney General of California can impose a civil penalty of up to $1,000,000 per violation for failing to publish required documents, making false statements, failing to report incidents, or not complying with its own frontier AI framework. This provides the sanctions that make the accountability relationship meaningful, though this only applies to large developers with more than $500M in annual gross revenue.
SB53 is generally a solid example of how you might go about legislating prospective AI accountability to prevent risks that are uncertain. It has provisions for updating over time that will be important for keeping it relevant as technology advances. Perhaps one of the biggest weaknesses I see is that the threshold of training compute used to define a “frontier model” (10^26 numerical calculations) is not required to be disclosed. At the end of the day it’s up to the model developer to raise their hand and say that this law applies to them and to identify which models it applies to. And by one estimate GPT-5 wouldn’t even fall in the remit of the law. We shall see how and whether the frontier model developers engage.
References
[1] Albert Meijer, “Transparency,” in The Oxford Handbook of Public Accountability, ed. Mark Bovens, Robert E. Goodin, and Thomas Schillemans (Oxford: Oxford University Press, 2014)
[2] Nicholas Diakopoulos, “Transparency,” in The Oxford Handbook of Ethics and AI. Eds. Markus Dubber, Frank Pasquale, Sunit Das. (Oxford: Oxford University Press, 2020)