General Regulatory Process for Advanced Technology

Purpose of This Document

Any increase in technological capacity inevitably generates a corresponding increase in risks. When a new technology delivers a step function in capability, there must also be a corresponding step function in safety assessment and regulation. In the absence of major innovation in the governance of technology, the world risks unprecedented catastrophic harms from a range of advanced and emerging technologies, which are growing exponentially in power, scope, and potential impact. This document presents an outline for a paradigm of safety capable of binding the power of emerging exponential technologies, and provides a framework for regulation for sustained, adaptive risk management.

Definitions

• Advanced Technology: any new technology that delivers a step function (i.e., multiples to orders of magnitude) in the speed, complexity, or scale of effects, relative to other well-established and regulated technologies.
• Exponential Technology: any technology that can be used to recursively improve itself, leading to a compounding set of effects. More broadly: any technology for which development and/or deployment follows some exponential curve. In the widest sense, exponential technology refers to any technology for which the input-to-output ratio follows an exponential curve, which includes (for example) exponential increases in:
  • the number of people affected;
  • the speed of adoption;
  • the degree of potential customization;
  • the scale of returns generated;
  • the total scope of potential impact.¹

The Problem

The history of technological development demonstrates that intended benefits are typically accompanied by unintended harms, including unanticipated externalities and malicious or rivalrous uses. This dynamic may be observed across all types of technology.

The case of leaded gasoline is illustrative: adding lead to gasoline to solve the problem of engine knocking resulted in countless deaths, ubiquitous health issues, and more than a billion points of IQ lost globally, with incalculable downstream effects. In nature, lead is bound in rocks, where it presents minimal risk of toxicity; humans developed systems of technology to extract this potent neurotoxin from ore and aerosolize it into the atmosphere via internal combustion engines. The connected set of technologies that made possible the single catastrophic act of adding lead to gasoline—such as the refining of oil, and the internal combustion engine itself—continue to deliver their own array of unintended downstream effects, including many environmental harms, impacts on human health and development, and prolonged geopolitical conflict. This linked set of examples serves to highlight how ecosystems of dependent technologies (e.g., gasoline and engines) create a broad range of harms across completely different domains (e.g., human health, ecology, conflict). A similar scale of under-appreciated harm and risk is associated with many categories of new technology, including for example pesticides and herbicides, fertilizers, mining, drones, pharmaceuticals, cyber technologies, and military technologies.

Most new technologies are developed on behalf of the market, in order to capitalize on scientific breakthroughs, with incentives for asymmetric focus on upside relative to downside (e.g., those focused on understanding risks in depth deliver products to market more slowly, while competitors downplaying possible harms win first-mover advantage). Most new technologies are deployed without meaningful regulation, and public understanding of risks is shaped almost entirely by communications and campaign budgets, marketing narratives, and pre-launch investment in lobbying, as well as targeted litigation and the strategic placement of individuals within regulatory bodies.

Regulation Follows Harm

Regulation tends to follow harms only once it occurs at scale. Examples of early advertising campaigns for cigarettes (“more doctors smoke Camel”) and DDT (“DDT is good for me-e-e!”) demonstrate how death and suffering at scale—coupled with enough severity, awareness, and concern—is often necessary prior to basic regulatory actions. Even then, retroactive regulation can only prevent more of the same kind of harm from occurring. When a new technology creates a profound dependence that cannot be replaced or removed (for example, as with internal combustion, nuclear weapons, or social media), the harm and risk simply has to be endured (and mitigated) as a permanent cost to all life on Earth.

The power, complexity, size of user base, and speed of deployment of emerging technologies are all following exponential curves—as are the risks and harms. AI and synthetic biology are representative of some of the most powerful categories of new technologies. If they have unintended associated harms comparable to previous technologies, by the time retroactive regulation is ready to respond, thresholds leading to global catastrophic harm will have already been crossed.

Market incentives encourage technologists to move fast and break things in pursuit of disruptive innovation, in the context of pre-existing and highly effective mechanisms to privatize gains and socialize losses. Although it can be hard to imagine an alternative to a world shaped by arms and market races, the only rational conclusion is that it is in our collective interest that adequate processes to ensure safety precede the development of unprecedentedly powerful new technologies. The inevitable outcome of any other approach is catastrophic harm.

Background

Technology can enable harm creation of three primary types:

1. It can enable certain actors to enact new harms due to the affordances enabled by the technology.² This includes purposeful or malicious harm. For this case, regulation is created around lawful use of the technology, and/or the user type.
2. In the process of producing the intended lawful effect, the technology produces unintended harmful consequences such as externalities to the environment or social sphere. For this case, regulation is created in the context of these externalities in order to reduce the risk to the environment and society.
3. The tech itself can also become a target (e.g., cyber systems such as supervisory control and data acquisition systems are constantly under attack). New capabilities enable new vulnerability and target attack space. Given the exponential rate of technology adoption, the resulting new concentrated dependencies and interdependencies encourage and enable attacks.

Every technological innovation that mediates human needs portends new risks and opportunities, both intentional and unintentional.³ Moreover, while satisfying a human need or creating a new dependency, technologists are accruing (and conferring to partners, networks, and allies) enormous amounts of power and influence. Many new technologies are designed to address a particular problem, and in this narrow context may appear beneficial to society’s improved functioning; a broader perspective often reveals many negative externalities, as well as opening moves in new technological arms races—the total effects of which are typically net-harmful to society.

The intended and unintended risks of new technologies require assessment and mitigation, and often result in the need for regulation. A major part of government departmental activity can be seen as anticipating, monitoring, and regulating these two types of risks associated with each category of technology. Departments aim to assess the societal risks and benefits associated with a technology and create regulatory frameworks for its development and deployment that allow for the advancement of the intended benefits, while seeking to identify unintended consequences and mitigate unacceptable harms.

Case Study: Civil Aviation

In civil aviation, regulation exists to address both intentional harms (such as the malicious act of flying a plane into a building), as well as unintentional harms (such as accidents caused by faulty manufacturing or poor maintenance). Most Federal Aviation Authority (FAA) regulations are not based on cases of intentional harm, but rather the human and technological processes necessary to ensure safety for passengers flying on a plane. This includes everything from training and testing of pilots, air traffic controllers, mechanics, and baggage handlers to the regulatory oversight of complex communication systems and flight path routing. The air travel industry affects the world in many ways beyond aircraft and passenger safety that also require safety assessment and regulation. These include (for example):

• supply chains needed to produce and maintain the aircraft, and process and distribute the fuel;
• processes for monitoring and controlling the transmission of pathogens or invasive species (for humans and cargo);
• agreements and other frameworks managing sensitive objects crossing national borders (weapons, chemicals, biological agents, drugs, sensitive information files, capital, etc.); and
• immigration controls.

The complete list of concerns relevant to regulation is vast. It includes everything from the health safety testing of the food served during flights to the noise ordinances that determine where airports are built. These safety assessments require coordination with many other regulatory agencies (including the Environmental Protection Agency (EPA) and the Transportation Security Administration (TSA), for example) to create, monitor, and enforce laws regarding security, law enforcement (both at home and abroad), toxic jet fuel emissions, and the externalities of the physical supply chains of product manufacturing.

Dependency

Attempting to assess and mitigate excessive harm caused by these complex supply chains and technological ecosystems is a major component of the work of maintaining a functional global civilization. And still, once dependency on these technologies is established, it becomes necessary to protect against supply disruption, which itself becomes a primary function of geopolitics and global finance (all the way up to and including direct conflict). The companies building and employing these technologies require oversight by agencies like the Federal Trade Commission (FTC) and the Securities and Exchange Commission (SEC) to address marketing and financial activities, the Internal Revenue Service (IRS) and state tax boards for tax compliance, and many other government agencies to oversee security risks. This incomplete overview covers only commercial airlines, and does not include the dual-use nature of aircraft manufacture for military use. The specialization of and coordination between regulatory bodies exists in an attempt to ensure that the total regulatory system is adequately detailed and comprehensive.⁴ Such processes are both already hard to maintain and subject to failure and corruption—and yet still far from adequate for the complexity, speed, and failure intolerance (i.e., risk magnitude) necessary to regulate new categories of advanced technology.

Market Incentive is Insufficient

Market incentive alone is insufficient to ensure the safety of new products, services, and technologies, even though it is in our shared interest to enact stronger mechanisms for adequate protections. Fundamentally, stronger mechanisms (and meaningful oversight) must be created, implemented, and continually evolved by regulatory bodies with incentives and institutional architectures better aligned to addressing the problems outlined above.

Furthermore, the state’s role in regulating industry provides an incentive for corporations to invest in lobbying, campaign finance, and other influence efforts to steer policies in a manner aligned with their own interests. This leads to a host of familiar mechanisms for regulatory capture, such as strategic litigation, campaign budgets, lobbying budgets, “revolving doors,” and Public-Private Partnerships (PPPs). PPPs represent a particularly insidious example of private influence over regulatory activities and public interests, as the private side of the partnership tends to be able to afford the lawyers that write most of the operating agreement underpinning the partnership, leading to a range of subtle loopholes and terms favorable to the private side of the deal. PPPs and other forms of regulatory capture generally enable undue influence on regulators, to the extent that such mechanisms can be seen as serving the interests of industry far more than the people and the commons. This may be more accurately described as crony capitalism or state-market-fusion oligopoly, as opposed to government oversight grounded in integrity and care for the public.

This reality should serve as motivation to innovate more robust and trustworthy regulatory processes, with enhanced checks and balances on power, and a focus on reducing susceptibility to capture or corruption. As technological capability grows exponentially and its deployment accelerates, concurrent exponential advances in the quality and nature of safety analysis and regulation are critically necessary. In their absence, this trend is existential.

The figures above illustrate the acceleration of scaling and adoption, and provide a sense of the rate at which billions of people are affected by new technologies. In the case of the software platforms referenced in these figures, there was no initial regulatory oversight on the technology’s development and deployment—and little subsequent regulation either. Products that run on personalized data-harvesting, split-testing, and behavior modification were deployed rapidly at a global scale, and have transformed almost every aspect of human life and relationships, with no ethical oversight and no mechanisms to anticipate, prevent, or correct harmful changes.

The Present

To underscore the reality of this unique period in history, it is worth considering the scale and scope of change we are pursuing through advanced technology. We are simultaneously developing:

• enormous neural networks that can radically outperform humans at an increasing number of critical functions, including strategy, war games, and rapid precision control of robotics;
• quantum computers capable of processing information below the level of classical physics;
• genome-editing technologies and new, synthetic life forms;
• self-replicating chemistry;
• advanced techniques for splitting atoms and fusing atomic nuclei; and
• high-energy particle accelerators to create temperatures hotter than supernovas.

We are engineering at the foundations of physics and life. Many of these technologies are advancing at exponential rates of change, with many of the exponential curves intersecting (and thereby delivering hard-to-predict, rapid phase changes, driving the erosion of technological constraints, and the emergence of new, key actors and centers of power). Humans have no intuitive mechanism to make sense of overlapping exponential functions, which in this case leads to a world of underappreciated and poorly characterized catastrophic risk.

Humanity’s track record of wielding its power does not lend much confidence for a future of exponentially more powerful technology. We can survive neither exponential externalities nor exponential arms races. A new paradigm of safety assessment and regulation is needed to address the new paradigm of technological capacity that is emerging.

Artificial Intelligence Is an Exponential Technology with Exponential Risks

A number of technologies are developing at faster-than-exponential rates, with multiple intersecting, exponential curves in supporting and adjacent technologies. For example, the field of AI is currently experiencing exponential rates of development in the capabilities and capacities of the underlying hardware, the number of parameters in the models, the scale and scope of training data, the number of sensors to gather more data, the number of cybernetic systems with which to engage, and the amount of money and talent flowing into the space. Other types of technology displaying a similar pattern include biotechnology and nanotechnology. Technologies that affect social behavior and operate according to network effects also deserve special consideration.

The negative externalities and weaponized applications of a technology scale with its overall power.⁵ This means exponential technologies typically drive exponential scaling of externalities and destructive capabilities. This is why it is likely that regulation created after the harms are enacted (as in most previous cases of tech deployment) will, in these novel cases, lead to outcomes that are both catastrophic and unrecoverable.

Perverse Incentives Regarding Safety

Technology development and safety analysis led by industry has resulted in the most harmful products in history, including leaded gasoline, asbestos, high fructose corn syrup, and many others. Despite clear evidence of harm, many products cannot be easily removed from the market due to vested economic interest and broader dependencies. Detailed upfront studies of possible externalities of a new technology cost money and time. First-to-market advantage is a more significant factor for new technologies than ever before; network dynamics now define the success of the most important and impactful digital businesses.⁶ Whenever a market is dependent on network dynamics, the first to reach a certain level of adoption is likely to be at the top of the power law for a long time (e.g., Amazon, Google, Facebook, PayPal, YouTube, Airbnb, Uber, etc.).

As currently structured, the market provides the incentive to build everything that can create returns on investment as quickly as possible, as long as it doesn’t clearly violate criminal law. Given the first-to-market incentives and resulting winner-takes-all dynamics associated with exponential tech, the incentive is to win the race at all costs. The incentives of the market promote minimal necessary safety assessment (accurately considered as “box checking” for plausible deniability), while promoting narratives that diminish the risks and exaggerate the benefits. The critical conclusion is that there is profound—indeed existential—misalignment between the incentives of the market and the degree of care and safety necessary. This misalignment is the appropriate frame for a conversation about regulation.

Existential Technology Requires Global Coordination

The advent of the nuclear age delivered the first instance of existential technology. Given the dual-use nature of fissile materials, even the application of nuclear energy (as distinct from weapons development) is tightly regulated and under extraordinary international oversight. Born of necessity, the creation of the International Atomic Energy Agency (IAEA) and subsequent regulation of nuclear technology is one of the strongest examples of effective international regulation.

The technologies in this “Advanced Tech” category (including AI, quantum computing, synthetic biology, and nanotechnology) portend in the worst cases even greater risks than nuclear technology. At the same time, they are much harder to monitor and regulate, given the relative ease of hardware development, radically lower cost and labor implications, and speed of technological advancement and proliferation. The risks of these technologies need safety and regulatory frameworks more like the IAEA than the FDA—and still, it is important to acknowledge that even an equivalent to the IAEA would be radically insufficient to the novelty of this risk space.

Advanced or Exponential Technology: Categorization and Regulatory Framework

This proposal claims that for certain advanced technologies that carry the potential for harms of all types—weaponized use, externalities, and the creation of new targets through dependence and hyperconnectivity—the risks are so significant that comprehensive safety analysis to inform safe regulatory processes must be completed as a pre-requisite for regulatory approval.

There is a distinction to be made between advanced technologies and all other categories of new technologies. Many types of technological innovation are new instances within a class of phenomena for which reasonable regulations have already been developed. This includes most innovations in materials science, energy production, manufacturing, waste management, transportation, communications, consumer goods, etc. While these are new technologies and require both new regulation and the adaptation of existing regulation, they are of a type that the existing regulatory frameworks have already addressed, and as such, new regulation could be developed along similar lines.

It is still the case, however, that under the regulatory frameworks for existing categories of technology, the world has paid a heavy price (in ecological, biological, sociological, psychological, and many other terms). Regardless of this clear insufficiency, given current rates of development in exponential technologies, a minimum-viable process for creating regulation to prevent impending global catastrophic threats must be prioritized. This includes technologies with:

• A catalytic role in catastrophic or existential risks;
• An intrinsic exponentiation as part of their mechanisms;
• The possibility of self-replication, including types of biotechnology, nanotech, AI, and social technologies;
• Inscrutability of causal processes;
• High combinatorial potential with other categories of tech and wide use-cases, such that the dual-use potential is both high and hard to assess;
• The ability to automate or disintermediate critical human social systems like healthcare, education, law, governance, or essential industries;
• The ability for undue influence on humans and human systems, such as IoT sensors and information technology applied to unwarranted behavioral nudging, government monitoring and enforcement, etc;
• The capacity to provide asymmetries of power such that no pre-existing checks or balances could be applied as a means of control, nor new, adequate versions be created (except via arms races involving that new type of tech);
• Exponentially fewer people, or less capital, hardware, or time needed to achieve global effects;
• Network effects that create functional monopolies.

Proposed Risk Assessment Framework

The potential risk associated with a new technology is approximately proportional to its power and inversely proportional to society’s ability to govern it well.

Any new technology’s power is a function of:

• the speed at which it can be developed, deployed, and improved;
• the scope and magnitude of potential effects;
• the complexity of the technology;
• the tech/capital/regulatory/behavioral ecosystem with which it will coemerge; and
• the total causal landscape it will affect.

Our ability to govern any new technology is dependent upon our ability to understand it and all of its effects, monitor its use, and control against excessively harmful or risky applications.

Safety guidelines and regulations can not only fail—they can also succeed as intended, and at the same time generate new and unforeseen problems. As such, the total risk analysis must include the assessment of the technology itself, as well as the risk analysis of the proposed safety mechanisms and regulatory frameworks.

Phases of risk assessment for a new type of advanced technology must proceed as follows, prior to deployment (and in some cases prior to development):

1. Assess the risks associated with a technology’s development, deployment, and use factoring all considerations outlined in this document.
2. Conduct a risk assessment of the robustness of the proposed safety protocols to address the risks of phase 1.
3. Conduct a risk assessment of the proposed regulatory framework itself, involving any requirements for monitoring and enforcement, processes for checking the power of that regulatory system, etc.

Clearly this proposal is recommending an undeniably complex and burdensome process. It is also the case, however, that any other approach is likely to lead to a near future of increasing catastrophic risk. There are no known alternatives for ensuring a desirable and safe future, beyond major innovation in the governance of technology. The process for delivering such a future will require unprecedented coordination between typically siloed disciplines and adversarial nation states. While these aims are lofty, it is important to accept that they are also fundamentally necessary—and at the same time, they do not violate any known laws of physics. The potential solution space for innovation in governance is profoundly underexplored.

Generalized Guidelines for Risk Assessment

Below is a set of generalized guidelines to determine safe use of certain advanced or exponential technologies, which includes both criteria for adequate risk assessments and guidelines for regulatory approval.

The risk assessment should both take into consideration the existing assessment criteria for less advanced technology and implement additional features for which there is no current precedent. Some of these include:

A. Combinatorial Effects

1. Risk cannot be assessed solely on the use of a technology in isolation, but must consider a technology as an enabling agent in combination with other types of tech, particularly in cases in which the combination can approach or cross thresholds of catastrophic harm. This approach considers new types of advanced tech in terms of their potential for catalytic effects as a part of catastrophic tech complexes (as opposed to perspectives that consider only standalone risk).
2. Existing regulations provide some precedent for an approach to combinatorial effects of powerful technologies. Examples include uranium ore (which is not by itself a weapon or even a fissile material) and pseudoephedrine (an uncontrolled pharmaceutical product that serves as a basic ingredient for a controlled substance, methamphetamine). A particular type of AI has the potential to be combined with many types of tech, whereby the true risks are most appropriately considered as the result of that total technological system—not the individual components. This can include the combination or addition of AI to pre-existing biological development tools, drones, cyber weapons, IoT, or other types of AI with potential for a particular catastrophic harm.
3. While the complexity of thinking through the combinatorial effects and adequate regulation of certain technologies may be daunting, it is now simply existentially necessary.

B. Affordances

1. Regulation must factor the fundamental capabilities enabled by the technology, beyond the proposed use case. All advanced tech is inherently dual-use/poly-use. The creation of a capability for any purpose radically lowers the barrier of entry for its use for other purposes and by other users. The default assumption must be that new technologies will be innovated upon and deployed by all agents for all purposes that they enable.

C. Cross-Jurisdiction

1. In the different fields of medicine, agriculture, public health, and security, it is typical to observe research and development into (for example) related types of bioengineering. The regulation of a particular type of technology, factoring risks, therefore must include all applications and authorities that would need to be involved, across sectors. Multi-jurisdictional authorities within a given country will also need to be considered.
2. Arms races or market races on advanced technology between different nation states, or commercial actors in different nation states, often make coordinated safety analysis and regulation impossible, due to competitive considerations. As such, safety analysis and regulation requires novel approaches to international agreement and cooperation.
3. Ensuring the safe application of these technologies will require coordination between countries, between federal departments, between national and local governments, between private and public sectors, and more.

D. Fragility of Regulatory System

1. It is necessary to factor the potential fragility of any proposed safety mechanism that leads to the approval for development of a certain advanced or exponential technology.
2. If a new type of technology creates new catastrophic risks, but a proposal is created for its safe deployment in service of positive applications, that safety proposal “green-lights” the development of a technology that (if ever decoupled from safety mechanisms) adds novel risk to the world irrevocably. Is it reasonable to assume that safety will be maintained enduringly, in every application, forever? If not and the technology is still deployed, the world has been made forever more fragile.
3. Assessing the full maintenance costs, complexities, and necessary redundancies for any safety protocol to be adequate, long-term, in the context of potential natural disasters, economic collapses, and wars, must be factored in the authorization process.
4. Risk assessment must be performed not only on the new technology, but also on the implementability and anti-fragility of the safety strategy, including its monitoring and enforcement.

E. Scrutability⁷

1. It is necessary to determine instances in which a technology is operating through mechanisms that are not adequately understood. If a technology is unexplainable, it is not possible to forecast emergent properties and behaviors in new situations and under new conditions, which inhibits comprehensive risk assessment.
2. Perfect scrutability is unlikely, so it is critical to define adequate scrutability for various contexts. The outcome of this process depends upon a technology’s relative novelty, the complexity of its interactions, as well as the degree to which its deployment environment is containable and controllable.

F. Safe-to-Fail Probing⁸

1. Only once thorough theoretical safety analysis has deemed it appropriate, small-scale highly controllable experiments should be conducted to observe for unanticipated effects. This should happen in an environment in which the worst-case scenario would still be containable.
2. Such probing serves to build knowledge of the system, risks, and failure modes in small, contained, and tolerable ways.
3. This process will require patience and multiple iterations, as some risks can seem small or dismissable in some environments but become larger or even catastrophic in others.
4. If a safe-to-fail probe is not possible, the technology should not progress.

G. Recursively Updating

1. Periodically, and upon breakthrough developments, any new capacities and risks must be assessed and regulatory guidelines updated accordingly.

General Principles

1. Prove Safety Beyond Reasonable Doubt

Given the potential scale of harm, prior approaches to risk-reward calculus that have been employed in relation to historical technological development must not be followed again, particularly in the context of global catastrophic risks. Under a scenario in which there is a plausible chance of destroying life as we know it, there is no amount of new upside that warrants taking such risk. In these cases, the precautionary principle under uncertainty must become the primary position.⁹ In scenarios in which there is meaningful uncertainty, consequentiality, and irreversibility, the burden of proof must be on safety, not on risk. Specifically, during the evaluation of technologies with potentially catastrophic risks, it is imperative that the standard of burden-of-proof be of the highest degree. Under all circumstances and use cases, the technologies must prove to be safe, prior to deployment, beyond reasonable doubt.

It is important to underscore the complexity of preventing harms, as opposed to the relatively simple task of achieving specified successes. The success only requires first-order effect assessment (i.e., assessment of the outcomes that the tech was designed to achieve directly), quantified via a small number of predefined success metrics. Assuring safety of the technologies in question requires factoring n^th-order effects on a very large number of unknown factors. This is an exponentially more complex (but unavoidable) task. Factoring this problem, a four-step assessment process for potential risks is necessary:

1. Engage a thorough yellow team process¹⁰ to specify all possible unintended effects of this technology. This process must consider and factor the direct effects of the technology in addition to those of the associated ecosystem of technologies, supply chains, and human factors required for and enabled by its proliferation; any physical and psychosocial externalities; and the effects of its deployment on power landscapes and the potential for new arms races. This analysis must necessarily involve an interdisciplinary process, so that specialists from adjacent domains can contribute to analysis, as well as representatives of various potentially affected stakeholder groups.
2. Once risks have been theorized, they should be integrated into the assessment model and tested. A comparison of predicted and real results is necessary, followed by iteration until no signs of concern show in safe-to-fail testing.
3. Following the conclusion of this process, minimal real-world deployment may begin with unfocused, wide witnessing, through which all changes in the world’s social, technological, psychological, physical (and other) landscapes should be considered. It is critical to consider whether the deployment of the technology could have a causal connection to a world event that was not anticipated. Wider deployment follows observed safety only once minimal deployment and safety analysis has been concluded. Ongoing monitoring for possible unanticipated consequences must continue.

2. Avoid Conflicts of Interest

While public-private partnerships and other forms of government contracting can seem convenient for certain joint ventures (including in the construction of infrastructure such as ports, bridges, and dams, as noted above), there are many instances in which the interests of private corporations conflict with the government’s ability to diligently assess and control risk. In the case of certain advanced technologies, where the risk types are radically consequential, unknown, and irreversible, no organization with vested interests can be involved in the risk analysis or regulatory process. To this end, however, there is unavoidable difficulty in finding the necessary number of people with domain expertise to assess risk with no prior industry experience. Given consequentiality, these risk assessments should have the maximum possible degree of oversight and auditing.

3. Enforcement Capacity

The enforcement capacity of any regulator is a critical component in ensuring safety. A process of any meaningful approach to regulating dangerous advanced technologies must therefore determine paths to enforcement, as well as the risks and costs posed by each approach (e.g., if a particular protocol requires a certain degree of surveillance in order to track and monitor instances of power concentration or abuse, associated downsides related to the risk of increasingly powerful surveillance must also be considered as part of safety protocol development). After full safety analysis, constraints on development, users, and on use cases should be considered in every relevant sector. Conceptually, this is similar to regulation regarding drug development and the requirement for a prescription to access certain drugs. Regulation is required for both development and use.

4. Irreversibility

If advanced technologies with catastrophic capabilities did not exist, the catastrophes they enable would be avoided entirely. Even when safety mechanisms are established to monitor and mitigate the risks, the world must forevermore carry the burden of the potential for catastrophe should the mechanisms fail. For this reason, it is important to acknowledge an irreversible direction of increasing fragility any time technology with potentially catastrophic capabilities is developed.