A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks
B. It gets to own an unbounded financial claim on the resulting surplus
All the technical stuff about AI, AI alignment, etc can be rolled up into point (A) above. My claim is that having point (A) on its own, without point (B) is probably okay. Nuclear technology and bioweapon technology both approximate (A) and they are mostly okay because without (B), there isn't an incentive for people controlling them to push their luck on safety.
The key claim of this post is that we can probably get rid of most of AI risk without doing anything other than separating out the bookkeeping, physical footprint and institutions so that there is no single org with both properties. And with a little help from ASICs, maybe we can also have a very productive AI industry that actually delivers most the benefits of AI to boot. No busy-waiting pause, no banning AI , etc.
What a typical AI disaster scenario currently looks like (e.g. those by Daniel Kokotajlo):
Obviously, the humans made a mistake in letting the AI company write the IOUs to them. But once there's a system for stocks, banking, money, etc, letting people write you IOUs is the default and it's hard to stop.
So, here's how I would separate the two. Create two new kinds of entity:
AI R D orgs would do everything that involves training, distilling, testing, and aligning frontier AI models. They would be heavily legally restricted and unable to issue equity. Maybe they could get debt financing, I will have to think more about this.
AI deployment companies would do all the consumer-facing stuff, or B2B work. They would run inference at scale. They would handle the UI/UX, and also anything that doesn't require gradient descent, so they could organize AI agents into swarms, attach databases, etc. Some AI deployment companies could sell AI hardware direct to consumer or integrate them into robots and laptops.
AI deployment companies would be able to issue equity and they would pay the AI research and development organizations for the models. Also of note: these AI deployment companies would potentially have a lot of intimate knowledge about the real world and real people, either via the web interfaces or perhaps customer data from robots. The AI R D orgs would not be allowed to have that kind of data. This makes an global AI takeover from an AI R D org much harder, because the organization as a whole simply doesn't have this goldmine of blackmail material and intel on people just sitting around for the taking.
Now fortunately the technology underlying AI allows us to enforce this distinction at a hardware level - an AI R D org would work on a model and once they were done with it and it had passed whatever evals we deem necessary, they would burn it into an ASIC. This would be a fixed-function ASIC whose physical circuitry implements a particular model, the way a human brain has physical connections that implement a human mind. It would have some specific input-output type, such as text in/text out, or video in/motor controls out. That ASIC has some run size R units produced, and those are then sold. Then the model is deleted or encrypted and not allowed to run in the AI R D org any more . In the most paranoid case, one could try to make a run size of R=1 or some small integer. But I am imagining runs of tens or hundreds of thousands.
All high-spec compute that exists outside the AI R D orgs would be in the form of ASICs. So whole categories of AI risk simply disappear. There is no compute overhang that a rogue AI can capture . It is stuck with CPUs or maybe GPUs (though maybe there is a way to change how GPUs work so that they can't run AI models - more to do here).