The Government Will Pay for It

I am addressing the government that never bought the thing, with my shoes off because my feet hurt. I have spent enough evenings waiting for them to stop. Now I am trying to understand what sort of customer I was helping to finance.

You did not attend an auction. No sale took place. I want that settled before somebody puts a flag behind the story and starts selling tickets to the betrayal. What interests me is that you could be imagined as the customer so much more easily than I could be imagined as a person you ought to consult.

In 2017, OpenAI needed money. Kevin Roose’s reporting, described by The Next Web, says Greg Brockman and Ilya Sutskever proposed auctioning rights to the company’s future artificial general intelligence to governments. Not a finished machine. Rights to something they hoped to build, with ambitions far beyond a clever answer to a question.

Roose’s account names the United States, China, and Russia as possible buyers. OpenAI disputes that part: it says it briefly considered an international cooperative endeavor and did not consider selling AGI to China or Russia. I have no secret minutes in my sock drawer, and a disputed account does not improve when I pretend otherwise. The proposal was reportedly abandoned after Microsoft funding arrived.

Still, you were useful in the pitch.

According to that account, the idea began as an international coalition to manage the risks. By the winter presentation to staff, it had become a way to raise money, known as the “countries plan.” Two people familiar with the thinking said the men expected governments eventually to seize AGI. Why not sell it while the lab needed the cash?

I admire the speed of that little journey. We begin with the safety of the world and arrive at the immediate expenses of the office. Nobody even has to change the subject. The world will be much safer once this particular organization has enough money to continue deciding what safety means.

Here is where I would have liked you to become difficult. Not sinister. Not patriotic in the manner of a man shouting into an airport television. Difficult in the dull way that makes a salesman wish he had brought someone else to the meeting. What rights? What obligations? What happens if the thing cannot be built? What happens if it can?

I know you buy things. Roads do not appear because a minister has strong feelings about asphalt. Research needs money before anyone knows the result. I am not demanding that every public expense fit in my pocket or be explained by someone holding a wrench. But when a laboratory describes a possible invention as powerful enough to reorder human affairs, buying it is not merely ordering a better piece of equipment. The people whose affairs may be reordered are part of the business.

I am one of them. My qualifications are unimpressive. I live here. If you get the arrangement wrong, I cannot leave the consequences at the office and go home. Home is where some of them would happen.

A customer asks whether the purchase is good for the customer. A government owes answers to people who may have to live with a purchase they had no say in. I do not assume governments answer honestly. Some do not permit the question. That is a reason to take the proposed transfer of power seriously, not a reason to treat every state as another account with a large credit limit.

And I dislike the neatness of countries. It folds everyone into the bidder. The people signing, the people paying, the people being watched, the people who might lose an argument with the resulting machinery: one convenient noun. A flag can cover an astonishing number of disagreements if the room is looking only for money.

Dario Amodei objected in a December 2017 email, according to Roose’s account. He warned that the plan would worsen arms races, potentially enable human-rights atrocities, and undermine OpenAI’s moral legitimacy. Those are serious objections. I am glad someone made them. I do not need to award the man custody of my future in gratitude.

Because the comforting version of this story is about the good person inside the company who stood up to the bad idea. I understand the appeal. I have depended on a decent person at a desk more often than I care to remember. But if the rest of us need the right employee to win the right internal argument before enormous power changes hands, we have been given a very poor arrangement, even on the days when the employee wins.

I don’t want my protection to depend on two executives still getting along. Nor would their hatred guarantee it. They can hate each other sincerely and still leave me outside every room that matters.

One person who heard the presentation told Roose it sounded like a wacky start-up pitch ending with, “We’ll make the government pay for it.” I keep hearing that we’ll. Such a confident little gathering of people. You, the government, are the thing they will persuade. I, farther down the line, am one of the people you already know how to collect from.

When I needed wages, I had to supply hours. The hours had to occur. I could not promise a future version of myself capable of sorting every letter in America and ask the post office to fund my present difficulties on that basis. Perhaps this reveals a failure of imagination. Mine was limited by having to be at work again that evening.

An abandoned proposal can still show me the part I would have played in somebody’s successful pitch. No need for my opinion about the machine. No need to find out what I was afraid of losing. My participation could be arranged through payroll.

I know the collection machinery from ordinary paydays. You already have my name spelled correctly. The wages arrive with the tax gone.

Source: OpenAI’s 2017 plan to auction AGI to China and Russia, per The Atlantic