I am reading a book right now called Robopocalypse. It is a cross between World War Z (the book) and The Terminator. It is a faux-historical account of a genocidal war between AI robots and humanity. It is, for the most part, good. I’m a fan of sci-fi and this is well done.
Reading it brings me to a point that both annoys and frightens the hell out of me: the lack of a manual override.
This comes across in sci-fi and real live one of two ways:
- There is no manual override when there obviously should be.
- There is a manual override and it get ignored because… plot.
In the beginning of Robopocalypse, scientists create an AI program. It becomes sentient and decides to wipe out humanity. OK, I’ve seen this plot a number of times before. You find out that the scientists have created this same AI a bunch of times before, it always wants to wipe out humanity, so they delete it and start over. On the last try, it “escapes” the system it was programmed in as a computer virus, etc., etc. Here’s the thing. The scientist in the lab, talking with the AI, has a “fail-safe” that is supposed to shut down the system. The AI compromises the “fail-safe” which means it’s not a real fail-safe.
Now, I’m not a computer engineer, so I don’t think digitally. If you said to me “J.Kb. there is a computer, in an underground lab, not hooked up to the outside world, and we put an infinitely intelligent, homicidal AI in it, and you need to design a fail-safe to keep it from getting out.” You know what I would do? Explosives. Explosives in the machine, attached to det-cort, attached to a firing device on the scientist’s desk. You know why? It is 100% un-hackable.
I see this same crap over, and over, and over again.
I can’t tell you how many TV shows I’ve seen where the plot of the episode is: hacker hacks something and uses it to kill people. This drives me up the wall and onto the ceiling. So the hacker uses his hacking ability to lock the people into the building. Then the hacker threatens to release poisonous gas into the building to kill everyone. First of all, who connects a poisonous gas line to the ventilation system when designing a building? But what really gets me screaming at my TV is WHERE IS THE SAFETY CHECK VALVE? I’ve worked around dangerous chemicals my entire adult life. You know what? When you have a line carrying flammable liquid or toxic gas, there is always, always, always a manual shutoff.
Character: “The hacker has locked us in the building, there is no way out.”
Me: “Seriously? There isn’t one emergency exit that has a manual door push-bar for fire or power outages? That got built to code?”
Character: “OH MY GOD, THE HACKER IS GOING TO FLOOD THE BUILDING WITH HALON!!!”
Me: “THERE SHOULD BE A KNOB ON THE HALON TANK, RIGHTLY TIGHTY LEFTY LOOSEY YOU FREAKING MORON!”
Don’t believe me? That was the plot of the Scorpion episode, Cliffhanger. Except instead of Halon, it was Sarin that was going to be pumped through the ventilation system. No, not kidding. First, who was the HVAC engineer that agreed to plug a container of Sarin into the AC unit? And who was the building inspector that signed off on that without some sort of manual shutoff valve?
As the idea of smart houses becomes more and more real, the idea of a hacker murdering someone with their smart home becomes a plot point. The very excellent sci-fi show Almost Human did an episode on this called Disrupt. If I’m going to have a smart house, there is going to be an good-ol’-fashioned deadbolt on at least one door.
Wife: “Honey, hackers have locked us into the house. They say if we don’t hand over all the jewelry, they will turn up the heat and roast us all to death.”
Me: (Walks over to back door) *click* (walks out of house).
Now, sometimes there is a manual override, and everybody conveniently forgets about it. In that episode of Scorpion, the Scorpion team is running around trying to out hack the hacker before they all die. Again, not being a computer engineer, I’m curious why nobody decided just to trip the breaker on the smart building’s computer? No power, no hacking.
This was infuriating in Die Hard 2 when the terrorists raised the floor on the airport ground approach system to crash a plane. Here’s the thing; airport runways are equipped with Visual Approach Slope Indicators. These are lights that can only be seen at certain angles, so a pilot knows that he is on the right approach angle and how far from the runway by the lights visible from the cockpit. These lights are bright enough to be seen in any weather, like an airport lighthouse. This system is designed so that no matter what the autopilot does, the pilot can still look out the window and see that he is approaching the runway correctly and judge the distance to ground. Except in the case, of Die Hard 2, the pilot was asleep at the stick or something, for purposes of plot. This many not exactly be a manual override, but it is a way for a pilot to land with all his instruments out.
Yes, I know. I’m ranting about books, TV, and movies. If I had my way, there would be no plot and so I have to accept this for the sake of the story. Except that life is imitating art.
A pair of hackers (security consultants) figured out how to hack a Jeep using the internet comparable entertainment system. Using the auto-park, they can control the steering and breaks. They could pretty much drive the car remotely. This was right out of the The Sontaran Stratagem episode of Doctor Who, except the Sontarans use the remote control of carts to kill people. With Drive-by-Wire tech, it is even easier to control the car by hacking the ECU.
But forget hacking for a second. My dad had a 2015 Mercedes with drive-by-wire. A piece of construction debris fell off the back of a truck in front of him on I-95. He ran it over. Killed the car. At 60 MPH he had no steering. The engine shut off and he lost his drive-by-wire and electronic power steering.
I had my 2003 Chevy truck cut out on me once while at speed. Clogged fuel filter. I lost engine power. I could still steer and break. Steel on steel on steel on rubber on asphalt. No power meant that I lost the hydraulic assist to my power steering, but the mechanical linkage was still there. Manual override.
BMW has no manual override in its security system. When you lock a BMW from the outside, it is locked. If you are inside, you are trapped. The door handles, lock buttons, they don’t do anything without the car unlocked. The ECU cuts control and there is no mechanical linkage to manually unlock the car. If a grown woman can die from being locked inside a car, you know you made an unsafe design.
I have two gun safes. One has a nice digital lock on it for quick access. The other has an all mechanical S&G lock. My collection is split between the two safes, so that no matter what happens, I still have access to guns by spinning a dial. You can’t hack or EMP gears and tumblers.
Red dots are getting cheaper and getting better everyday. Optics make shooting easier and have made getting into shooting easier. But batteries die, glass cracks, and sometimes electronics don’t handle cold weather. I’m not going to deride optics, but all my defensive guns have irons on them.
It saddens me that so many new guns come with such terrible iron sights, especially .22 rifles. Shooting with irons is an invaluable skill. If you can hit with irons, you can hit with optics, the reverse is not true. A fundamental skill of every shooter, especially every shooter who keeps a gun for self defense is the ability to hit a clay pigeon size target at, at least 15 yds with a pistol and 50 yards with a rifle using iron sights. I know it’s not tacticool, but if Sergeant Alvin York could capture a German battalion with a bolt gun and peep sights, you should be able to do a little better than just hit the broadside of a barn with the same.
I know. By know I’ve ranted myself way off course. I get that digital technology is great and all. It has made our lives better in many ways; and as an engineer, it would be hypocritical to not embrace new technology. But as good as digital technology is IT NEEDS TO COME WITH A D—N MANUAL OVERRIDE!!!
There was soooooo much wrong with Die Hard 2 from an aviation perspective. I won’t go into all the pilot nerd details, but the Instrument Landing System (ILS) can’t “change the ground minus 200 feet” with the click of a touch screen pen. The Glide Slope part of the ILS is a set of antennas that emit 2 signals in a fixed direction from a ground station located at the approach end of the runway. Now, a more believable, but still deeply flawed, plot device would have been for General Stewart and pals to give Captain Miles O’Brien a bad altimeter setting. Further, 200 feet is the minimums for most Category I ILS approaches and in those days there would have been an “Inner Marker Beacon” that would have indicated they were reaching 200 feet no matter what their altimeter had said. Also, a 200 feet difference is not enough to cause the catastrophic crash that was was depicted as the aircraft would have easily been able to flare enough into a hard(er than normal) landing. Also, there would have been almost no post impact fire even if the plane did crash but was “dry as a martini” on fuel.
If you really want the rest of that aspect of the movie roasted, just ask. I’ve got more.
A better solution is to turn off the airport’s ILS and spoof the aircraft with a fake beacon. That’s how Ron Brown was killed.
RE: BMW, Mercedes, et al Three words: Automatic. Center. Punch.
About $15 at Home Despot, more if you buy a Starrett or Brown & Sharpe from a high end tool seller. Works the same. Used by automobile burglars all across the country and those who drive on frozen lakes (Minnesota, Wisconsin, Canuckistan, etc.) and canal-infested terrain (Florida, Louisana, etc.). One press-and-click shatters side window safety glass.
If you’re not smart enough to figure out your car will entomb you (RTFM), too stupid to purchase and use an automatic center punch (or secure it well near the center of the vehicle where it can quickly be found in an accident rather than buried in the so-called glovebox) or are too rotund to climb out the window. just relax; you won’t be missed, at least not by very many, and having the body remain in the vehicle after sinking makes things easier for the cops and coroners who will (eventually) recover the vehicle. Not to mention it absolutely destroys the plot of poorly-written fiction.
Speaking of manual overrides – who was the engineer in charge of designing the nuclear bomb’s 6′ manual detonation cord used in Armageddon?
The computer thing is always too much for me. If the computer is physcially isolated and connected to nothing but power it can’t communicate to the outside world. So pretty simple, it’d be like locking the AI in isolation and the only contact it has are the researchers. Of course the human element is the downfall here and if they are motivated for any reason to let the AI out or make a mistake then all bets are off.
I’m sure someone will figure out how to communicate over an AC power connection if they haven’t already done it.
Another big problem with security in the internet of things is no security measusre is 100%. And this is very troubling when you consider that medical devices and cars are being Internet connected more and more and people have already figured out how to hack a plethora of them, insulin pumps and Jeep and BMW are two prominent examples of this.
Really scary will be when we get near some ghost in the shell stuff where there are internet connected implants of varying degrees.
“I’m sure someone will figure out how to communicate over an AC power connection if they haven’t already done it.”
Done and done…
http://www.amazon.com/ZyXEL-Ethernet-Powerline-Wall-Plug-PLA4111/dp/B00D7GF4XC
Also RE: DH2
Didn’t part of the plot include the complete shutdown of *ALL* ground lights?
(it’s been many moons since I watched it…)
I completely forgot about those; a coworker outfitted his living room with those to avoid running wires!
Although I must admit, what I envisioned was something more like the pc’s power supply sending blips back up the pipe sort of thing.
My dad used those a few times. Problem is you need something that can communicate on both sides, and even if an AI could figure out how to produce some sort of signal by changing its own power draw, it needs to have something on the other end to communicate with.
And then that can all be solved by keeping it on its own power grid. Let’s see how far a hostile AI gets when it’s dependent on a 4-gallon gasoline tank for life support.
Amen.
Well, as everyone knows, “Hacking” = “Typing REALLY Fast”, and the Manual Override always tends to be broken (or in a really inconvenient place).
You mean like this?
As an aside, this also shows the sheer folly of “smart” guns (actually they are dumb guns).
Batteries, hacking, debris, interference, glitches, updates, EMP, jamming, use by friendlies, and so makes me want to smack people who even suggest it.
My proof is that since my iPhone cannot 100% of the time pair with my Fitbit, we never trust such fanciful crap to solve a minor problem of unauthorized users.
I have a zombie story you might like, Jay.
http://www.xkcd.com/734/
I remember a story from decades ago that Frank Sinatra watched a friend of his die in a car fire, because it was a then-fancy car with electronic door locks. The electrical system went out in a collision, locking the man in a burning car. (I believe it was a Jaguar.) This supposedly was one of a series of unfortunate incidents that led to the re-emergence of manual door locks on cars that had previously been equipped only with electronic locks.
Re: Bad robots.
What happens when these killing machines run out of fuel, energy, ammunition, or break down and need maintenance? Robots have to be a lot smarter than simply being programmed to kill if they’re going to take on any substantial number of humans, who eat, know how to reload their weapons (and where to get ammo for their guns – fighting robots’ chain gun ammo and anti-personnel missiles aren’t so easy to get hold of), and how to fix each other up to get back in the fight.
Sci-fi:
Look up an old movie called “Forbin: The Colossus Project.” One of the earliest, and still one of the best “computers take over the world” movies. Colossus’s message, “There is another system,” still sends a chill through me every time I watch the movie. (FYI, my computer is named “Colossus” after the computer in this film. Every now and then, it tries to take over the world. I just reboot him and he’s fine.)
Thanks for the info on the book, I read it in a day after seeing this post…good stuff!
As for the whole AI escape aspects…I think any super-intelligence would be able to escape regardless of safeguards.
I use the analogy: take the dumbest functional human in the world and have them design a prison, then put the smartest person in the world and have them try to escape.
Another good one is…imagine if bees were trying to destroy humanity…they might propose “ah, lets eliminate their nectar supply and attempt to kill their queens”…
The issue is we design safeguards based on what we think and how our lives, biologists, and societies function…an intelligence far beyond a person would likely have an escape method that we wouldn’t consider.
A far worse thought…the idiots of the world have often been convinced to do things against their long term best interest by the smarter folks…why is it inconceivable that a 9-10 (or even 100+ sigma) human-IQ equivalent AI wouldn’t be able to convince the scientist to simply open the cage?
PS, check out Bostrom’s book on the topic of superintelligence