Maciej Cieglowski’s wonderful, and at times hilarious, talk on super intelligence which he gave in 2016. Yes 2016. It’s a long read, but worth every second of your time.
On intelligence level:
Our intelligence level, cognitive speed, set of biases and so on is not predetermined, but an artifact of our evolutionary history.
In particular, there’s no physical law that puts a cap on intelligence at the level of human beings.
A good way to think of this is by looking what happens when the natural world tries to maximize for speed.
If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster.
But of course we know that there are all kinds of configurations of matter, like a motorcycle, that are faster than a cheetah and even look a little bit cooler.
But there’s no direct evolutionary pathway to the motorcycle. Evolution had to first make human beings, who then build all kinds of useful stuff.
So analogously, there may be minds that are vastly smarter than our own, but which are just not accessible to evolution on Earth. It’s possible that we could build them, or invent the machines that can invent the machines that can build them.
There’s likely to be some natural limit on intelligence, but there’s no a priori reason to think that we’re anywhere near it. Maybe the smartest a mind can be is twice as smart as people, maybe it’s sixty thousand times as smart.
That’s an empirical question that we don’t know how to answer.
On general intelligence:
The concept of “general intelligence” in AI is famously slippery. Depending on the context, it can mean human-like reasoning ability, or skill at AI design, or the ability to understand and model human behavior, or proficiency with language, or the capacity to make correct predictions about the future.
What I find particularly suspect is the idea that “intelligence” is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture.
With no way to define intelligence (except just pointing to ourselves), we don’t even know if it’s a quantity that can be maximized. For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.
Or maybe it would become obsessed with the risk of hyperintelligence, and spend all its time blogging about that.
On AI cosplay:
The most harmful social effect of AI anxiety is something I call AI cosplay. People who are genuinely persuaded that AI is real and imminent begin behaving like their fantasy of what a hyperintelligent AI would do.
In his book, Bostrom lists six things an AI would have to master to take over the world:
- Intelligence Amplification
- Strategizing
- Social manipulation
- Hacking
- Technology research
- Economic productivity
If you look at AI believers in Silicon Valley, this is the quasi-sociopathic checklist they themselves seem to be working from.
Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election.
Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don’t like to be manipulated. You can’t tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
I’ve even seen people in the so-called rationalist community refer to people who they don’t think are effective as ‘Non Player Characters’, or NPCs, a term borrowed from video games. This is a horrible way to look at the world.
So I work in an industry where the self-professed rationalists are the craziest ones of all. It’s getting me down.