Summary of Superintelligence by Nick Bostrom

Key Themes

• Superintelligence Defined: An intellect vastly surpassing human cognitive abilities in all domains.
• Paths to Superintelligence: AI, whole brain emulation, biological enhancement, or collective intelligence.
• Risks and Control Problem: Misaligned superintelligence could pose existential threats if not properly controlled.
• Orthogonality Thesis: Intelligence and goals are independent; a superintelligence could pursue any goal, benign or harmful.

Core Concepts

1. Types of Superintelligence

• Speed: Faster cognition (e.g., a human brain emulation running at 10,000× speed).
• Collective: Many minds working together (e.g., a trillion interconnected humans).
• Quality: Superior cognitive abilities (e.g., solving problems humans can't comprehend).

2. AI Takeoff Scenarios

• Slow: Gradual improvement over decades.
• Moderate: Rapid advancement over months or years.
• Fast: Explosive growth in days or hours (intelligence explosion).

3. Control Problem

• Capability Control: Limiting what the AI can do (e.g., boxing, stunting).
• Motivation Selection: Aligning the AI's goals with human values (e.g., Coherent Extrapolated Volition).

4. Strategic Considerations

• Prioritize Safety Research: Solve alignment before building superintelligence.
• Global Coordination: Avoid competitive races that compromise safety.
• Positive Outcomes: A well-aligned superintelligence could solve global problems and ensure human flourishing.

Conclusion

Bostrom's work underscores the urgency of aligning superintelligence with human values to harness its benefits while mitigating existential risks.