Can we mathematically prove that an AI system is safe?
Distant horizon
Why it matters
Wrong outputs from neural networks can have costly consequences, and tests alone may not show that an AI system avoids dangerous behaviour. Mathematical proofs could give strong safety guarantees, like the high standards already required for critical infrastructure.
What we know
Formal verification already proves the correctness of some critical software.
What blocks us
Neural networks are far too large and opaque for current proof methods.
Current leads
Verifying small parts or outputs, AI that writes its own checkable proofs.
Sub-questions
How can a vague idea like 'harm' be turned into a precise rule that a computer can check?
Can we build models of the world accurate enough to prove what an AI's actions will cause?
Can AI systems be trained to write the safety proofs that are now too much work for human experts?