A claim that AI has entered a self-improving phase is not proof of a breakthrough, but it does spotlight the next hard question for defenders: how much autonomy should an AI system be allowed to have?