Technology is famously hard for the law to keep up with, but I don’t think that’s what’s going on here. We have laws for when a person uses a technical tool to obtain unauthorized access, exfiltrate data, or do other malicious things. These don’t necessarily require any specific intent. If you’ve set up an AI tool and pointed it in the direction of hacking without proper airgapping, you’ve probably violated the CFAA if you’re in the US when it’s successful. Other countries generally have similar laws. Prosecutors don’t need to try someone if it’s really ambiguous or unclear what they actually did. Is this really unreasonable under the legal system?

To the extent there’s a gap, it’s the system around the law. Lawyers and judges aren’t technical experts, and may not recognize “it wasn’t me, it was the ai” as the same argument as “it wasn’t me who shot him, it was my gun”, in that it’s usually foolish and unserious, but there can be a convoluted scenario in which it’s a good argument.