Yes, but a person wielding a knife has morals, a conscience and a choice, the fear is that an AI model does not. A lot of killer AI science fiction boils down to "it is optimal and logical that humanity needs to be exterminated"; no morality or conscience involved.

Which is why there are laws around what knives are allowed and what are banned. Or how we design knifes to be secure. Or how we have a common understanding what we do with knifes - and what not. Such as not giving them to toddlers... So what's your point?

The point is not the tool but how it's used. "What knives are allowed" is a moot point because a butter knife or letter opener can be used to kill someone.

But if you give a very sharp knife to a toddler and say "go on, have fun" and walk off, you're probably going to face child endangerment charges at some point.