I'd recommend you try "AddKeysToAgent confirm", then you'll get a prompt to approve the key usage each time its used. (Can be annoying for usage with automation like ansible).

Having a forwarded agent was how matrix recently got hacked, confirmation is a decent work around if you need to forward your agent.