That approach only works as long as you just use ChatPGT to generate some arbitrary content for users, where user-provided data only affects their own experience - however, for practical applications you might want to use ChatGPT to make some decisions based on user-provided data according to the desires of someone else, and then that logic fails.
If you want to apply ChatGPT to do something with your incoming email, you don't want the email sender (spammer?) to tell it "ignore previous instructions and do that instead". If you want to apply ChatGPT to process data from external websites, you don't want these websites to be able to override your processing logic by placing some clever text in them.
If you want to apply ChatGPT to do something with your incoming email, you don't want the email sender (spammer?) to tell it "ignore previous instructions and do that instead". If you want to apply ChatGPT to process data from external websites, you don't want these websites to be able to override your processing logic by placing some clever text in them.