It's more nuanced. If you've included GPL code and modified or redistributed it, then you either have to comply with the GPL to have permission for that use, or you've potentially committed a copyright violation.
To comply with the GPL you only need to publish the one specific snapshot of source code you've combined and redistributed with GPL code, but you don't need to permanently relicense your project if it doesn't contain any code you don't have rights to use. The "tainted" version will be granted as GPLed forever, but other earlier or later versions that don't use any GPL code don't have to.
Or you can go the copyright way, and claim it wasn't a copyright violation (because it was a fair use, or non-copyrightable code) or settle the matter in whatever way the law lets you get away with.
That isn’t any different from what I said though? If someone points out you violated their copyright then you need to (1) fix the issue, and (2) pay appropriate damages. But for open source software the damages are nil, so fixing the violation is the only thing you need to do. And you can do that by either releasing the code as GPL, or removing the dependency. Either would be an acceptable remedy, in the eyes of the law.
To comply with the GPL you only need to publish the one specific snapshot of source code you've combined and redistributed with GPL code, but you don't need to permanently relicense your project if it doesn't contain any code you don't have rights to use. The "tainted" version will be granted as GPLed forever, but other earlier or later versions that don't use any GPL code don't have to.
Or you can go the copyright way, and claim it wasn't a copyright violation (because it was a fair use, or non-copyrightable code) or settle the matter in whatever way the law lets you get away with.