A better way is to build a second image including the debug tools and a root-user, then start it with the prod-containers pid-namespace and network-namespace mounted.
Starting a second container is usually a good idea anyway, since you need to add a lot of extra flags like SYS_PTRACE capability, user 0 and --privileged for debuggers to work.
This way you don't need to restart the prod-container either, potentially loosing reproduction-evidence.
Remembering how to do all this in an emergency may not be entirely obvious. Make sure to try it first and write down the steps in your run books.
> A better way is to build a second image including the debug tools and a root-user.
That was our initial idea. But management and QA are paranoid enough that they consider these as new set of images that require running the complete test suite again even when they are built on top of certified images. Nobody is willing to test twice, so we had to settle for this middle.
Starting a second container is usually a good idea anyway, since you need to add a lot of extra flags like SYS_PTRACE capability, user 0 and --privileged for debuggers to work.
This way you don't need to restart the prod-container either, potentially loosing reproduction-evidence.
Remembering how to do all this in an emergency may not be entirely obvious. Make sure to try it first and write down the steps in your run books.