In most cases just switching the color scheme will slow you down. So going by that, if you spend enough time in dark mode, switching to light will also be slower.
Also, you can't just reuse the same colors for syntax highlighting in dark mode. Saturation needs to be turned down compared to light mode or everthing looks like a mess of colors.
