He means capturing things that benchmarks don't. You can use Claude and GPT-5 back-to-back in a field that score nearly identically on. You will notice several differences. This is the "vibe".

I would assume that it is testing how well and appropriately the LLM responds to prompts.