Yeah, I have to agree that you really start to see diminishing returns. Going from Koss PortaPros to HD595's was a huge change and I spent most of my free time the next week listening to music. I just upgraded again to HifiMan HE500's and thought they sounded better, but my listening experience is more or less unchanged. Worse, they are quite heavy planar magnetic headphones and I don't enjoy wearing them as much. Maybe I need a better headphone amp, but part of me is convinced I might have hit my ceiling for my listening perception. Maybe I should have just gotten those Grados...
I have a similar setup than you do and bought the schiit modi and magni (~200$ dac and amp). I'm very happy with the purchase and recommand it as a first dac/amp.