Have we hit the limit of the analogy, or have we hit a limit in our understanding? Both neural networks and actual brains have behaviors that emerge from the interactions of smaller components. Neural networks have trivial connections compared to brains, but our understanding of the emergent behaviors seems very limited. To me, this is a sign not that the analogy has reached a point of breaking down, but that our tools aren't sufficient to work on even then trivial connections. I do expect the analogy will break at some point, but I'm not sure we have reached that point yet.
I hoped neuroscience, as a field, was on the cusp of a physical theory of learning and memory. I dreamt of an intersection of information theory, neuroscience, and ML.
Alas, state of the art in neuroscience / neural engineering is closer to bloodletting than a mechanistic theory of learning and memory.
