Yes, exactly. As you context switch between the high-level and low-level language, you sacrifice performance. If you want to write a fast extension, you don't do this; you let the high-level language call your pointer an object, and when you invoke a function, you invoke it on that pointer. This way, you don't marshal back and forth (except perhaps incidental parameters and results or exceptions).
What I am saying is that the whole concept of doing the heavy logic in the high level language and do the work in the C implementation cannot work when your logic depends on handling many objects.
For example, I wrote a few years ago a small audio app to track frequencies lines in audio. Each track was a list of positions (integers), generally a few tens items max, and I needed to handle millions of such lists. Now, just optimizing the track object to be in C is not that efficient because I needed to access the content of each list in the high level logic.
Basically, If your logic needs to be able to handle those millions small objects, you end up writing almost everything in the compiled language. Abstracting this becomes very difficult.
Oh, python has a nice solution in this problem space - NumPy. But again, this only works in some cases. Unfortunately, the model fails performance-wise once you use non native type - think array of arbitrary precision floats. Haskell being much more powerful and expressive, I would expect it to be able to express those concepts in a more elegant manner, but I could be wrong (I know next to nothing about haskell).