My programming knowledge base is strongest with C++ and Python. I always start writing projects in Python. Then if psyco hasn't gotten me the performance in the (often very small) sections of code that do the heavy lifting a quick C++ extension is whipped up.
This method is so productive it always surprises me how quickly things get done!
It's the best of both worlds, really. You write everything in a language that's fast to write, then you write the bottlenecks in a language that's fast to execute.
This works only if your bottlenecks are a small portion of your work. For example, if your twisted application is slow, you will have a hard time optimizing it by writing a bit of C code.
Because of the python <-> C marshalling cost. Let's say you have some code which uses a lot of small objects which interact with each other, maybe each object has a small list with a few integers. Replacing this in C would make sense because you can be much faster than python. But you still have the cost of interfacing with python (creating the objects, boxing integers back and forth, etc...).
That's why something like numpy manages to be so fast if used well: it internally uses efficient (C-like) representation, but as soon as you need to interact with every item in your array, then it is not only much slower, but even slower than using standard python containers because of this cost.
What language do you think CPython's internal data structures are implemented in?
I know when I write an XS extension in Perl, though, that getting the C struct associated with my Perl object requires one C function call -- no overhead at all. Passing an object to Perl involves allocating memory and adjusting the pointers. All very fast. If this sort of thing is your bottleneck, it's time to step back and rethink what you are trying to achieve.
Being implemented in C does not make things fast. Integers are implemented in C in python, but they are really slow, because of the boxing/unboxing thing (you need to chase one pointer pointer to get the actual C int from the Python object PyObject_Int, and that cost alone is high). Numpy is much faster for this kind of thing because it does not have this cost, and run the core loops + function calls in pure C, without going through the python runtime at all.
Yes, exactly. As you context switch between the high-level and low-level language, you sacrifice performance. If you want to write a fast extension, you don't do this; you let the high-level language call your pointer an object, and when you invoke a function, you invoke it on that pointer. This way, you don't marshal back and forth (except perhaps incidental parameters and results or exceptions).
What I am saying is that the whole concept of doing the heavy logic in the high level language and do the work in the C implementation cannot work when your logic depends on handling many objects.
For example, I wrote a few years ago a small audio app to track frequencies lines in audio. Each track was a list of positions (integers), generally a few tens items max, and I needed to handle millions of such lists. Now, just optimizing the track object to be in C is not that efficient because I needed to access the content of each list in the high level logic.
Basically, If your logic needs to be able to handle those millions small objects, you end up writing almost everything in the compiled language. Abstracting this becomes very difficult.
Oh, python has a nice solution in this problem space - NumPy. But again, this only works in some cases. Unfortunately, the model fails performance-wise once you use non native type - think array of arbitrary precision floats. Haskell being much more powerful and expressive, I would expect it to be able to express those concepts in a more elegant manner, but I could be wrong (I know next to nothing about haskell).
This method is so productive it always surprises me how quickly things get done!