Unless you cannot use GPL, I suggest to use x264 for encoding. It is simply the best there is. There is nothing similar to it.