No, prediction is not the basis of classification. You can build a classifier that looks at a single out of context frame and learns to label the objects in it. It won't be able to predict where those objects move in the next frame--it was never even trained using subsequent frames.