A 16x64 character screen like the TRS-80 needed 1k of RAM. IIRC, the Altair didn’t have that much memory out of the box.

The video memory would either be in the computer or the terminal. At least if it were in the computer, you could use it for other stuff.