> I have had to work with large 1GB+ JSON files, and it is not fun.
I had also had to work with large JSON files, even though I would prefer other formats. I had written a C code to split it into records, which is done by keeping track of the nesting level and of whether or not it is a string and the escaping in a string (so that escaped quotation marks will work properly). It is not too difficult.
> I was also contemplating the mentioned formats for another project, but they are hardly usable when you need to store binary data, such as images, compressed data, or simply arbitrary data. Storing binary data as base64 strings seems wasteful.
I agree, which is one reason I do not like JSON (I prefer DER). In addition to that, there is escaping text.
> Random access into these files is also an issue, depending on the use case. Sometimes it would be a nice feature to jump over some data, but for JSON, you cannot do that
With DER you can easily skip over any data.
However, I think the formats with type/length/value (such as DER) do not work as well for streaming, and vice-versa.
I had also had to work with large JSON files, even though I would prefer other formats. I had written a C code to split it into records, which is done by keeping track of the nesting level and of whether or not it is a string and the escaping in a string (so that escaped quotation marks will work properly). It is not too difficult.
> I was also contemplating the mentioned formats for another project, but they are hardly usable when you need to store binary data, such as images, compressed data, or simply arbitrary data. Storing binary data as base64 strings seems wasteful.
I agree, which is one reason I do not like JSON (I prefer DER). In addition to that, there is escaping text.
> Random access into these files is also an issue, depending on the use case. Sometimes it would be a nice feature to jump over some data, but for JSON, you cannot do that
With DER you can easily skip over any data.
However, I think the formats with type/length/value (such as DER) do not work as well for streaming, and vice-versa.