> "Oh, we're slowly building up a custom, buggy, binary framing protocol. Lets just use protobuf/msgpack"
By using protobuf/msgpack you lose the ability to precisely control the layout and encoding of your data on the wire. Most applications don't care, but this results in your wire representation being defined by "whatever protobuf says".
Say I want to transmit an unsigned 16 bit integer with protobuf. How do I do that? The documentation doesn't include 16 bit integers as a datatype, so I'd probably have to wrap it in 32 bits and/or use some varint stuff. It would be simpler to just write a big endian 16 bit int though.
I wish there was a simpler alternative to protobuf that gives more control to end users and doesn't try to be smart. Until then, making your own binary protocol is not over-engineering.
It might be different if you have to talk to an ASIC that cannot understand protobuf, or send billions of values at line speed. But generally I don’t have to care anymore whether a number is sent in exactly sixteen bits, for the same reason I long since stopped caring about message framing or parity or run-length limits or 0-5 vs ±12 V busses. Expressing any of those constraints takes more effort than letting the machine use the commonly-supported default.
It wouldn't be ad hoc per se, basically you would have a set of guidelines on how to transmit data and that by itself would be a standard.
Something like "use fixed length 8/16/32/64 bit signed/unsigned integers in big endian, length prefix can be 8/16/32 bits, bool is 1 byte (00 = false, 01 = true)" etc, without extra stuff like varints or bit packing, which a lot of current formats are doing.
In short, just use the most straightforward way of encoding while also using the least amount of data. Big endian for ints is very common, simple and relatively compact if you only use the bit width that you need.
I agree; sometimes writing your own binary format is the right call. To my point upthread, the trick is knowing when that’s the right choice and when it’s better to use protobuf or something standard. (Or, when to just stick to json).
Developing good instincts for this stuff takes a lifetime.
By using protobuf/msgpack you lose the ability to precisely control the layout and encoding of your data on the wire. Most applications don't care, but this results in your wire representation being defined by "whatever protobuf says".
Say I want to transmit an unsigned 16 bit integer with protobuf. How do I do that? The documentation doesn't include 16 bit integers as a datatype, so I'd probably have to wrap it in 32 bits and/or use some varint stuff. It would be simpler to just write a big endian 16 bit int though.
I wish there was a simpler alternative to protobuf that gives more control to end users and doesn't try to be smart. Until then, making your own binary protocol is not over-engineering.