RE: binary formats

Hi all,

 

SAP’s most important consumers of OData services are browser-based _javascript_ apps, and the most critical performance indicator is the time it takes for the first results to show on the UI. Besides server-side
 data acquisition this time includes sender-side packing + network transfer + receiver-side unpacking.

 

All relevant browsers have built-in GZIP (de)compression and JSON.parse(),  so gzipped JSON is the baseline. A binary format that wants to compete has to be faster end-to-end, i.e. the time saved by sending less
 over the wire has to overcompensate the time invested to pack and unpack the data. These considerations obviously went into the choice of GZIP as the predominant compression method, see
http://tukaani.org/lzma/benchmarks.html for a comparison of compression algorithms in terms of speed and memory footprint.

 

We do have one pain point with the current JSON format: sending thousands of entities with hundreds of properties using DescriptivePropertyNamesExactlyDescribingTheSemanticsOfTheirPossibleValues. Here the ratio
 between data and “package material” is getting rather low, especially if many of the properties have their default value. While GZIP does a decent job to deflate this kind payloads, the time spent by on server to zip plus the time spent on the client to first
 unzip and then parse the large text stream into _javascript_ objects is quite noticeable.

 

A more compact JSON format for “tabular” data as initially proposed in
http://www.odata.org/blog/an-efficient-format-for-odata/ might help, but OData’s flexibility with $expand, dynamic properties, and inheritance don’t make that as straight-forward as it might
 seem. We’ve experimented with “Recursive JSON”, 
http://www.cliws.com/e/06pogA9VwXylo_GknPEeFA/, which addresses the problem of long property names and deals with the flexible structure. Combined with omitting properties with default value,
https://issues.oasis-open.org/browse/ODATA-818, this might take us far.

 

Thoughts?

 

Thanks in advance!

--Ralf

 

 

From:  [mailto:]
On Behalf Of Ireland, Evan

Sent: Dienstag, 29. September 2015 20:25

To: Mark Stafford <>; Mark Biamonte <>; 

Subject: [odata] RE: JSON as default format

 

Mark,

 

I am not so sure of the shift to binary formats that you predict.

 

I did some “real data” comparison of SAP enterprise data using JSON vs. BSON format, and found BSON to take up more space. Binary formats that encode lengths can chew up more space than text-based
 formats. (Similar to CORBA GIOP, where you have a bunch of fixed-length (32-bt) length fields, as well as padding, ibn the binary encoding).

 

Now for some ODaata services I am toying with, HTML is the default format. Not a standard format for OData, but very useful for system administration tasks like system monitoring (logs, metrics,
 etc).

 

I think if the client doesn’t send an Accept header or specify $format in a URL, then all bets should be off. I suppose I am saying let the server decide a default, if even it has one rather than
 insisting on finding format in URL or Accept header.

 

From:  [mailto:]
On Behalf Of Mark Stafford

Sent: Monday, 28 September 2015 11:26 p.m.

To: Mark Biamonte <>;


Subject: [odata] RE: JSON as default format

 

Typically we say that the default is up to the server. The server only needs to support one of the standardized serialization formats – but since JSON is the only format that is fully standardized at this point,
 we typically expect JSON to be the default response.

 

I can also say that I *hope* it’s not put down somewhere in the standard as the default. I personally believe we’re at the height of the maturity curve for JSON, and I think as HTTP debugging tools continue
 to rapidly improve, that we will see a shift in the default serialization format from JSON to a binary format, similar to what we’re seeing happen with HTTP2. So in my personal ideal future state, we would see something like Avro take over as the default serialization
 format for OData payloads – but of course that depends upon the ability of the server to choose the right default for the API.

 

Thoughts? Pushback?

 

From:  [mailto:]
On Behalf Of Mark Biamonte

Sent: Tuesday, September 29, 2015 4:06 AM

To: 

Subject: [odata] JSON as default format

 

I know that the default format for services in OData v4 changed to be JSON, but I am having difficulty in the spec finding where it explicitly states that.  I would have expected to find something in the definition of the accept header
 and the $format query parameter.  Something along the lines of if the accept header and $format query parameter are not present then the JSON format is used.

 

Mark