On 2015-03-05 19:05, Gareth Oakes wrote:
> Querying 2.4PB (2400TB) of data is going to take a
> long time regardless of how you do it.
The best way is probably not doing it, where possible - e.g. project a
subset and query that.
[...]
> MarkLogic is marketed as petabyte-scale,
> but XML may not be the most optimal representation of the data, it is
> worth considering the alternatives.
Luckily neither MarkLogic nor most other XML-native databases store XML;
rather, they store an internal representation that can be mapped to the
XPath and XQuery Data Model (the XDM).
A common mistake people make (I'm not saying you're making it, but
people reading your post might jump to an incorrect conclusion) is
thinking that XML databases store XML (e.g. store pointy brackets),
which is a bit like thinking that relational databases operate on CSV
files.
> Distributed storage and query are probably the way forward,
Depending on the nature of the queries, that does seem likely.
--
Liam Quin, W3C
XML Activity Lead;
Digital publishing; HTML Accessibility
