I am not suggesting that we change the filtering by added_after at all. I personally believe you are correct in thinking that one could paginate by added_after in most circumstances. I also think an added_before would be a nice addition.
The only problem I have heard about pagination in regards to using a date added value is you could run into a problem if large amounts of records were added in a single transaction and that transaction used the same "date added" for every record in the transaction.
One way some systems might address this is to use microsecond precision for the date added value and then try and ensure each record gets its own timestamp or that no more than X records use the same microsecond timestamp. This should give enough room, even under heavy load. But I am sure as soon as I say this, people will have an example where that will even fall over.
Bret
From: Jason Keirstead <>
Sent: Monday, September 17, 2018 10:29:52 AM
To: Bret Jordan
Cc:
Subject: [EXT] Re: [cti-taxii] New Work - Pagination
Can we outline the problem sets / challenges with item-based paging, so everyone is on the same page WRT the discussion?
As well, currently you can already "page by date" in TAXII (like you could in TAXII 1) - so in effect we have two mechanisms already. Are we talking about changing that as well?
- Jason Keirstead Lead Architect - IBM.Security
www.ibm.com/security
"Things may come to those who wait, but only the things left by those who hustle." - Unknown
From: Bret Jordan <>
To: "" <>
Date: 09/17/2018 01:11 PM
Subject: [cti-taxii] New Work - Pagination
Sent by: <>
All,
Now that TAXII 2.1 Working Draft 03 is up for Committee Specification Draft 01 ballot, I would like to start requesting proposals for how we should address and solve pagination for TAXII 2.1. As you know, we tried to use HTTP's native "Item" based pagination in TAXII 2.0, but after doing additional research and trying to implement this, we discovered that not only is this a known "do not do for RESTful APIs" but it is also near impossible to implement in code given the STIX dataset.
I know several of you have expressed ideas for how this could be done/fixed in TAXII 2.1, so I am asking as one of your sub-committee co-chairs to please write up your ideas and submit them to the TAXII email list as proposals.
Thanks
Bret
