Re: [STIX] STIX in JSON

Great question Chris, and very timely.

As many of you know, I have been beating the drum for JSON support for a long time, nearly 18 months now. I seem to get several private emails a week from the community asking about it. Even Facebook is using JSON for their ThreatExchange platform.  

Where we stand today, to the best of my knowledge is:

1) STIX 1.2.1 will be ratified with OASIS using the current XML binding as the only reference implementation.  However, the STIX team as always maintained that you can do your own reference implementation.  And to that end several companies and groups have done just that.

2) There is support in the current Python APIs for a to_dict() to make it JSON.  I believe Soltra is using this on the back side as they store their STIX data in JSON format.  However, this to_dict() it is not a standardized form of JSON.  It was just put together by the developer of the API and some of the constructs are more XMLized JSON.

3) Several vendors and startups are doing their own STIX based JSON implementations for all of the reasons I have brought up in the past.  Intelworks is one of the few that have made public announcements about it.  The rest are still not public and thus I can not speak about them.  

4) JSON based TAXII is done and the specification can be found here [1].  I worked on this specification with lots of people and it has gone through several revisions in the TAXII community.  I even have fully working APIs for JSON based TAXII.

5) I am hoping that for STIX 2.0, we can have an official JSON binding for STIX that everyone can use.  But in the mean time, I would contact Intelworks and do what they are doing.  My project, FreeSTIX [2] is going to follow what Intelworks has done.

Everyone in this community that would like to see a JSON based STIX implementation should speak up.

[1] https://github.com/freetaxii/json-schema

[2] https://github.com/freestix 

Thanks,

Bret

Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

On Aug 20, 2015, at 10:53, Christopher Roblee <> wrote:

Signed PGP part
I would like to resurrect this conversation on the new list.Standard JSON bindings for STIX would greatly simplify interoperabilityfor us, and facilitate broader incorporation of the schema into our datamodels. It sounds like this is the case for other vendors as well.@Bret: could you please update the group as to where we are in terms ofgetting JSON support in the standard libraries?Thanks,ChrisOn 6/16/15 4:11 PM, Jordan, Bret wrote:> I fully and 100% support your proposal as outlined below.>> Thanks,>> Bret>>>> *Bret Jordan CISSP*> Director of Security Architecture and Standards | Office of the CTO> Blue Coat Systems> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303> "Without cryptography vihv vivc ce xhrnrw, however, the only thingthat can not be unscrambled is an egg.">>> On Jun 16, 2015, at 16:58, Aharon Chernin <<mailto:>> wrote:>>>> I do not see this discussion as XML vs JSON. In fact, we are one ofthose companies who use primarily JSON then convert to XML right beforeit goes out our pipes. This is a discussion around STIX interoperability.>>>>>> Consumers chose the winners. This choice is not necessarily relatedto ease of vendor implementation. Some good examples of this are VHS vsBetamax or the success of the SCAP standards outside of the US Federalgovernment. If we give users a poor STIX experience, by making STIXincompatible with STIX, then users are going to spread negativity aboutSTIX within the user community. They will instead chose products that"just work", instead of products who are doing the right thing byadopting STIX. Don't get me wrong, I am all for reducing vendorcomplexity, but user experience comes first.>>>> Let me give you a poor user experience with STIX:>> 1) ​OASIS ratifies new JSON binding of STIX for STIX 1.x>> 2) Consumers now begin to discover that tools that have "adopted STIX1.x" cannot communicate with each other>> 3) OASIS releases STIX 2.0, which is not backwards compatible withSTIX 1.x. (I support this btw)>> 4) Users now have STIX 1.x tools that cannot talk to each other andSTIX 2.x tools that cannot talk to STIX 1.x enabled products>>>> Aren't we supposed to be increasing interoperability?>>>> Discussion suggestions:>> *) We discuss the future of STIX 1.x in OASIS. We can discuss holdingoff new features in STIX 1.x - and only bug fix. We leave the onlyagreed upon binding as XML to allow all STIX 1.x client/servers tocontinue communications. At this point we begin to immediately beginwork on STIX 2.x.>> *) We discuss using JSON as the default binding type for STIX 2.0(which I would support btw). We already assume that STIX 1.x and 2.xwould be incompatible. This is our perfect chance to change exchangeformats.>>>> The above proposal only breaks interoperability between major STIXversions. Major version incompatibility can be easily explained to usersand in some cases is even expected by users.>>>>>> Aharon Chernin>> CTO>> *SOLTRA* | An FS-ISAC & DTCC Company>> 18301 Bermuda green Dr>> Tampa, fl 33647>> 813.470.2173 | <mailto:><mailto:>>> www.soltra.com <http://www.soltra.com/>>> ------------------------->> *From:* Jordan, Bret <<mailto:>>>> *Sent:* Tuesday, June 16, 2015 4:25 PM>> *To:* Aharon Chernin>> *Cc:* <mailto:>>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON>>>> The steps as I see them:>>>> 1: Finalize the JSON implementations for the various UML spec(s)>> 2: Implement the JSON version in the various APIs, including theMITRE Python/JAVA libraries (we also need libraries for a lot of otherlanguages to really go mainstream, not everyone uses or likes Python forcommercial products)>>>> At this point the market can decide which becomes the main methodpeople end up using.   My honest guess is it will be JSON, simplebecause it will be easier and faster for web developers, app developers,and open-source developers to use.  Most companies I have spoken withtoday are already doing some sort of JSON based STIX on the back end andthen trying to do some conversion to XML to send it over the wire.  Sowhy not just send it over the wire in JSON format?  Seems like a way toget people up and moving more quickly.>>>> When I look at it, if we sum up the total number of lines of codethat have been written so far for STIX, it will amount to less than 5%of the sum total amount of code that has yet to be written.  One of themajor goals of the standard should be easy of implementation and use.If the standard is too hard to implement, then people will not do it orthey will only do selective parts of it.>>>> The success of STIX and TAXII, in the end, will be based on thenumber of platforms and projects that implement it.  If we get acrossthe chasm and go mainstream, then in my view we can expect to seehundreds of apps on the various app stores and IMO, all of them will beusing JSON based STIX.>>>> Thanks,>>>> Bret>>>>>>>> *Bret Jordan CISSP*>> Director of Security Architecture and Standards | Office of the CTO>> Blue Coat Systems>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303>> "Without cryptography vihv vivc ce xhrnrw, however, the only thingthat can not be unscrambled is an egg.">>>>> On Jun 16, 2015, at 13:55, Aharon Chernin <<mailto:>> wrote:>>>>>> As it stands today, 99% of all STIX and TAXII traffic is XML based.I am afraid we risk fragmenting the STIX community. My hope is that STIXusers (users: not vendors) don't have to experience a world where STIXenabled products don't talk to each other due to one vendor picking JSONover XML. Users won't blame the data exchange format, they will blameSTIX. We need to avoid that.>>>>>> One of the ways we can mitigate this is by including support forJSON within the libraries used by most of the vendors who are alreadycreating STIX content. Mitre, are there plans to integrate the emergingJSON format of STIX into PythonSTIX?>>>>>>>>> Aharon Chernin>>> CTO>>> *SOLTRA* | An FS-ISAC & DTCC Company>>> 18301 Bermuda green Dr>>> Tampa, fl 33647>>> 813.470.2173 | <mailto:><mailto:>>>> www.soltra.com <http://www.soltra.com/>>>> ------------------------->>> *From:* Jordan, Bret <<mailto:>>>>> *Sent:* Tuesday, June 16, 2015 1:08 PM>>> *To:* <mailto:>>>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON>>>>>> This is such great news.  I will be working very closely withInterworks and others interested in JSON to get my implementation inperfect alignment.>>>>>> I would challenge all of you interested in JSON, or that havealready started using JSON based STIX / TAXII in your implementations tojoin us.  We need this community to push for a JSON based working groupin OASIS and we need all of your best ideas to help make this successfulas quickly as possible.>>>>>>>>> Thanks,>>>>>> Bret>>>>>>>>>>>> *Bret Jordan CISSP*>>> Director of Security Architecture and Standards | Office of the CTO>>> Blue Coat Systems>>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303>>> "Without cryptography vihv vivc ce xhrnrw, however, the only thingthat can not be unscrambled is an egg.">>>>>>> On Jun 16, 2015, at 08:09, Joep Gommers <<mailto:>> wrote:>>>>>>>> Dear all,>>>>>>>> With the excellent work going on from @Bret Jordan on STIX in JSON,we thought it helpful to share Intelworks approach to STIX in JSON andensure the community learned from our mistakes and investments. Props tolist- and team member @Wouter Bolsterlee for his work on this!>>>>>>>> *In short, lessons learned*>>>>>>>>   * Compound structures are objects>>>>   * Attributes and child elements are key/value pairs>>>>   * Relations are nested objects (or arrays of objects)>>>>   * Flat is better than nested>>>>   * And some ID and corner cases – see below>>>>>>>> Full details further down in this email. Your feedback is muchappreciated.>>>>>>>> We do have work-in-progress libraries available for (store-less)bi-directional transformation of XML, JSON and YAML notations – whichmight help those implementing STIX in JSON down the road. If you’d liketo know more, please contact me off-list.>>>>>>>> Best regards,>>>> Joep>>>>>>>> Founder & CEO>>>> Intelworks – Intelligence Powered Defence>>>> www.intelworks.com <http://www.intelworks.com/>>>>>>>>> Find me at>>>> +31 615489825>>>> @joepgommers>>>>>>>>>>>> ====>>>>>>>> The STIX language uses quite a few advanced XML modellingtechniques (multiple namespaces, xsi:type substitutions in instancedocuments, QName identifiers, and so on), making it quite complex towork with/implement. The JSON format used by Intelworks tries to be muchsimpler to work with. Structurally it mirrors most of the original XMLtree structure, but the resulting tree structures are not identicalsince the JSON representation favours flat objects over nested structures.>>>>>>>>>>>>       Compound structures are objects>>>>>>>> In general, each compound structure is converted into a JSON object(dict in Python). These objects always have a|type| key to indicate thetype of the structure:>>>>>>>> {>>>>   "type": "indicator",>>>>   "...": "...">>>> }>>>>>>>> Each of the main STIX constructs (see the STIX architecture<http://stixproject.github.io/getting-started/whitepaper/#architecture>)is represented as a JSON object. The |type| keys used are:>>>>>>>> Defining schema    XML Schema type    Object |type| field>>>> STIX (Core)    |STIXType|    |package|>>>> STIX (Campaign)    |CampaignType|    |campaign|>>>> STIX (Course of Action)    |CourseOfActionType|    |course-of-action|>>>> STIX (Exploit Target)    |ExploitTargetType|    |exploit-target|>>>> STIX (Incident)    |IncidentType|    |incident|>>>> STIX (Indicator)    |IndicatorType|    |indicator|>>>> STIX (TTP)    |TTPType|    |ttp|>>>> STIX (Threat Actor)    |ThreatActorType|    |threat-actor|>>>> CybOX    |ObservableType|    |observable|>>>>>>>> Secondary constructs use these additional types (this list is NONEXHAUSTIVE! And just a representation of potential)>>>>>>>> Defining schema    XML Schema type    Object |type| field>>>> STIX (Common)    |IdentityType|    |identity|>>>> STIX (Common)    |InformationSourceType|    |information-source|>>>> STIX (Common)    |StatementType|    |statement|>>>> STIX (Course of Action)    |ObjectiveType|    |objective|>>>> STIX (Indicator)    |ValidTimeType|    |valid-time|>>>> STIX (Markings)    |MarkingSpecificationType||marking-specification|>>>> STIX (Markings)    |MarkingStructureType| (and extensions)|marking-structure|>>>> STIX (TTP)    |InfrastructureType|    |infrastructure|>>>> STIX (TTP)    |MalwareInstanceType|    |malware-instance|>>>> STIX (TTP)    |ResourceType|    |resource|>>>> STIX (TTP)    |ToolInformationType|    |tool-information|>>>> STIX (TTP)    |VictimTargetingType|    |victim-targeting|>>>> CybOX    |MeasureSourceType|    |measure-source|>>>> CybOX    |ObjectType|    |cybox-object|>>>> CybOX    |ToolInformationType|    |tool-information|>>>>>>>>>>>>       Attributes and child elements are key/value pairs>>>>>>>> Both the attributes and child elements defined for a compoundstructure usually map to additional key/value pairs of the JSON objects:>>>>>>>> {>>>>   "type": "indicator",>>>>   "negate": false,>>>>   "title": "This is the title.">>>> }>>>>>>>>>>>>       Relations are nested objects (or arrays of objects)>>>>>>>> For one-to-one relations, the value is a nested object, and the keyis a singular noun (|observable| in the example):>>>>>>>> {>>>>   "type": "indicator",>>>>   "observable": {>>>>     "type": "observable",>>>>     "...": "...">>>>   },>>>>   "...": "...">>>> }>>>>>>>> For one-to-many relations, the value is a JSON array containing thechild objects, and the key is a plural noun (|indicators| in the example):>>>>>>>> {>>>>   "type": "package",>>>>   "indicators": [>>>>     {>>>>       "type": "indicator",>>>>       "...": "...">>>>     },>>>>     {>>>>       "type": "indicator",>>>>       "...": "...">>>>     }>>>>   ],>>>>   "...": "...">>>> }>>>>>>>> Additionally, the many |RelatedXYZ| constructs (and the surroundingcontainer objects) in STIX are also flattened: the target of therelation is the child object (or a list of those), and any additionalrelationship information is embedded into the child object(s):>>>>>>>> {>>>>   "type": "indicator",>>>>   "indicated_ttps": [>>>>     {>>>>       "type": "ttp",>>>>       "relationship": "...",>>>>       "relationship_information_source": "...",>>>>       "...": "...">>>>     },>>>>     {>>>>       "type": "ttp",>>>>       "relationship": "...",>>>>       "relationship_information_source": "...",>>>>       "...": "...">>>>     }>>>>   ],>>>>   "...": "...">>>> }>>>>>>>> See also the notes about nesting below.>>>>>>>>>>>>       Flat is better than nested>>>>>>>> The STIX XML representation is deeply nested, partly due to the wayXML is typically used. The JSON representation tries to be a bit morepragmatic and adheres to the "flat is better than nested" adage.>>>>>>>> In practice, this means that nested container structures areflattened as much as possible. Unnecessary container structures aresimply removed. For example, the |<stix:Indicators>| container structureused in the XML representation does not exist as such in the JSONrepresentation, since using an array is sufficient.>>>>>>>> To further reduce the number of nested objects, various XMLconstructs using container elements with (optional) attributes areflattened into the parent object by using multiple related keys. This isbest explained using an example.>>>>>>>> For example, the |StructuredTextType| used in both STIX and CybOXis basically a string that can optionally carry a|structuring_format|attribute. A naive conversion would require a nested object to representthis:>>>>>>>> {>>>>   "type": "...",>>>>   "description": {>>>>     "structuring_format": "html",>>>>     "value": "Description goes here.">>>>   },>>>>   "...": "...">>>> }>>>>>>>> Since the |structuring_format| is optional, this approach wouldoften result in a small nested object with only a single key/value pair(the |value|). To avoid this, /objectivistix/ takes an alternativeapproach using two related keys in the containing object:>>>>>>>> {>>>>   "type": "...",>>>>   "description": "Description goes here.",>>>>   "description_structuring_format": "html",>>>>   "...": "...">>>> }>>>>>>>> In case the |structuring_format| is not specified, the|description_structuring_format| key/value pair would simply not be present:>>>>>>>> {>>>>   "type": "...",>>>>   "description": "Description goes here.",>>>>   "...": "...">>>> }>>>>>>>>>>>>       ID handling>>>>>>>> All |id| and |idref| attributes in STIX XML are not simply stringvalues, but qualified names (QName in XML), meaning that they contain anamespace prefix which resolves to a namespace URI. To avoid anyexplicit mappings for these prefixes and their associated namespace URI,the JSON representation always expresses |id| and |idref| values intheir canonical form using the so-called Clark notation<http://www.jclark.com/xml/xmlns.htm>, which looks like this:|{http://example.com/ns/uri}local-name|.>>>>>>>> The top level object may optionally contain an |id_namespaces|mapping that maps prefixes to namespace URIs. This mapping will be usedto determine the prefixes used for |id| and |idref| attribute valueswhen converting the object to XML, as illustrated by the example below:>>>>>>>> {>>>>   "type": "package",>>>>   "id":"{http://example.org/}Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c",>>>>   "id_namespaces": {>>>>     "example": "http://example.org/">>>>   }>>>> }>>>> <stix:STIX_Package>>>>   xmlns:stix="http://stix.mitre.org/stix-1">>>>   xmlns:example="http://example.com/">>>>   id="example:Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c">>>>>   …>>>> </stix:STIX_Package>>>>>>>>> In case no |id_namespaces| mapping is present, a unique namespaceprefix will be used instead. The |id_namespaces| can safely be left outwith no semantical loss, since the prefix is arbitrary and only used forserialized XML data, and not for the in-memory model.>>>>>>>>>>>>       Special conversion notes>>>>>>>>  *>>>>>>>>     STIX package header>>>>>>>>     The package header is not treated as a first-class structure.Since the |STIX_Header| construct only applies to|STIX_Package|, it ismerged completely into the main |package| object (this avoids having anadditional nested object for the header):>>>>>>>>     {>>>>       "type": "package",>>>>       "description": "Description goes here.",>>>>       "...": "...">>>>     }>>>>  *>>>>>>>>     Structured text>>>>>>>>     The |StructuredTextType| construct is not transformed into achild object. Instead, the keys |foo| and(optionally)|foo_structuring_format| are added to the containing object.>>>>>>>>  *>>>>>>>>     Observable composition>>>>>>>>     An ‘observable composition’ structure does not result in anested object for the composition itself. Instead, the|composition| keycontains the child objects, and the |composition_operator| specifies theoperator:>>>>>>>>     {>>>>       "type": "indicator",>>>>       "observable": {>>>>         "composition_operator": "or",>>>>         "composition": [>>>>           {>>>>             "type": "observable",>>>>             "...": "...">>>>           },>>>>           {>>>>             "type": "observable",>>>>             "...": "...">>>>           }>>>>         ]>>>>       },>>>>       "...": "...">>>>     }>--Chris RobleeDirector of EngineeringTruSTAR TechnologyMobile: +1 781 248 2828OpenPGP key ID: 2C9D0D20

Attachment:
signature.asc

Description: Message signed with OpenPGP using GPGMail