Data Source File Format And Structure

[Fred]

After struggling with the data definition part of the file (the most important bit) I've been reading and googling about how others do this and I found some good stuff on both W3C and Oracle about it.

http://download.oracle.com/docs/cd/B104 ... px_d3l.htm

I think something similar to D3L would be good for parts of our document structure. Other parts are too unique to use it. D3L is proprietary, however the design of it is available, and we don't need to use their processor.

more reading:

http://documentation.progress.com/outpu ... Types.html
http://documentation.progress.com/outpu ... 03773.html
http://documentation.progress.com/outpu ... ments.html

http://www.w3.org/2005/07/xml-schema-patterns.html
http://www.w3.org/TR/2004/REC-xmlschema-2-20041028/

Another thing that I just realised how to articulate is that we are talking about two different data descriptions here. Two levels. Level one is the XSD that describes precisely how the XML itself should look, but not at all how the data in FreeEMS should look. Then, embedded in the XML is the second level of description which uses the XML to describe the binary structures in FreeEMS.

Fred.

[Fred]

http://www.la-solutions.co.uk/content/M ... Struct.htm

That could be just what we need, ready to roll almost.

Fred.

[Fred]

Go and study the megatunix datamap files (Gui subdir) for the types of information neede to be bound to controls. take special note of combo boxes.

I don't see what is special about them, they just look to be analogous to bit fields.

Please review commit hash 0ab4c40404efa21470910e452c7b91a6336d1fdd and path /docs/interface/FreeEMS-DDL.mm

https://github.com/fredcooke/freeems-va ... EMS-DDL.mm

There are plenty of problems with it and things that I'm not sure about, most are noted as a question in a leaf node. Types are noted in the leaf nodes too. However it will make a good starting point. Please either comment here, or modify the file and upload for review.

The file represents what we will transfer to the XSD file which controls and tests structure of an XML document. The document will have multiple nodes in place of things like errorid and instance and structure etc.

~30 errors
~40 locations
~20 structs

etc.

Fred.

[dandruczyk]

Go and study the megatunix datamap files (Gui subdir) for the types of information neede to be bound to controls. take special note of combo boxes.

I don't see what is special about them, they just look to be analogous to bit fields.

They are with one important exception, in Mtx's implementation you can "skip" bits or even have a choice affect multiple bits, mainly due to the "choices" and "bitvals" keys which line up with each other. The impl use in MS ini.msq doesn't allow this at all.

Please review commit hash 0ab4c40404efa21470910e452c7b91a6336d1fdd and path /docs/interface/FreeEMS-DDL.mm

https://github.com/fredcooke/freeems-va ... EMS-DDL.mm

There are plenty of problems with it and things that I'm not sure about, most are noted as a question in a leaf node. Types are noted in the leaf nodes too. However it will make a good starting point. Please either comment here, or modify the file and upload for review.

The file represents what we will transfer to the XSD file which controls and tests structure of an XML document. The document will have multiple nodes in place of things like errorid and instance and structure etc.

~30 errors
~40 locations
~20 structs

etc.

Fred.

Will read when i'm feeling better and have time, perhaps tomorrow.

[dandruczyk]

The oracle stuff looks convoluted and ugly, I'm having a hard time making sense of it before my mind turns to goop, and I have to stop.

Fred maybe you should look into how QT does it with it's "moc" tool (meta object compiler)

[Fred]

Yeah, but did you look at my mindmap, and if so, what are your thoughts on it? Any ideas for missing bits? etc.

[EssEss]

I started looking at the mm .. It's very dense and I'm walking through it very slow.
I don't really understand instances. They aren't as descriptive as something like notes/errorids/payloadids/structures ..

1) could you please explain a little more what instances are describing ? are they just config items (I see loc_id in there) ?
1.5) interface version is only in relation to how I would parse the FreeEMS-DDL, right ?

I think structures and instances appear to have some kind of relationship but its not readily apparent.

now some picky stuff:

2) I'm not sure how to phrase this, but instead of having all these free-form 'arbitrary string' values in payloadid/structure/etc why not just 'link back' to a note in notes .. any note not related to a specific item is therefore just a general 'global' note. in the future when we have to extend things, you're not having to write custom arbitrary parsers for each free-form field .. instead you use a general note parser. if notes ever need migrated to a different format its easier to write a simple tool against it also.

3) instances->instance->pre-modifiers ... I feel attributes instead of pre-modifiers would be more descriptive.

4) I also feel that payloadid's should have a link back to specific errorid's .. ids which could possibly be 'thrown' from that specific pkt. any error id which is not related to a pkt need not be in the group of errorids.

4.5) I'd like to see bitfields tied back to a primitive in some way to maintain natural padding in the pkt content. if only 2 bits of a uint8 are used, the width of that field in the pkt is guaranteed to be 8bits wide even though some bits are not used. I could see a naive implementation misinterpreting the spec and trying to pack/shift things in funny ways. which leads to my next question:

4.75) for bitfields .. I'd like some way to describe 'bit padding' .. maybe a few bits in the middle of a primitive are unused.

here's some other things to think about:

5) I'd like a very firm guarantee that the content in a pkt will be packed. this puts somewhat of a burden on the firmware to flatten things in a device independent way (if its an acceptable tradeoff). a specific firmware implementation can still use native alignment, but when that structure is serailized I want assurance that it packed when flattened before shipping off.

I'll pause here, I didn't figure I'd have so many ideas/questions mindmaps really get me moving mentally.

[Fred]

I started looking at the mm .. It's very dense and I'm walking through it very slow.

GREAT, thank you very much! :-)

I don't really understand instances. They aren't as descriptive as something like notes/errorids/payloadids/structures ..

I would have thought from an OO code perspective they are perfectly descriptive, BUT, I'm looking at it from a code perspective, not a UI perspective... so maybe something that matches both is better. They are instances of structures.

The semantics, not present in the document, but assumed to be understood are:

• A structure can contain X primitives and other types, including other structures. (it might be nice if nested functions occur before the ones that contain them, it would save a quite a bit of code to force ordering.
• An instance is an instance of a structure (hence name) and contains data and field names etc. For example, there will be a mainTable structure and there will be a dozen instances of that each with different names, descriptions, labels, etc.

1) could you please explain a little more what instances are describing ? are they just config items (I see loc_id in there) ?

See above. Plus, no, could be config, flash only config, ram config, ram variables RO, ram variables RW, etc. Datalog structures, etc too.

1.5) interface version is only in relation to how I would parse the FreeEMS-DDL, right ?

No, interface version is the key in the firmware to how the interface works. For example version 0.5 might come out with interface 0.25 and a week later someone will discover a bug in the implementation of the nissan wheel decoder requiring a fix. 0.5.1 will be released and will still have interface 0.25 on it, with tuning authors requiring no change. Having the location id service etc provides even closer keying of document to reality and verification that the iface version is accurate (if the service returns a different list or different masks, the interface has changed and the firmware should be rejected as buggy, or something (could be used as pre-release check).

However, you raise an excellent point, the format needs a version too, and it needs to be in the file somehow. Noted.

I think structures and instances appear to have some kind of relationship but its not readily apparent.

Described above, the format will likely need a small guide explaining some relationships like this as it probably will never be obvious from xml/xslt/xsd documents.

2) I'm not sure how to phrase this, but instead of having all these free-form 'arbitrary string' values in payloadid/structure/etc why not just 'link back' to a note in notes .. any note not related to a specific item is therefore just a general 'global' note. in the future when we have to extend things, you're not having to write custom arbitrary parsers for each free-form field .. instead you use a general note parser. if notes ever need migrated to a different format its easier to write a simple tool against it also.

Interesting idea, I was thinking notes were release/changelog/known bugs/etc, though. I thought about reuse of things. We need a mechanism for values that are common across multiple things. I'm not sure using notes for it is a good idea though, but something similar, ok, yes, sure.

3) instances->instance->pre-modifiers ... I feel attributes instead of pre-modifiers would be more descriptive.

This is for C generation, so we should use whatever the correct C word is, and make it apparent that whatever it is goes in front of the type and name, ditto post. What is the correct word for volatile/const/whatever?

4) I also feel that payloadid's should have a link back to specific errorid's .. ids which could possibly be 'thrown' from that specific pkt. any error id which is not related to a pkt need not be in the group of errorids.

Not going to happen as it would be human generated and therefore prone to errors. Plus, they are there to be displayed and key a dev back to a line of code from a user report. All errors should be displayed to the user as it indicates a code issue on ems or pc or a comms issue or a user action error. Provided we can give good descriptions for them, the name is already nearly enough. It's fine as is. You can theoretically get any error from any action, what it is is irrelevant, a human needs to get involved and deal with it. Linking such things like this is for machine consistency. I guess you're thinking "if error == xyz then do some action" ? I would say "do some action" is always just tell the user in an obvious way.

4.5) I'd like to see bitfields tied back to a primitive in some way to maintain natural padding in the pkt content. if only 2 bits of a uint8 are used, the width of that field in the pkt is guaranteed to be 8bits wide even though some bits are not used. I could see a naive implementation misinterpreting the spec and trying to pack/shift things in funny ways. which leads to my next question:

I agree, I had them as primitives, but they didn't fit there. how about we have:

Code: Select all

bitfield group:
      primitive type:
             uint8/uint16/uint32/uint64 - compulsory, only these types allowed
      bits:
             blah
             omg
             wtf
             spare
             spare
             notthere
(two implicit spares)

And if there are too many bits for the width, error on parse. If there are not enough, pad at the end. If you want padding in the middle, spec some spares. They could be keyed in a gui by name, and the spare ones unused or attached to generic controls or soemthign. spare1 spare2 spare3 etc.

4.75) for bitfields .. I'd like some way to describe 'bit padding' .. maybe a few bits in the middle of a primitive are unused.

See above.

5) I'd like a very firm guarantee that the content in a pkt will be packed. this puts somewhat of a burden on the firmware to flatten things in a device independent way (if its an acceptable tradeoff). a specific firmware implementation can still use native alignment, but when that structure is serailized I want assurance that it packed when flattened before shipping off.

I've heard you use "flattened" before, I assume you mean serialised from an object form? What do you mean by packed? Content in a packet is what it is. IE, if packet 124 says it has two 16 bit unsigned int fields meaning xyz and abc, then 50 bytes of data, then it is what it is. Please explain further whether I answered or not so it's clear reading the thread.

I'll pause here, I didn't figure I'd have so many ideas/questions ;) mindmaps really get me moving mentally.

Yes, me too, love them :-)

Fred.

[Fred]

Fixed:

• Added Version field for file format itself.
• Changed bit field groups structure adding width to accommodate non single bit fields.
• Pre-Modifiers changed to Qualifiers per GCC terminology
• Post-Modifiers changed to Attributes per GCC terminology

Committed, see hash: ebf1b3f

Fred.

[EssEss]

we need to specify bit ordering in bitfields .. using the name:width struct trick is non portable... your code needs to generate correctly irrespective of platform, right?

some kind of tag which says 'lsb first' or 'msb first' is all thats needed I think. compilers usually have little_endian or big_endian defines which allows the code to be generated both ways, but only one is selceted at compile time. so the code generator can layout both structures (lsb first or msb first) and then the compiler takes care of the rest.... the xml definition in mostly for the benefit of tuners.

this may explain better: http://en.wikipedia.org/wiki/Most_significant_bit