Data Source File Format And Structure

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Data Source File Format And Structure

Post by Fred »

I started a thread on how to convert files around for OAOO reasons here. However we also need to design the format of such files, hence this thread.

I'm going to take a rough crack at what they need to contain in the next post and update htis post with the results of discussions below as we move forward. Please contribute any/all thoughts.

As a start we need:
  • Sufficient info to generate C headers and data definitions
  • Sufficient info to process the data, describe it to the user in a UI and save it to backup file(s)
The devil is in the details, though, so on we go....

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

To create C we need:
  • names for the types we are creating
  • names for the objects we are creating
  • structural controls to generate different types and nested types
  • the data to populate the initialisation objects with
Only the third item from that short list needs to be defined here, so let me take a crack at it :
  • comments for each item with doxygen tags for web display and firmware dev IDE click through assistance
  • structs consisting of other types, probably ONLY structs, no raw types
  • the integer types, char, short, long in signed and unsigned variants
  • bit fields in struct format, possibly with a union to a normal unsigned integer type
  • arrays of whatever else, usually integers, though
  • various memory declaration controls or a way of inserting arbitrary text into the declarations
To generate the UI and process the data we need:
  • locationID
  • Descriptions for each field, user and/or dev preferably one copy for both.
  • Data for defaults to load from file in offline
  • Names for each field
  • Raw types for each field
  • Internal units for each field
  • Conversion information on how to get the internal unit from the raw value
  • Position information, preferably inferred from file position, previous content, and raw types, to explicitly state it would violate OAOO as saying "2 bytes" does not indicate type at all.
It would also be cool if the structure itself, NOT the data, was checksummed somehow, and included in the firmware in an array with matching locationID so it could be returned from the data service with the other info. This would key the firmware to the data provided rather well if it was also included in these files. We can live without this for now, though, but it would make it SUPER robust.

What stuff is missing from above, I literally just typed it sequentially, there must be mistakes...

Correct them! :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Data Source File Format And Structure

Post by dandruczyk »

Fred wrote:To create C we need:
To generate the UI and process the data we need:
  • locationID
  • Descriptions for each field, user and/or dev preferably one copy for both.
  • Data for defaults to load from file in offline
  • Names for each field
  • Raw types for each field
  • Internal units for each field
  • Conversion information on how to get the internal unit from the raw value
  • Position information, preferably inferred from file position, previous content, and raw types, to explicitly state it would violate OAOO as saying "2 bytes" does not indicate type at all.
It would also be cool if the structure itself, NOT the data, was checksummed somehow, and included in the firmware in an array with matching locationID so it could be returned from the data service with the other info. This would key the firmware to the data provided rather well if it was also included in these files. We can live without this for now, though, but it would make it SUPER robust.

What stuff is missing from above, I literally just typed it sequentially, there must be mistakes...

Correct them! :-)

Fred.
Other important things are control interdependancy handling, i.e. "master feature bits", thinking out-loud that would be items like different types of warmup mgmt, water injection, traction control, nitrous mgmt, etc). i.e. if bit X is set, then these controls in this symbolic group name need to be en/disabled. Also should be able to handle compound cases (multiple bit combinations). In MegaTunix I do this by assigning user manipulatable controls (referring to ecu memory locations) to symbolic groups as needed, and enable and disable them as needed, it works, but the implementation could be improved.

I'd also suggest "help information" the gui will use this for tooltips (context sensitive help). This is end-user facing information and should be worded appropriately with that in mind.

Fred, you also forgot offset within that locationID, and you sort of mentioned it but didn't clarify it well, but a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth. This is required for the tuning software to be able to manipulate ecu memory, as it'll use this to determine the length field required when sending data to the ECU, as well as handling any required endianness conversion (depends on host platform).

There's prolly more which will come up as we move on...
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Data Source File Format And Structure

Post by dandruczyk »

Another idea.

fred days theres about 350K of flash. Why not take all the xml stuff the tuning software(s) will want, and tar/bzip2/lzma it an stuff it in a known static location ID in flash, the firmware itself doesn't need it to run (so no worry about having the firmware need to deal with decompress crap) let the tuning software request the length of that block and download the info on demand, uncompress/untar it and it solves the problem of needed to distribute xml and/or other conf files to users with each release, it could just be embedded into the firmware image itself on firmware compilation.

Limitations, would be that it can only be so large (though XML compresses VERY well), and downloading it from the ECU make take a few moments on tuning software startup, and loading the firmware to the ECU (a rare one time operation) will take longer, due to the larger image size, but these are relatively minor issues so far.

About the only things that may be needed is minor firmware tweaks to handle a potentially very large location ID (longer than 64K, thus requiring use of long integers, or careful handling of under/overflow))

Testing the compression of XML, the megatunix .glade files are XML and compression all of them with tar.bz2 results in better than a 100:1 compression ratio.

36.7 megs down to 255K, and that's about 416659 lines of XML. I'd consider this a best case scenario, though still very good. I could shove in all of Megatunix's glade files into the freeems flash and still have space to spare.
Last edited by dandruczyk on Mon Jan 10, 2011 12:46 am, edited 1 time in total.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

I just thought of a problem with the large block thing. The mem map is only 64k so a larger block than 16k can't exist in memory at one time, ie, we'd probably need a special call for that, but that isn't a problem really. It would take a lot of work, and i cant think of a clean way, to implement multi page retrieves from big flash items.

We'd need to see how big it is compressed before judging this too. It might not be feasible, but it probably is.

(12:41:46) Dave: as a test, the 1.3 Megs of .datamap files (what mtx uses for the gui,) compresses to 90k with tar.bz2, XML should be similar
(12:43:56) Dave: testing the .glade files (XML), total size of all of em is 36.7 Megs, which compresses down to 255K (that's 416659 lines of XML (via wc -l))
(12:44:08) Dave: so thats over 100:1 compression
(12:44:22) Dave: I'd consider that best case

Next reply is to the previous post.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

mtx_man wrote:Other important things are control interdependancy handling, i.e. "master feature bits", thinking out-loud that would be items like different types of warmup mgmt, water injection, traction control, nitrous mgmt, etc). i.e. if bit X is set, then these controls in this symbolic group name need to be en/disabled. Also should be able to handle compound cases (multiple bit combinations). In MegaTunix I do this by assigning user manipulatable controls (referring to ecu memory locations) to symbolic groups as needed, and enable and disable them as needed, it works, but the implementation could be improved.
Currently there is no requirement for this, however, there possibly will be, in some ways. I'd prefer something like "has valid pin or hw interface assigned" if yes, use/configure/run, if no, dont. I'm unsure how that might work. What is true, though, is that that stuff is a function of the data itself as the FreeEMS firmware will need to know that too. I'll keep it in mind. Thanks.
I'd also suggest "help information" the gui will use this for tooltips (context sensitive help). This is end-user facing information and should be worded appropriately with that in mind.
That's what I meant by description/comments. If we can use the same thing in the gui and the code comment, great, it might want a seperate field, though.
Fred, you also forgot offset within that locationID
No I didn't! :-)

What I said was that I expect you to know that one plum, two apples and one watermelon take up (1 + 4 + 4 + 30) = 39 units of size. If each field has an offset, it's asking to break all the time when editing. Lets make it dynamic on parse.

Code: Select all

{
    char flags = 45;
    short length = 339;
    long[4] times = {893, 324, 23, 982};
}
If that psuedo code was represented in XML then as you parsed it you could tally up the byte count. This is MUCH better than explicitly stating offsets. If we do the other, if you edit a field at the beginning of the block, you need to adjust them ALL, which is totally unacceptable and annoying. I've heard you complain about that in ms land with james/ken editting stuff. Lets fix the process as I'll probably edit stuff too.
a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth.
Yes, I mentioned this too, we could do it that way, maybe, or some other, it's an item that needs discussing.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Data Source File Format And Structure

Post by dandruczyk »

Fred, you also forgot offset within that locationID
No I didn't! :-)

What I said was that I expect you to know that one plum, two apples and one watermelon take up (1 + 4 + 4 + 30) = 39 units of size. If each field has an offset, it's asking to break all the time when editing. Lets make it dynamic on parse.

Code: Select all

{
    char flags = 45;
    short length = 339;
    long[4] times = {893, 324, 23, 982};
}
If that psuedo code was represented in XML then as you parsed it you could tally up the byte count. This is MUCH better than explicitly stating offsets. If we do the other, if you edit a field at the beginning of the block, you need to adjust them ALL, which is totally unacceptable and annoying. I've heard you complain about that in ms land with james/ken editting stuff. Lets fix the process as I'll probably edit stuff too.
a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth.
Yes, I mentioned this too, we could do it that way, maybe, or some other, it's an item that needs discussing.

Fred.
That doesn't look like it has enough information to DETERMINE the base offset from the beginning of the location id however. it needs to have enough info to determine that during parse however, if it does, there'll be no issue.

Since MegaTunix gui layouts are currently done in glade, there needs to be a way to marry the glade to your XML, the most likely way is via some sort of name in your files, making sure there are no duplicates is important, as well as keeping in mind that namespace changes will break things, and I don't forsee anyway around this.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

mtx_man wrote:That doesn't look like it has enough information to DETERMINE the base offset from the beginning of the location id however. it needs to have enough info to determine that during parse however, if it does, there'll be no issue.
There will be no issue, AND, I'll be free to rearrange things freely, yay :-)
Since MegaTunix gui layouts are currently done in glade, there needs to be a way to marry the glade to your XML, the most likely way is via some sort of name in your files, making sure there are no duplicates is important, as well as keeping in mind that namespace changes will break things, and I don't forsee anyway around this.
Yes, the only way is to provide a mapping file between glade and my stuff, but that's just a stupid pointless indirection. Much better to minimise change on my end, choose good names up front, and fix your end where it makes sense.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

viewtopic.php?f=8&t=504

Old version of this thread for reference.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Data Source File Format And Structure

Post by dandruczyk »

Go and study the megatunix datamap files (Gui subdir) for the types of information neede to be bound to controls. take special note of combo boxes.
Post Reply