View unanswered posts | View active topics It is currently Thu Jan 21, 2021 11:20 am



Reply to topic  [ 28 posts ]  Go to page 1, 2, 3  Next
Data Source File Format And Structure 
Author Message
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
I started a thread on how to convert files around for OAOO reasons here. However we also need to design the format of such files, hence this thread.

I'm going to take a rough crack at what they need to contain in the next post and update htis post with the results of discussions below as we move forward. Please contribute any/all thoughts.

As a start we need:

  • Sufficient info to generate C headers and data definitions
  • Sufficient info to process the data, describe it to the user in a UI and save it to backup file(s)

The devil is in the details, though, so on we go....

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Sun Jan 09, 2011 10:55 pm
Profile WWW
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
To create C we need:

  • names for the types we are creating
  • names for the objects we are creating
  • structural controls to generate different types and nested types
  • the data to populate the initialisation objects with

Only the third item from that short list needs to be defined here, so let me take a crack at it :

  • comments for each item with doxygen tags for web display and firmware dev IDE click through assistance
  • structs consisting of other types, probably ONLY structs, no raw types
  • the integer types, char, short, long in signed and unsigned variants
  • bit fields in struct format, possibly with a union to a normal unsigned integer type
  • arrays of whatever else, usually integers, though
  • various memory declaration controls or a way of inserting arbitrary text into the declarations

To generate the UI and process the data we need:

  • locationID
  • Descriptions for each field, user and/or dev preferably one copy for both.
  • Data for defaults to load from file in offline
  • Names for each field
  • Raw types for each field
  • Internal units for each field
  • Conversion information on how to get the internal unit from the raw value
  • Position information, preferably inferred from file position, previous content, and raw types, to explicitly state it would violate OAOO as saying "2 bytes" does not indicate type at all.

It would also be cool if the structure itself, NOT the data, was checksummed somehow, and included in the firmware in an array with matching locationID so it could be returned from the data service with the other info. This would key the firmware to the data provided rather well if it was also included in these files. We can live without this for now, though, but it would make it SUPER robust.

What stuff is missing from above, I literally just typed it sequentially, there must be mistakes...

Correct them! :-)

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Sun Jan 09, 2011 11:20 pm
Profile WWW
LQFP112 - Up with the play

Joined: Sun Apr 06, 2008 6:30 pm
Posts: 100
Fred wrote:
To create C we need:
To generate the UI and process the data we need:

  • locationID
  • Descriptions for each field, user and/or dev preferably one copy for both.
  • Data for defaults to load from file in offline
  • Names for each field
  • Raw types for each field
  • Internal units for each field
  • Conversion information on how to get the internal unit from the raw value
  • Position information, preferably inferred from file position, previous content, and raw types, to explicitly state it would violate OAOO as saying "2 bytes" does not indicate type at all.

It would also be cool if the structure itself, NOT the data, was checksummed somehow, and included in the firmware in an array with matching locationID so it could be returned from the data service with the other info. This would key the firmware to the data provided rather well if it was also included in these files. We can live without this for now, though, but it would make it SUPER robust.

What stuff is missing from above, I literally just typed it sequentially, there must be mistakes...

Correct them! :-)

Fred.


Other important things are control interdependancy handling, i.e. "master feature bits", thinking out-loud that would be items like different types of warmup mgmt, water injection, traction control, nitrous mgmt, etc). i.e. if bit X is set, then these controls in this symbolic group name need to be en/disabled. Also should be able to handle compound cases (multiple bit combinations). In MegaTunix I do this by assigning user manipulatable controls (referring to ecu memory locations) to symbolic groups as needed, and enable and disable them as needed, it works, but the implementation could be improved.

I'd also suggest "help information" the gui will use this for tooltips (context sensitive help). This is end-user facing information and should be worded appropriately with that in mind.

Fred, you also forgot offset within that locationID, and you sort of mentioned it but didn't clarify it well, but a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth. This is required for the tuning software to be able to manipulate ecu memory, as it'll use this to determine the length field required when sending data to the ECU, as well as handling any required endianness conversion (depends on host platform).

There's prolly more which will come up as we move on...

_________________
-- David
http://www.megatunix.com
http://megatunix.sourceforge.net


Mon Jan 10, 2011 12:24 am
Profile
LQFP112 - Up with the play

Joined: Sun Apr 06, 2008 6:30 pm
Posts: 100
Another idea.

fred days theres about 350K of flash. Why not take all the xml stuff the tuning software(s) will want, and tar/bzip2/lzma it an stuff it in a known static location ID in flash, the firmware itself doesn't need it to run (so no worry about having the firmware need to deal with decompress crap) let the tuning software request the length of that block and download the info on demand, uncompress/untar it and it solves the problem of needed to distribute xml and/or other conf files to users with each release, it could just be embedded into the firmware image itself on firmware compilation.

Limitations, would be that it can only be so large (though XML compresses VERY well), and downloading it from the ECU make take a few moments on tuning software startup, and loading the firmware to the ECU (a rare one time operation) will take longer, due to the larger image size, but these are relatively minor issues so far.

About the only things that may be needed is minor firmware tweaks to handle a potentially very large location ID (longer than 64K, thus requiring use of long integers, or careful handling of under/overflow))

Testing the compression of XML, the megatunix .glade files are XML and compression all of them with tar.bz2 results in better than a 100:1 compression ratio.

36.7 megs down to 255K, and that's about 416659 lines of XML. I'd consider this a best case scenario, though still very good. I could shove in all of Megatunix's glade files into the freeems flash and still have space to spare.

_________________
-- David
http://www.megatunix.com
http://megatunix.sourceforge.net


Last edited by dandruczyk on Mon Jan 10, 2011 12:46 am, edited 1 time in total.



Mon Jan 10, 2011 12:39 am
Profile
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
I just thought of a problem with the large block thing. The mem map is only 64k so a larger block than 16k can't exist in memory at one time, ie, we'd probably need a special call for that, but that isn't a problem really. It would take a lot of work, and i cant think of a clean way, to implement multi page retrieves from big flash items.

We'd need to see how big it is compressed before judging this too. It might not be feasible, but it probably is.

(12:41:46) Dave: as a test, the 1.3 Megs of .datamap files (what mtx uses for the gui,) compresses to 90k with tar.bz2, XML should be similar
(12:43:56) Dave: testing the .glade files (XML), total size of all of em is 36.7 Megs, which compresses down to 255K (that's 416659 lines of XML (via wc -l))
(12:44:08) Dave: so thats over 100:1 compression
(12:44:22) Dave: I'd consider that best case

Next reply is to the previous post.

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Mon Jan 10, 2011 12:44 am
Profile WWW
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
mtx_man wrote:
Other important things are control interdependancy handling, i.e. "master feature bits", thinking out-loud that would be items like different types of warmup mgmt, water injection, traction control, nitrous mgmt, etc). i.e. if bit X is set, then these controls in this symbolic group name need to be en/disabled. Also should be able to handle compound cases (multiple bit combinations). In MegaTunix I do this by assigning user manipulatable controls (referring to ecu memory locations) to symbolic groups as needed, and enable and disable them as needed, it works, but the implementation could be improved.

Currently there is no requirement for this, however, there possibly will be, in some ways. I'd prefer something like "has valid pin or hw interface assigned" if yes, use/configure/run, if no, dont. I'm unsure how that might work. What is true, though, is that that stuff is a function of the data itself as the FreeEMS firmware will need to know that too. I'll keep it in mind. Thanks.

Quote:
I'd also suggest "help information" the gui will use this for tooltips (context sensitive help). This is end-user facing information and should be worded appropriately with that in mind.

That's what I meant by description/comments. If we can use the same thing in the gui and the code comment, great, it might want a seperate field, though.

Quote:
Fred, you also forgot offset within that locationID

No I didn't! :-)

What I said was that I expect you to know that one plum, two apples and one watermelon take up (1 + 4 + 4 + 30) = 39 units of size. If each field has an offset, it's asking to break all the time when editing. Lets make it dynamic on parse.

Code:
{
    char flags = 45;
    short length = 339;
    long[4] times = {893, 324, 23, 982};
}


If that psuedo code was represented in XML then as you parsed it you could tally up the byte count. This is MUCH better than explicitly stating offsets. If we do the other, if you edit a field at the beginning of the block, you need to adjust them ALL, which is totally unacceptable and annoying. I've heard you complain about that in ms land with james/ken editting stuff. Lets fix the process as I'll probably edit stuff too.

Quote:
a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth.

Yes, I mentioned this too, we could do it that way, maybe, or some other, it's an item that needs discussing.

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Mon Jan 10, 2011 1:00 am
Profile WWW
LQFP112 - Up with the play

Joined: Sun Apr 06, 2008 6:30 pm
Posts: 100
Quote:
Quote:
Fred, you also forgot offset within that locationID

No I didn't! :-)

What I said was that I expect you to know that one plum, two apples and one watermelon take up (1 + 4 + 4 + 30) = 39 units of size. If each field has an offset, it's asking to break all the time when editing. Lets make it dynamic on parse.

Code:
{
    char flags = 45;
    short length = 339;
    long[4] times = {893, 324, 23, 982};
}


If that psuedo code was represented in XML then as you parsed it you could tally up the byte count. This is MUCH better than explicitly stating offsets. If we do the other, if you edit a field at the beginning of the block, you need to adjust them ALL, which is totally unacceptable and annoying. I've heard you complain about that in ms land with james/ken editting stuff. Lets fix the process as I'll probably edit stuff too.

Quote:
a variables' "size" i.e. signed/unsigned 8/16/32/64 bit. For example in MTX, I use a textual representation, like:
_S08_,_U08_, _U16_, _S16_,, and so on and so forth.

Yes, I mentioned this too, we could do it that way, maybe, or some other, it's an item that needs discussing.

Fred.


That doesn't look like it has enough information to DETERMINE the base offset from the beginning of the location id however. it needs to have enough info to determine that during parse however, if it does, there'll be no issue.

Since MegaTunix gui layouts are currently done in glade, there needs to be a way to marry the glade to your XML, the most likely way is via some sort of name in your files, making sure there are no duplicates is important, as well as keeping in mind that namespace changes will break things, and I don't forsee anyway around this.

_________________
-- David
http://www.megatunix.com
http://megatunix.sourceforge.net


Mon Jan 10, 2011 1:37 am
Profile
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
mtx_man wrote:
That doesn't look like it has enough information to DETERMINE the base offset from the beginning of the location id however. it needs to have enough info to determine that during parse however, if it does, there'll be no issue.

There will be no issue, AND, I'll be free to rearrange things freely, yay :-)

Quote:
Since MegaTunix gui layouts are currently done in glade, there needs to be a way to marry the glade to your XML, the most likely way is via some sort of name in your files, making sure there are no duplicates is important, as well as keeping in mind that namespace changes will break things, and I don't forsee anyway around this.

Yes, the only way is to provide a mapping file between glade and my stuff, but that's just a stupid pointless indirection. Much better to minimise change on my end, choose good names up front, and fix your end where it makes sense.

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Mon Jan 10, 2011 9:41 am
Profile WWW
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 15376
Location: Home sweet home!
viewtopic.php?f=8&t=504

Old version of this thread for reference.

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Thu Jan 13, 2011 8:18 am
Profile WWW
LQFP112 - Up with the play

Joined: Sun Apr 06, 2008 6:30 pm
Posts: 100
Go and study the megatunix datamap files (Gui subdir) for the types of information neede to be bound to controls. take special note of combo boxes.

_________________
-- David
http://www.megatunix.com
http://megatunix.sourceforge.net


Fri Jan 14, 2011 10:48 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 28 posts ]  Go to page 1, 2, 3  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software for PTF. ColorizeIt.