Data Source File Format And Structure

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

http://cboard.cprogramming.com/c-progra ... ndian.html

Wow, you learn something every day. OK, what you suggested is fine. But, given you suggested it, you can name the slot :-)

Re packing, I disagree with your statement. I think it is unreasonable to tell the implementation to send packed data. What I'm going to suggest amounts to the same thing, though. I suggest that if a device is sending non-packed data, it must specify where the gaps are. IE, what is specced in the xml file must match what is sent. How that is achieved is upto the designer (specify pad0, pad1, pad2, etc or stuff it in before sending, either way works the same externally then).

With FreeEMS the packing thing is a non issue. All the structs are manually packed and flash memory aligned also. IE, any padding would cause the build to fail.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

C gives no guarantee of the ordering of fields within machine words, so if you do use them for the latter reason, you program will not only be non-portable, it will be compiler-dependent too. The Standard says that fields are packed into ‘storage units’, which are typically machine words. The packing order, and whether or not a bitfield may cross a storage unit boundary, are implementation defined.
From the second page of that thread.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

http://www.sweetscape.com/010editor/man ... fields.htm
http://stackoverflow.com/questions/1490 ... -alignment
http://stackoverflow.com/questions/8640 ... endianness

Thank you EssEss :-)

I need to think about this some more - perhaps another phone call and you can educate me and clear my mind.

Right now it doesn't matter as we need the show on the road and it does not affect FreeEMS.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Data Source File Format And Structure

Post by dandruczyk »

In Mtx (and I donno how any other tuning software person does it). the gui layout files (.glade) ONLY define the position and layout of the graphical elements, allowing it to be easily reworked/moved around etc, there isn't any way to "link" the controls within the .glade files to data/locations in the ECU memory. MegaTunix currently uses files named ".datamap" (ini style format). that define extra keys assigned to each control. The datamap/glade data are linked by control name. i.e when Mtx loads a glade file it recursively scans it and looks for matching sections in the .datamap, if found it parses and loads the additional data and "binds" it to the widgets. There's nothingreally preventing mtx from using some other file format to link to the .glade files. so it DOES NOT have to be XML, it could as well be .ini, .json or something else. so don't get hungup on it needing to be XML for Mtx, hell you could probably make your tool spit out files specific to various types of now/future software. IMHO XML is much harder to parse than some other file types.

The way Mtx is designed, all parameters to be bound are specified in a key=value format. The keys specific to a control are defined as "keys=key,key,key....", so mtx knows which keys to look for instead of blindly searching for every possible one. XML/JSON/other of course will do this differently and i am NOT an XML wiz at all, for all I know I horribly misuse it throughout mtx). All the values are strings, however within megatunix specific key names that should be integers or enumerations internally are defined as such (see stringmatch.c), andthey are translated upon load, any key NOT found in stringmatch's hash table is assumed to have a string value. (not an int, enum or other)

grep -v 'keys\|^\[.*$\|^#\|^$' *map.in |cut -f2 -d: |cut -f1 -d= |sort|uniq -c |sort -nu in Mtx's Gui dir will give you a list of all keys ever used in every datamap from the beginning of MTX. There are currently 64 keys of which maybe 15-20 are really applicable. There are a lot of special case ones for odd MS'isms, that I'd prefer to never see again.

Things specific to all controls:
page (tableID)
offset (offset in bytes from the beginning of the page/tableID)
size (in mtx this is a string representation of an enum), i.e. _U08_, _S08_, _U16_,_S16_,_U32_,_S32_
handler < This is an enum as well, this is primarily for special case controls that require specific handling or have an important interdependancy tht must be calculated BEFORE sending to the ecu. normally this is defined as _GENERIC_. I could handing this var being MISSING and default to _GENERIC_
group This is a string name of another section within the datamap file to get the rest of the attributes, (reduces duplication significantly)
tooltip The "help text" presented to the user if they hover over the control when tooltips are enabled.
bind_to_list This is a comma sep string list of arbritrary strings. Each string is a group name. The purpose is to group controls together so that they can be enabled/disabled en-mass from a bitmask related control (combo box, or radio/checkbutton)

Things specific to Entries:
dl_conv_expr,ul_conv_expr Mathematcial expressions for converting the ecu units to real world units. This accepts pretty much any sort of reasonably complex equation, but it DOES NOT handle case/binary logic. These are likely to be renamed real-soon now to toecu_conv_expr and fromecu_conv_expr asdl/ul gets confusing depending on your point of view.
precision Number of significant digits that should be displayed, typically only valid if there's a [ud]l_conv_expr in place.
temp_dep This is a boolean meaning this variable is a temperature, and needs to be converted from ECU temp scale to whatever scale the user is viewing in MTX. This CAN be used with [du]l_conv_expr, behavior is, on READ from ecu, the raw value is first converted via the expression, THEN temp scale conversion takes place.
use_color Boolean that enables the cell to be colored, the color is based upon the current value wiht reference to its upper and lower limits.
raw_lower,raw_upper These define the valid range of the control in ECU UNITS. The reason i use ecu units vs real world, is that in the case of some math expressions rounding could inadvertantly cause an overflow, hence ecu units limits will clamp and prevent that from ever happening. These two are REQUIRED when using use_color.

Things specific to comboboxes/radio/checkbuttons
bitval The bit value of interest when this control is enabled
bitvals a List of bitvals that correspond with the choices (combo box only)
bitmask The mask applied
choices A list of choices used to populate the combo box, must have same number of values as the bitvals key, otherwise an error will be reported
toggle_groups In the case of radio/check buttons, a list of groups to be toggled depending on this control's state. the groups are COLON separated. In the case of a combo box (multiple choice), the syntax is similar however each choice's groups are command separated. So basically its the groups that are corresponding to that choice are ENABLED, and all other choice's groups are disabled, the group state can be inverted with a preceding ! symbol:
eg. radio button's toggle_groups

Code: Select all

toggle_groups=somecontrols:someothercontrols
combobox's toggle_groups (when there's 3 choices)

Code: Select all

toggle_groups=choice1,choice2:choice3,choice1:choice4

Code: Select all

Toggle_groups=,choice2,choice3:choice4
(3 choice combo, no groups enabled on first choice, but groups enabled on the other two remaining choices

Its OK to have no groups for one of the choices, but there needs to be the correct number of comma's to have the same number of toggle_groups stanzas as there are choices in the combobox. (this can look kinda ugly for combo's with lots of choices)

By default in megatunix when setting up the gui and parsing the datamap, there's alway a section called "Default" which has GLOBAL defaults for all controls on that tab only, so the way it works for controls is first: assign defaults, second: assign any group defined, then third: assign local settings (defined for that ctrl), which allows the local settings to override the defaults or group settings, and allows group settings to override the defaults.
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

Re: Data Source File Format And Structure

Post by EssEss »

Fred wrote:What I'm going to suggest amounts to the same thing, though. I suggest that if a device is sending non-packed data, it must specify where the gaps are. IE, what is specced in the xml file must match what is sent.
fine with me.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

http://issues.freeems.org/view.php?id=169

Formally part of the todo list now!
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

EssEss, btw, the S12 compiler does pad structs! it just turns out that I always had even numbers in them! and arranged in pairs for bytes, so I never hit it. The hazards of being neat and tidy! :-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Data Source File Format And Structure

Post by Fred »

Dumping material out of the repo:
a structure can have N components
each component can be a primitive or other component

location has many packet types and one structure, structure has units and labels
structure has size, shape and structure name
packet types have a structure of their own and optionally a payload section where another structure can be embedded.
May be out of date.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply