EMS Data Map Format - Please Discuss!

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

EMS Data Map Format - Please Discuss!

Post by Fred »

We need a way of representing what is inside the EMS in terms of data and how to get it out. This should be maintained by myself and/or other firmware guys. I was originally thinking that I learn to write a python data object, BUT, thats no use to Marcos or David or anyone else wanting to interface with it.

So, we need a generic format that each app can parse and use to build structures and interact with the device. I need to maintain it, and you all need to be able to parse it. I want to maintain it so that I have some control over the quality of the mapping from raw data to meaningful objects.

There will be the following blocks of data :

2 sets of
{
3 x 1k VE tables
1 x 1k lambda table
2 x 1k fuel timing
2 x 1k ign timing
}
= 16 tables each block is one data structure

2 sets of
{
1 x 1k live tunable vars and small tables
1 x 1k live tunable vars and small tables
1 x 1k live tunable vars and small tables
1 x 1k live tunable vars and small tables
}
= 8 1k blocks of data with many smaller tables and arrays and vars embedded. I intend to make each small table referencable by ID individually, but burning down will need to be done as a block for now.

1 set of
{
1 x 1k fixed non live tunable configs
1 x 1k fixed non live tunable configs
}
= 2 1k blocks in flash only with similar embedded settings etc.

1 set of
{
1 x 2k IAT lookup table
1 x 2k CHT lookup table
1 x 2k MAF lookup table
}
= 3 2k blocks in flash only for converting ADC readings to real values.

29 blocks so far, IDs are 0 - 28

RAM read and write for those that are in ram
Copy from ram to flash for those that are in ram
Flash read and write for all
tables can be adjusted in 3 ways other than a full block write : x axis, y axis, cell by x and y.
full tables are validated when sent. the embedded tables may be validated too, but unsure yet. may take too much flash in code...

inside the config stuff, each item will have a size and offset. it should also have a reference by name and a display name, and a description of what it does and is about. I should provide most of that as I'm writing the code that uses them...

So, what format do we want, plain XML, what structure, XSD driven for preciseness?

Please shoot with ideas and or examples of structure definition files if you are feeling like designing them!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by sry_not4sale »

XML??? why not JSON?

xml has alot of overhead, and there are plenty of lightweight json interpreters now.
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
SleepyKeys
LQFP144 - On Top Of The Game
Posts: 549
Joined: Mon Feb 11, 2008 10:52 pm
Location: Arizona
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by SleepyKeys »

For what it's worth XSD would give it a refined touch. Just one more area
where we smoked the other guys. There's tons of XLM libs out there for tons
of platforms.
You snooze, you lose!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

sry_not4sale wrote:XML??? why not JSON?

xml has alot of overhead, and there are plenty of lightweight json interpreters now.
Because I hadn't even heard of it :-)

Do you want to take the info above and take a stab at writing the spec as it stands (roughly)?

On another note, this could define an interchangable settings format too.

It should probably have meta data such as firmware interface version and for the stored configs, generating tuning app name/id etc.

Ideas on how to link stored data, communication ideas, display ideas etc all together are welcome.

Speed isn't necessarily important as you are only going to load this once at boot if you use it for your apps internals, and only at save/load time for config storage etc.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

I've had a read up and direct conversion is possible between xml -> json, however the reverse is not true.

Unless there is a better suggestion than either of these I think I should generate an xml document, check it with an xsd, transform it with a tool, check that somehow and release both formats.

I'd like to see some discussion on fields and info that we would like to see present in the datalog and setting files that will help us keep track of things and keep them compatible etc.

I may take a preliminary stab at this shortly as I would like to bring a more final look to the me and the world such that its the display stuff for them to work on and the math and realtime stuff for me to work on. This makes the most sense to me in terms of overall progress.

I wish that I had identified this need to interface with other designers (soft and hard) earlier as it would have smoothed up this time for all of us.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

Gearhead pointed out that we need field scaling and offset fields too. At that point we probably need units too and field names for aspects of tables etc. Things to think about :-)

He also suggested that var names in the code and interface and tuner should match fairly closely, I agree with this and will endeavour to make the code ones meaningful anyway, but tuning types should stick with names similar to what I use, or the ones I provide in the up coming doc and keep the description to clarify rather than renaming arbitrarily.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

A highly incomplete first attempt.

Missing are units, scaling and offset for each data type and probably other things.

Unclear is whether each access method should be described separately or whether a list of payload ids and a receive payload format is enough?

Thoughts, modified version by you guys? Comments, criticism etc etc.

This is my first JSON file, and I do like it, but this is going to be a HUGE job...

Maintaining it will be better though.

It will definitely need a schema and validator to ensure its in good shape as it will be hand written...

Fred.
Attachments
first.attempt.zip
(1021 Bytes) Downloaded 740 times
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

Karsten was too lazy to post this link :

http://en.wikipedia.org/wiki/Asn.1

Perhaps he will post some thoughts or perhaps not :-)

He did say
(12:15:13 AM) Karsten: yuck
(12:15:18 AM) Karsten: that looks very ad-hoc
About the MSQ and INI ms formats :-) I concur. Let's do this right the first time!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by sry_not4sale »

Looks good, much better than xml dont ya think ;)

One comment - is it worth maybe replacing the spaces in keys with underscores (or just CamelCase)? In case some parsers have a json -> object/strut option that doesn't allow for spaces?

Who knows if its just a legitimate concern, but I think it would allow more flexibilty and less parsing, without sacrificing much readability
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: EMS Data Map Format - Please Discuss!

Post by Fred »

I agree and had thought about it. Still slightly scared by the size of it etc though.

Yes, it is nicer than XML, but that wouldn't be hard :-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Locked