Generating Data Source Files

Official FreeEMS vanilla firmware development, the heart and soul of the system!
Post Reply
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Generating Data Source Files

Post by Fred »

Recently, with inspiration from Dave, I have developed a fairly comprehensive mechanism for live from-the-running-firmware data layout interrogation. This is fantastic, because now all it needs is the actual field by field structure to be defined outside the ECU and the tuning software has all it needs, without overly complex unmaintainable files.

The next problem is one of maintenance also:

Duplicate data definitions.

If I keep my existing C files and write matching xml/js/json files then for every change I manually have to update both locations, make no typos and get it all perfect. If not, Dave (and users) will get angry and have less faith. It is also dirty and not at all OAOO to do this. Thus, I won't be doing this.

So, theree are two options:

2) Parse the source and generate XML/js/json files from it
1) Create and maintain XML/js/json files and parse them and output valid C (and possibly stripped down XML/js/json for use by tuner authors/programs.

Given that option 1 (Intentionally mis ordered/numbered to show bias) is far less error prone and better defined and more flexible, I strongly believe that is the best approach.

So, why a thread? I want it to be cross platform so the build will work on mac/lin/win equally well. We *could* revert to cygwin for win, but I'd prefer it stayed native too.

The first thing that comes to mind is XSLT

http://en.wikipedia.org/wiki/XSLT

But that would mean using XML, which I'm not a huge fan of. I'm not a huge fan of json either, but...

Is there something like this for json... searches :

http://goessner.net/articles/jsont/
http://goessner.net/download/prj/jsont/

How would I go about integrating that into the build process (which tools!) such that the C is generated before the compilation process?

Two or three or four transformations would need to be done:

Source data description > C headers
Source data description > C data defintions
Source data description > Simplified data description (structure)
Source data description > Simplified data description (data)

The last two aren't needed if the original rich file is suitable for parsing into an app, which it probably should be. The last one isn't needed if we don't mind pulling such default data out of a live ecu instead.

I'm not sure about the best way to proceed here, so I'd like some help, please! I know what I want, but I don't know how to achieve it. Hackers, scripters, germans, help me out! :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Generating Data Source Files

Post by dandruczyk »

I like option 1 better. (use some markup language and generate source from that). All of this can be done with makefiles,and fredcooke, you're probably better at it than I am.

The markup will need to be fairly descriptive and that's a battle for another day, bu the parser for it has to be cross platform. I like XML because I'm most familiar with it, (though i probably use it very incorrectly, but that can change) MTX already uses it and has libxml2 which is full featured, documented and avail on all platforms...
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Generating Data Source Files

Post by dandruczyk »

Depending on your language of choice for the markup, either develop a tool in C, or in somescripting langauge (perl?) that you compile FIRST as part of the build process that can then do the parsing/code generation in the next stage, and finally compile all that into objects and distributables that the tuning software will need. Ideally I'd like to see the distributables web-available so the tuning software if on a web-enabled system can query and download them as needed, which simplifies the distribution model somewhat.

This is similar to how QT projects are made, they use "moc" (meta object compiler), to generate code from other files as part of the build process, qmake does a lot of the behind the scenes stuff to generate the makefiles and rules (good reference), but the model for freeems is similar and can work with regular makefiles without needing a tool like qmake.
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

Re: Generating Data Source Files

Post by EssEss »

I definitely vote for #1 - gen code from data + template
Have you ever tried to make a c parser ?
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Generating Data Source Files

Post by Fred »

There is one in the bin directory that was being used briefly, but not anymore. I should probably purge it.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Generating Data Source Files

Post by Fred »

viewtopic.php?f=8&t=1007 - The data format itself. Please contribute if you have any idea what you're talking about, or not.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply