Future Tuning Software Architectural Wish List (lower level)

Free Open Source Software project discussion forum. Post your Free Open Source software projects here!
User avatar
ababkin
LQFP112 - Up with the play
Posts: 215
Joined: Tue Jan 15, 2008 5:14 pm

Re: Future Tuning Software Architectural Wish List (lower level)

Post by ababkin »

i am guessing someone just has to look into writing a FreeEMS backend for the mtx and assist Dave in factoring out all that is general to MS/MSe/FreeEMS into a front-end
But we need a formal FreeEMS-comms spec figured out and defined first for that (to the extent MS/MSe has it defined).

Alex
Legal disclaimer for all my posts: I'm not responsible for anything, you are responsible for everything. This is an open and free world with no strings attached.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: The Great Tuning Software Wish List (your suggestions here)

Post by Fred »

I too am grateful for Dave bringing his comments here for us. And of course for all the work he's done towards a cross platform tuning tool for MS also.
jbelanger wrote:the MS2/extra variable sized tables was also easy to do in MT but not so in MTX
That may be so, but what a dirty hack it is to include them like that in the first place. There were several issues with that code once released to the world. The memory locations didn't line up etc etc. Including 12x12 for someone that says "auto analysing 16x16 tables with my 486 takes too long" is far from a good move. Decisions should be made for the right reasons, not because more people whinged about one thing than the other etc. The later method is what is known as politics and has made most of the world a mess. End rant.

What I mean is that although that could indicate a lack of flexibility, I'm not convinced that it's valid to say that because that is fairly special case in itself. What would happen to either MT or MTX or Serj's tool if you confronted it with data that needed to be analysed and stored in an indexable way prior to transfer to the box? All three would need a recode. It just isn't possible for something like that to be added via config IMO.

Additionally, you don't catch OEM standalone companies knocking out configurable tuning tools do you? Sure, someone might adapt them to something like MS if they did, but additionally, they are developing a pair of programs that are a platform to work with. A change to one almost *should* cause a change to the other to maintain perfect compatibility in the cleanest way.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
jbelanger
LQFP144 - On Top Of The Game
Posts: 387
Joined: Sat Feb 23, 2008 8:58 pm
Contact:

Re: Future Tuning Software Architectural Wish List (lower level)

Post by jbelanger »

I actually wanted to say something about it but forgot. I totally agree that this is a dirty hack which was done for all the wrong reasons and probably implemented because it was possible to hack with a relatively simple ini trick. Even though it might not even be a desirable feature, I think it's interesting to see why there are "limitations" on one side and not on the other and see if it can be as flexible as possible (even if it can have "undesirable" side-effects like the hacked variable table size). Dave has done something great with the way he did the gauges configuration which, as far as I understand, is completely done using external XML configuration files. On the other hand MT is seriously constrained on that aspect.

As for commercial tuning software, especially those dealing with OEM, the target audience is quite different and the number of firmware versions to support is quite limited so the constraints are not the same. Also, in a closed environment, the desire to keep things closed and under tight control is usually there for financial reasons.

Jean
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Future Tuning Software Architectural Wish List (lower level)

Post by Fred »

I agree, it is interesting to consider what allowed and disallowed certain aspects to change or not regardless of the goodness of it all.
jbelanger wrote:As for commercial tuning software, especially those dealing with OEM, the target audience is quite different and the number of firmware versions to support is quite limited so the constraints are not the same. Also, in a closed environment, the desire to keep things closed and under tight control is usually there for financial reasons.
I agree, however, the example I was trying to portray is that the person that mods the firmware is probably the same one that mods the tuning tool or at least collaborating closely with him. This is how I would hope we would work too. Either sequentially, I add xyz, then Stu/Dave/Eric (LOL) adds support afterwards, this is how it currently still occurs with Al and Lance for ms2 base, and Dave and everyone for MTX. It works well enough for the versions that are incremental changes. Besides which, if I want to test my change I'm probably going to add support for it myself whether a recompile (I mean, we are devs right, recompile is a minutely occurrence for us, it's not a big deal really) this is what J&K do with ms2e and MT. OR in parallel, ie, we discuss a change and agree on an interface spec before hand and then both go away and implement it. This is what I would prefer and is how a real commercial software system gets built with a clear interface between any two components developed first and coded to. This makes the most sense from a clean designed, not fiddled point of view.

discuss and agree
design interface first
code each side second
test together third
make further changes as required

In most cases I would guess adding a front end for a field or table will be much quicker than building the logic etc to live behind it and do the work so before the firmware is ready to go the tool will be waiting most likely.

Excuse the poor articulation above :-)

Change of subject :

xml (msq) vs binary stream for storing a configuration.

This is not what happens now, BUT, if the responsibility was placed on the developer to specify exactly what the changes are from previous to current and what things need to be added/changed a diff tool could be used to "fix" an xml file and change it from one version to another with a patch file or similar. Currently it is hit and miss, BUT, there is nothing to stop the firmware guys releasing a patch for xml/msqs to move them between releases. This could work too. It would just take a bit more responsibility and care and attention to detail on the part of the firmware dev.

I have to ask, what does an ms1extra mtx user need to do when he moves from one version to another? Can someone explain this process to me please? Surely not start from scratch? If so that would be a pain :-( it also wouldn't fit with the "user friendly" ideology very well.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Future Tuning Software Architectural Wish List (lower level)

Post by dandruczyk »

Fred wrote: I have to ask, what does an ms1extra mtx user need to do when he moves from one version to another? Can someone explain this process to me please? Surely not start from scratch? If so that would be a pain :-( it also wouldn't fit with the "user friendly" ideology very well.

Fred.
Due to megatunix's simplistic backup/restore, the only method suitable is for the user to EXPORT the ve/spark/afr other tabls as VEX files (another spec that SUCKS!), then start anew, importing those tables and entering the rest in manually. I o NOT like VEX files, they are a free-form ascii format that is a bitch to parse, and a trestrictive design .(XML works well for here as all tables have a common structure). Fatfingering is a pain, but it assures that everything is set.

Yes, this is VERY nonoptimal, I know. I haven't come up with an elegant enough way to compensate for it. It's an extremely sticky problem if you want to migrate settings between diverse firmware versions. (i.e. msns-e 024 to msms-e-highres-029), where the variabels changed, got shifted or moved. If I used a namespace system like megatune's MSQ's, I'm now locked into making 100% sure namespaces are consistent and unique across all possible firmware revisions (even from different developer groups), which becomes an insurmountable task. This is a problem, especially since many of the variable names are amiguous or just poorly chosen. if one decided to "clean up" the ini/namespace, it would be impossible to load an earlier MSQ onto that one due to the namespace differences, and numerous controls would go silently unset, leading to an increase in the problems already experienced by MT users when they load an earlier MSQ and wonder why some strange problem" came up, and they find the only solution was to NOT use the MSQ and fat finger everything in.. Thus megatune's .ini/msq's lock you into a fixed namespace, whcih is very restrictive, and if u change it later, everything goes to hell.

Namespaced based storage formats (XML, MSQ's, etc) work GREAT for a single vendor/source system where there is only 1 central source, not multiple competing diverging sources like in the diy community, as that requires to much inter-team communication and coordination (which slows development down), and the inherent disagreements aboiut how things should be named and structured. Everyone has their own opinions, and in the open source world, "everyone is right", or at least thinks they are.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Future Tuning Software Architectural Wish List (lower level)

Post by Fred »

I understand. It's one of the single most difficult aspects of the project IMO. That and the port management that I would like to be generic.

Thanks for taking the time to explain in such detail, very much appreciated!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Future Tuning Software Architectural Wish List (lower level)

Post by dandruczyk »

Fred wrote:I understand. It's one of the single most difficult aspects of the project IMO. That and the port management that I would like to be generic.

Thanks for taking the time to explain in such detail, very much appreciated!

Fred.
port management, r u talking in the ecu, or the serial port for hte tuning software? If serial I laready have that covered in megatunix, it's currently automatic.

As for ECU port mgmt. I guess it depends a lot on the capability of hte cpu. (i.e. TPU, (time program unit, like an IO co-processor), output compare, raw bitbanging, etc t. Each has various capabilities. TPU is probably the most powerful, but potentialy harder to code for (gcc may not have support) harder and more expensive. output compare is nice, but do u get enough ports?, bitbang adds a lot of SW complexity and timing issues.

I liek the idea of a few port maps with internal routing (no hardcoded port bindings wiht port classes). i.e. define a couple sets of ports:
high speed, OC/TPU ports (ignition outputs, injector if doing diesel)
moderate speed high precision (OC), injectors. PWM outputs
low speed gerneral I/O (bitbang), egr, A/C, fan, pumps, etc.

There's also the possibility of using i-wire devices for low speed I/O ports. i.e. seria ldatastream shared by 1 wire bus to tell, " component A go on, component B go off, etc..) IT's NOT for high precision/high speed devices, but when did you need your A/C compressor kickout to be done within 100 miliseconds.

Just an off the cuff idea though, as I'm not a HW designer.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Future Tuning Software Architectural Wish List (lower level)

Post by Fred »

mtx_man wrote:port management, r u talking in the ecu, or the serial port for hte tuning software?
Both, but not how you thought on the tuning side. Port/Pin selection for various features and functionality. I want it to be as free as possible and generic in implementation. I want that implementation (on both tuning and firmware sides) to include provision for serial/can/spi/I2C stuff also.

I want pwm boost control, and I want it on external spi peripheral 1 (from 1000 listening?) and pin number 5 of that device.

OR I want pwm boost control and I want it on internal pin (device zero?) 54 (which is a hardware pwm pin).

Or maybe even I want pwm boost control and I want it on internal pin 23 which is a gpio pin.

And let the code deal with making it work.

Perhaps it's an unattainable goal? I can see a pwm port driver that handles GPIO and PWM for those groups of pins and is just fed a duty and polarity and just updates all pins that are enabled constantly with a new value from a buffer. The buffer update frequency can be different to the actuator one. you could change the pwm duty every 1ms for all pins and update the value it reads from every 20ms or whatever.

My thoughts are that there are a certain number of each type of port that can do each subset of task. 8 pwm, 8 ic/oc, 24 interrupt pins, 90 digital input 90 digital output, 16 ADC input, etc The inputs should be swappable between variables if at all possible, the IC/OC channels are to be used in a hardcode way for fuel, and a set of gpio pins similarly for ignition. The balance of those pins will be available as gpio from some sort of table/system/mapping.

I would like some sort of internal system that the tuning software can maybe access where you can assign a feature to any port that has PWM capability and it will know how to use it. A list of ports should be maintained that has a bitmask of capabilities for each pin/port and a "in use" flag. The perhaps you can just fetch a list, go "these 3 are available with pwm, and the other 5 are used by these features, put which ever where ever you like. For pwm for example, the frequency is shared across more than one port, so you would put 4 things on one frequency, and 4 things on another etc. the frequency of a set of pins would not be tied to a particular feature but rather an independent setting that affects all features on a pin set.
As for ECU port mgmt. I guess it depends a lot on the capability of hte cpu. (i.e. TPU, (time program unit, like an IO co-processor), output compare, raw bitbanging, etc t. Each has various capabilities. TPU is probably the most powerful, but potentialy harder to code for (gcc may not have support) harder and more expensive. output compare is nice, but do u get enough ports?, bitbang adds a lot of SW complexity and timing issues.
Yes, it does, however for now we are using the XDP512 and in the future maybe XEP100, the same as what they are apparently putting on the MS3. The architecture should probably be generic enough to handle whatever core such that the tuning tool of choice is more easily moved to a new core with a different set of features.
I liek the idea of a few port maps with internal routing (no hardcoded port bindings wiht port classes). i.e. define a couple sets of ports:
high speed, OC/TPU ports (ignition outputs, injector if doing diesel)
moderate speed high precision (OC), injectors. PWM outputs
low speed gerneral I/O (bitbang), egr, A/C, fan, pumps, etc.
I like this too, but I don't think it is appropriate for the core fuel and ignition pins. For everything else, yes, that's exactly what we want.
There's also the possibility of using i-wire devices for low speed I/O ports. i.e. seria ldatastream shared by 1 wire bus to tell, " component A go on, component B go off, etc..) IT's NOT for high precision/high speed devices, but when did you need your A/C compressor kickout to be done within 100 miliseconds.
Exactly, I want to integrate support for this sort of thing from the start.
Just an off the cuff idea though, as I'm not a HW designer.
A good idea none the less though :-)

Fortunately we probably won't need to go to external stuff for the average user, but extending logging with lots more external ADCs is a great idea IMO. Log the whole car :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Future Tuning Software Architectural Wish List (lower level)

Post by dandruczyk »

If the ECU can provide INFO to the tuning SW regarding ports and capabilities. I can make it work.

i.e. lets say a command to retrieve the list of "global" pins, capabilities, module,module_pin

Rough idea of returned data:
uint16,uint32,uint16,uint16, uchar[16]
global_pin,capabilities_mask,module_num,mod_pin, global_pin_name

The ECU should be able to interrogate all devices on all busses to generate that table.
This allows for 65535 "global" pins, 32 possible capabilities per pin, 65535 possible modules, 65535 max "pins" per module.

If the ECU has engouh RAM,FLASH it would be ideal to be able to set that table in the code with an additional textual field, allowing the pin to be named from the tuning SW side, thus if anotehr tunig sw loads it it can present that info to the user. IT's nicer than having a hardcoded table with all of the supposedly "thoguht up possibilities" with a bucnh of generic_1, generic_2 and so on's.

Thus the tuning SW can run a command to get the table, and then present the data however is needed (i.e. sort by capability, then by pin number, or textual name, etc. Thus allowing the user to define new pins in the appropriate groups and assigning names to then for easier reference. In terms of using those pins, we'd have towork out a way to genericize the I/O in the firmware to make it modular so things aren't as "hardcoded" as they are now. i.e defien a mask of "used pins" and create various prioritized handlers for each active pin. (some handlers are more intensiv,e i.e. pwm vs simple digital I/O ), not sure how crazy u wana go wit hthe firmware though as this can be rather tricky todo and make it run as u want.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Future Tuning Software Architectural Wish List (lower level)

Post by Fred »

Before I start, I'd like to say this : We want to keep as much stuff out of the ECU as possible while retaining as much information inside the ECU as possible. What I mean by that is that if it can be done in the tuning tool, it should be done in the tuning tool unless that impacts functionality on the ECU side in a negative way. This keeps max resources available for doing real things.
mtx_man wrote:If the ECU can provide INFO to the tuning SW regarding ports and capabilities. I can make it work.
The ECU should certainly tell you what the current state of the table looks like.
i.e. lets say a command to retrieve the list of "global" pins, capabilities, module,module_pin

Rough idea of returned data:
uint16,uint32,uint16,uint16, uchar[16]
global_pin,capabilities_mask,module_num,mod_pin, global_pin_name
Yes, something just like this. I have to question the value of using up space on the chip to store names for the pins though. The tuning tool is already going to need to know about all the variables and how to access them etc etc, what is a simple 1:1 table of index:name going to hurt? It seems to me, not much?
The ECU should be able to interrogate all devices on all busses to generate that table.
Much like I'm not a fan of autotune and AMC and EGO correction, I don't think something like this should be automated. I think it should be setup explicitly. Such external devices will be strictly for advanced users anyway, so I don't see there being a need to make it overly simple for them at the expense of extra code and complexity on the ECU side.
This allows for 65535 "global" pins, 32 possible capabilities per pin, 65535 possible modules, 65535 max "pins" per module.
Sure, in terms of index it does, but the table on the ECU will have to have a finite limit of slots anyway, and at the end of the day, there are only so many features to pass off to external devices before you don't even have to drive the car anymore :-) A reasonable table size might be 256 entries? that's between 2 and 3 times the chips pin count which is already 3x the ms2's pin count. I think that should be fairly sufficient. We aren't trying to be a MOTEC here :-) In any case, the table size will be determined by a particular version anyway as some might not have room for such a large one with all the extra features they want to add in etc.
If the ECU has engouh RAM,FLASH it would be ideal to be able to set that table in the code with an additional textual field, allowing the pin to be named from the tuning SW side, thus if anotehr tunig sw loads it it can present that info to the user. IT's nicer than having a hardcoded table with all of the supposedly "thoguht up possibilities" with a bucnh of generic_1, generic_2 and so on's.
True, and there may be, certainly it doesn't need to be nailed down now. User assigned names could be stored in the tuning tools config pretty easily too. Switching tuning tools is a fairly significant thing anyway, and setup of pinouts is something you only do at initial setup or when you make a change to the system. Not something you reference each time you crack open the tool like VE or Advance. I'm tending to look at it like this though :

In terms of a pin, you only care about A what it is hooked up to externally which you need the Freescale name for B what it can do which the bitmask tells you. Calling it "pwm boost" when if you have pwm boost assigned to it, you should be able to display that without having it explicitly named to that seems frivolous to me.
Thus the tuning SW can run a command to get the table, and then present the data however is needed (i.e. sort by capability, then by pin number, or textual name, etc. Thus allowing the user to define new pins in the appropriate groups and assigning names to then for easier reference. In terms of using those pins, we'd have towork out a way to genericize the I/O in the firmware to make it modular so things aren't as "hardcoded" as they are now. i.e defien a mask of "used pins" and create various prioritized handlers for each active pin. (some handlers are more intensiv,e i.e. pwm vs simple digital I/O ), not sure how crazy u wana go wit hthe firmware though as this can be rather tricky todo and make it run as u want.
Yeah, I can see it being tricky for sure. It will definitely require a lot of thought to implement.

The more we discuss it, the closer to knowing how to do it we will get though :-)

Thank you for your input! Very much appreciated.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply