View unanswered posts | View active topics It is currently Sun Nov 19, 2017 10:25 pm



Reply to topic  [ 10 posts ] 
Haruspex 
Author Message
LQFP112 - Up with the play

Joined: Sat Apr 24, 2010 1:42 am
Posts: 119
Location: New Zealand
So, I have a hard time trying to watch the road and laptop while driving, and so I thought I'd try come up with a tool to do it for me. The idea is to read through a log file, collate EGO readings at each VE cell, and made predictions on how to adjust the VE table.
https://github.com/johnhowe/haruspex

Code:
$ ./haruspex.py /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-KPALoad.h /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-RPM.h /home/john/git/freeems-vanilla/src/main/data/tables/ve/JohnsAE86VE.h  /home/john/git/freeems-vanilla/src/main/data/tables/lambda/JohnsAE86-AFR.h ~/ULV/2016.01.30-21.01.53.bin-export.csv

RPM axis file: /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-RPM.h
KPA axis file: /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-KPALoad.h
VE table file: /home/john/git/freeems-vanilla/src/main/data/tables/ve/JohnsAE86VE.h
AFR table file: /home/john/git/freeems-vanilla/src/main/data/tables/lambda/JohnsAE86-AFR.h
Datalog file:  /home/john/ULV/2016.01.30-21.01.53.bin-export.csv
Parsing datalog                                 
166503 samples seen; ignored 12351 warmup, 226 rpm, 49319 ego, and 13966 dtps
129356 samples rejected

 VE(40.0), VE(46.0), VE(45.2), VE(38.2), VE(34.4), VE(37.3), VE(40.6), VE(45.0), VE(45.0), VE(45.0), VE(45.0),
 VE(41.0), VE(43.0), VE(44.8), VE(35.1), VE(28.6), VE(33.7), VE(36.0), VE(40.5), VE(43.9), VE(45.0), VE(46.0),
 VE(45.0), VE(44.0), VE(45.4), VE(34.2), VE(30.2), VE(35.8), VE(39.3), VE(42.1), VE(43.9), VE(45.0), VE(46.0),
 VE(50.0), VE(48.0), VE(49.6), VE(36.0), VE(32.4), VE(36.0), VE(42.9), VE(49.8), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(51.1), VE(52.2), VE(39.8), VE(38.8), VE(38.2), VE(43.9), VE(49.8), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(55.0), VE(52.1), VE(42.4), VE(42.2), VE(39.9), VE(42.7), VE(49.1), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(55.0), VE(55.0), VE(43.5), VE(46.1), VE(43.1), VE(43.9), VE(52.4), VE(54.8), VE(55.0), VE(55.0),
 VE(55.0), VE(60.0), VE(60.0), VE(44.2), VE(50.8), VE(48.8), VE(48.8), VE(56.8), VE(59.7), VE(60.0), VE(60.0),
 VE(60.0), VE(65.0), VE(65.0), VE(50.6), VE(61.1), VE(56.8), VE(65.6), VE(63.2), VE(64.8), VE(65.0), VE(65.0),
 VE(60.0), VE(90.0), VE(60.0), VE(60.3), VE(67.7), VE(66.7), VE(70.0), VE(69.6), VE(69.9), VE(70.0), VE(70.0),
 VE(90.0), VE(90.0), VE(60.0), VE(60.1), VE(77.1), VE(81.3), VE(71.7), VE(80.4), VE(80.0), VE(80.0), VE(80.0),


Image
Image
Image
Image
Image
Image


Sat Jan 30, 2016 11:58 pm
Profile
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 14871
Location: Home sweet home!
Neat! A couple of questions/comments:

1) Why are you rejecting records on the basis of EGO? Railed values? Or?
2) Are the rejections done with some sort of lag or temporal input?
3) Why rejecting based on RPM?
4) There will be more rejection reasons, eg fuel cut/ignition cut, I'm guessing you haven't hit the limiter while doing this, but many would and it would render the readings useless.
5) Is it possible to vertically flip the tables so they look right?
6) Does it apply corrections with low confidence values? Will you add the option to not apply under a certain threshold?
7) Fix your target lambda table! :-p Your VEs will look more reasonable and your engine will run better if it's closer to usable. In particular, richen up the WOT area and close to WOT areas a bit, stoich your idle and cruise, and lean out your trailing throttle a bunch.

More later if I think of anything! :-)

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Sun Jan 31, 2016 4:23 am
Profile WWW
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 14871
Location: Home sweet home!
Does the confidence take into account the spread of values, std deviation wise, or just volume of points processed against a cell?

Does the confidence take into account how much weighting the cell had in that usage? / Are you properly spreading the reading's influence across multiple cells (4, 2, 1)?

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Sun Jan 31, 2016 4:26 am
Profile WWW
LQFP112 - Up with the play

Joined: Sat Apr 24, 2010 1:42 am
Posts: 119
Location: New Zealand
Quote:
1) Why are you rejecting records on the basis of EGO? Railed values? Or?

Yeah railed values, when EGO < 0.6 or EGO > 1.4
Quote:
2) Are the rejections done with some sort of lag or temporal input?

No lag compensation at the moment. I figured that would be good enough for now as it is rejecting pretty strictly on dTPS.
Quote:
3) Why rejecting based on RPM?

<100 RPM, from when the engine isn't running
Quote:
4) There will be more rejection reasons, eg fuel cut/ignition cut, I'm guessing you haven't hit the limiter while doing this, but many would and it would render the readings useless.

Yeah of course, this is just a starting point. Feel free to
Quote:
5) Is it possible to vertically flip the tables so they look right?

Could do, although it doesn't bother me. It matches the firmware header files.
Quote:
6) Does it apply corrections with low confidence values? Will you add the option to not apply under a certain threshold?

There is a veChangeFriction value which makes it harder for a VE change. This isn't a threshold though, it is stuffing the cells with that many samples at the target lambda value.
Quote:
7) Fix your target lambda table! :-p Your VEs will look more reasonable and your engine will run better if it's closer to usable. In particular, richen up the WOT area and close to WOT areas a bit, stoich your idle and cruise, and lean out your trailing throttle a bunch.

Sure, though that is irrelevant to this. How would changing the lambda table change the look of a VE table?
Quote:
8) Does the confidence take into account the spread of values, std deviation wise, or just volume of points processed against a cell?

Confidence is just sum of weightings on each cell, it has no relation to the EGO readings themselves
Quote:
Does the confidence take into account how much weighting the cell had in that usage? / Are you properly spreading the reading's influence across multiple cells (4, 2, 1)?

Yes, and yes


Sun Jan 31, 2016 6:17 am
Profile
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 14871
Location: Home sweet home!
With a 0.95 flat lambda table you have two choices:

1) Tune VE perfectly to obtain that result while running, car runs like crap at higher loads and burns excess fuel at lighter loads.
2) Tune the VE to compensate for the lambda map and get a nice running engine, in this case lambda and EGO will be far apart on purpose.

Given that you're trying to match lambda and EGO, you'd probably best put lambda in the right place so you end up with a VE table that makes your engine run nicely.

Make sense now?

Thanks for the rest of the answers, I might add some more tomorrow, but now I must sleep.

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Sun Jan 31, 2016 12:31 pm
Profile WWW
LQFP112 - Up with the play

Joined: Fri Sep 16, 2011 8:37 pm
Posts: 146
Location: New Zealand
Lubuntu Xenial required:

sudo apt-get install python-matplotlib python-parse python-seaborn


Wed Jun 29, 2016 10:44 am
Profile
LQFP112 - Up with the play

Joined: Fri Sep 16, 2011 8:37 pm
Posts: 146
Location: New Zealand
Gave it a fang and got this:
Code:
rob@rob-Satellite-L300:~/Projects/RobRB25$ python ../haruspex/haruspex.py src/main/data/tables/axis/FredsTruck-Load.h src/main/data/tables/axis/FredsTruck-RPM.h src/main/data/tables/ve/RobRB25VE.h src/main/data/tables/lambda/RobRB25Lambda.h ~/ULV/2015.09.19-21.29.30.bin-export-post-tune.csv
/usr/lib/python2.7/dist-packages/matplotlib/__init__.py:874: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))

RPM axis file: src/main/data/tables/axis/FredsTruck-RPM.h
KPA axis file: src/main/data/tables/axis/FredsTruck-Load.h
VE table file: src/main/data/tables/ve/RobRB25VE.h
AFR table file: src/main/data/tables/lambda/RobRB25Lambda.h
Datalog file:  /home/rob/ULV/2015.09.19-21.29.30.bin-export-post-tune.csv
Traceback (most recent call last):
  File "../haruspex/haruspex.py", line 297, in <module>
    main()
  File "../haruspex/haruspex.py", line 62, in main
    afrTable = importTable(args.afr[0], 'AP')
  File "../haruspex/haruspex.py", line 151, in importTable
    assert len(table), "No table data found"
AssertionError: No table data found


Hacked the script to look for LR() instead of AP() and got this:
Attachment:
ve.png
ve.png [ 277.13 KiB | Viewed 2017 times ]


With this difference:
Attachment:
deltaVe.png
deltaVe.png [ 166.92 KiB | Viewed 2017 times ]


On the first run I fed it tables from the wrong checkout/base tune. The script didn't seem to notice that the base table was different from the table within the logs. It should probably check and warn if the log is for a different base tune.

Very cool tool! Will chuck through some longer logs when...if I ever get some.


Wed Jun 29, 2016 11:53 am
Profile
LQFP112 - Up with the play

Joined: Sat Apr 24, 2010 1:42 am
Posts: 119
Location: New Zealand
Yeah ideally it would process the raw log files rather than require exporting as csv. Basically it will work with the way I have set up my engine (eg AFR table has same dimensions as VE table), and if by chance yours is the same then it should hopefully work.

My AFR table uses the 'AP' macro, so that is what it is looking for when reading the table https://github.com/johnhowe/haruspex/bl ... pex.py#L62
There is a simple check that the data is AFR rather than lambda before converting, so by switching to 'LR' and giving lambda values should still work as expected https://github.com/johnhowe/haruspex/bl ... pex.py#L65


Thu Jun 30, 2016 2:09 am
Profile
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 14871
Location: Home sweet home!
I reckon the CSV aspect is OK, as you would often want to trim your log, just as we did last night, cutting off 90% of the front full of tuning changes.

Good to know it handles petrol AFR and lambda equally well :-) Reminds me why the macro is AP and not AF or AFR, too :-)

The big issue in my mind is that it was happy applying a generated diff from the log file to a completely different table input and telling the user to apply that table to their tune. I guess predicting the future based on intestinal matter isn't always accurate, but I reckon you could do better :-D

So, suggestions:

1) Ensure log values are *close* to the table values (they might be exactly the same, but should be within the values of the cell boundaries)
2) Ditto, for both VE and Lambda inputs.
3) JSON import/export for tables including decouple of the axes for VE and lambda
4) A readme.md with some of the tips/gotchas from this thread, updated as fixed :-)
5) Config file with filters etc that overrides app-defaults and documents what is possible

If you JSONised it, I'd give it to Rob right now to play with on his Miata.

Anyway, it's useful for developer types as is. Good work :-)

Fred.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Thu Jun 30, 2016 3:54 am
Profile WWW
Moderator
User avatar

Joined: Tue Jan 15, 2008 2:31 pm
Posts: 14871
Location: Home sweet home!
johntramp wrote:
Yeah ideally it would process the raw log files rather than require exporting as csv.

Not sure if I realised this when I wrote the above post, but I did just realise it now, you meant to pull the
tables out of the log, right? I just meant to ensure that the data records in the log match the interpolated
data from the table you're supplied. You'd probably still want to sanity check this even if you pulled the
table out of the log (wrong table for eg). And if the tune is being modified, you'd want to discard everything
from before the final modification and ensure the user confirmed that they had supplied a table that was
obtained after the tuning found in the log, or confirm it yourself by checking the log data against it/them.

Checking for VE and Lambda is straight forward to do, and equally worthwhile as both things affect the end
result. In fact, VE is far more likely to have been modified in the log, too.

_________________
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!


Mon Aug 22, 2016 4:51 am
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 10 posts ] 

Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software for PTF. ColorizeIt.