Page 1 of 1

Haruspex

Posted: Sat Jan 30, 2016 11:58 pm
by johntramp
So, I have a hard time trying to watch the road and laptop while driving, and so I thought I'd try come up with a tool to do it for me. The idea is to read through a log file, collate EGO readings at each VE cell, and made predictions on how to adjust the VE table.
https://github.com/johnhowe/haruspex

Code: Select all

$ ./haruspex.py /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-KPALoad.h /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-RPM.h /home/john/git/freeems-vanilla/src/main/data/tables/ve/JohnsAE86VE.h  /home/john/git/freeems-vanilla/src/main/data/tables/lambda/JohnsAE86-AFR.h ~/ULV/2016.01.30-21.01.53.bin-export.csv

RPM axis file: /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-RPM.h
KPA axis file: /home/john/git/freeems-vanilla/src/main/data/tables/axis/JohnsAE86-KPALoad.h
VE table file: /home/john/git/freeems-vanilla/src/main/data/tables/ve/JohnsAE86VE.h
AFR table file: /home/john/git/freeems-vanilla/src/main/data/tables/lambda/JohnsAE86-AFR.h
Datalog file:  /home/john/ULV/2016.01.30-21.01.53.bin-export.csv
Parsing datalog                                 
166503 samples seen; ignored 12351 warmup, 226 rpm, 49319 ego, and 13966 dtps
129356 samples rejected

 VE(40.0), VE(46.0), VE(45.2), VE(38.2), VE(34.4), VE(37.3), VE(40.6), VE(45.0), VE(45.0), VE(45.0), VE(45.0),
 VE(41.0), VE(43.0), VE(44.8), VE(35.1), VE(28.6), VE(33.7), VE(36.0), VE(40.5), VE(43.9), VE(45.0), VE(46.0),
 VE(45.0), VE(44.0), VE(45.4), VE(34.2), VE(30.2), VE(35.8), VE(39.3), VE(42.1), VE(43.9), VE(45.0), VE(46.0),
 VE(50.0), VE(48.0), VE(49.6), VE(36.0), VE(32.4), VE(36.0), VE(42.9), VE(49.8), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(51.1), VE(52.2), VE(39.8), VE(38.8), VE(38.2), VE(43.9), VE(49.8), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(55.0), VE(52.1), VE(42.4), VE(42.2), VE(39.9), VE(42.7), VE(49.1), VE(50.0), VE(50.0), VE(50.0),
 VE(50.0), VE(55.0), VE(55.0), VE(43.5), VE(46.1), VE(43.1), VE(43.9), VE(52.4), VE(54.8), VE(55.0), VE(55.0),
 VE(55.0), VE(60.0), VE(60.0), VE(44.2), VE(50.8), VE(48.8), VE(48.8), VE(56.8), VE(59.7), VE(60.0), VE(60.0),
 VE(60.0), VE(65.0), VE(65.0), VE(50.6), VE(61.1), VE(56.8), VE(65.6), VE(63.2), VE(64.8), VE(65.0), VE(65.0),
 VE(60.0), VE(90.0), VE(60.0), VE(60.3), VE(67.7), VE(66.7), VE(70.0), VE(69.6), VE(69.9), VE(70.0), VE(70.0),
 VE(90.0), VE(90.0), VE(60.0), VE(60.1), VE(77.1), VE(81.3), VE(71.7), VE(80.4), VE(80.0), VE(80.0), VE(80.0),
Image
Image
Image
Image
Image
Image

Re: Haruspex

Posted: Sun Jan 31, 2016 4:23 am
by Fred
Neat! A couple of questions/comments:

1) Why are you rejecting records on the basis of EGO? Railed values? Or?
2) Are the rejections done with some sort of lag or temporal input?
3) Why rejecting based on RPM?
4) There will be more rejection reasons, eg fuel cut/ignition cut, I'm guessing you haven't hit the limiter while doing this, but many would and it would render the readings useless.
5) Is it possible to vertically flip the tables so they look right?
6) Does it apply corrections with low confidence values? Will you add the option to not apply under a certain threshold?
7) Fix your target lambda table! :-p Your VEs will look more reasonable and your engine will run better if it's closer to usable. In particular, richen up the WOT area and close to WOT areas a bit, stoich your idle and cruise, and lean out your trailing throttle a bunch.

More later if I think of anything! :-)

Fred.

Re: Haruspex

Posted: Sun Jan 31, 2016 4:26 am
by Fred
Does the confidence take into account the spread of values, std deviation wise, or just volume of points processed against a cell?

Does the confidence take into account how much weighting the cell had in that usage? / Are you properly spreading the reading's influence across multiple cells (4, 2, 1)?

Re: Haruspex

Posted: Sun Jan 31, 2016 6:17 am
by johntramp
1) Why are you rejecting records on the basis of EGO? Railed values? Or?
Yeah railed values, when EGO < 0.6 or EGO > 1.4
2) Are the rejections done with some sort of lag or temporal input?
No lag compensation at the moment. I figured that would be good enough for now as it is rejecting pretty strictly on dTPS.
3) Why rejecting based on RPM?
<100 RPM, from when the engine isn't running
4) There will be more rejection reasons, eg fuel cut/ignition cut, I'm guessing you haven't hit the limiter while doing this, but many would and it would render the readings useless.
Yeah of course, this is just a starting point. Feel free to
5) Is it possible to vertically flip the tables so they look right?
Could do, although it doesn't bother me. It matches the firmware header files.
6) Does it apply corrections with low confidence values? Will you add the option to not apply under a certain threshold?
There is a veChangeFriction value which makes it harder for a VE change. This isn't a threshold though, it is stuffing the cells with that many samples at the target lambda value.
7) Fix your target lambda table! :-p Your VEs will look more reasonable and your engine will run better if it's closer to usable. In particular, richen up the WOT area and close to WOT areas a bit, stoich your idle and cruise, and lean out your trailing throttle a bunch.
Sure, though that is irrelevant to this. How would changing the lambda table change the look of a VE table?
8) Does the confidence take into account the spread of values, std deviation wise, or just volume of points processed against a cell?
Confidence is just sum of weightings on each cell, it has no relation to the EGO readings themselves
Does the confidence take into account how much weighting the cell had in that usage? / Are you properly spreading the reading's influence across multiple cells (4, 2, 1)?
Yes, and yes

Re: Haruspex

Posted: Sun Jan 31, 2016 12:31 pm
by Fred
With a 0.95 flat lambda table you have two choices:

1) Tune VE perfectly to obtain that result while running, car runs like crap at higher loads and burns excess fuel at lighter loads.
2) Tune the VE to compensate for the lambda map and get a nice running engine, in this case lambda and EGO will be far apart on purpose.

Given that you're trying to match lambda and EGO, you'd probably best put lambda in the right place so you end up with a VE table that makes your engine run nicely.

Make sense now?

Thanks for the rest of the answers, I might add some more tomorrow, but now I must sleep.

Fred.

Re: Haruspex

Posted: Wed Jun 29, 2016 10:44 am
by Corbon
Lubuntu Xenial required:

sudo apt-get install python-matplotlib python-parse python-seaborn

Re: Haruspex

Posted: Wed Jun 29, 2016 11:53 am
by Corbon
Gave it a fang and got this:

Code: Select all

rob@rob-Satellite-L300:~/Projects/RobRB25$ python ../haruspex/haruspex.py src/main/data/tables/axis/FredsTruck-Load.h src/main/data/tables/axis/FredsTruck-RPM.h src/main/data/tables/ve/RobRB25VE.h src/main/data/tables/lambda/RobRB25Lambda.h ~/ULV/2015.09.19-21.29.30.bin-export-post-tune.csv 
/usr/lib/python2.7/dist-packages/matplotlib/__init__.py:874: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))

RPM axis file: src/main/data/tables/axis/FredsTruck-RPM.h
KPA axis file: src/main/data/tables/axis/FredsTruck-Load.h
VE table file: src/main/data/tables/ve/RobRB25VE.h
AFR table file: src/main/data/tables/lambda/RobRB25Lambda.h
Datalog file:  /home/rob/ULV/2015.09.19-21.29.30.bin-export-post-tune.csv
Traceback (most recent call last):
  File "../haruspex/haruspex.py", line 297, in <module>
    main()
  File "../haruspex/haruspex.py", line 62, in main
    afrTable = importTable(args.afr[0], 'AP')
  File "../haruspex/haruspex.py", line 151, in importTable
    assert len(table), "No table data found"
AssertionError: No table data found
Hacked the script to look for LR() instead of AP() and got this:
ve.png
With this difference:
deltaVe.png
On the first run I fed it tables from the wrong checkout/base tune. The script didn't seem to notice that the base table was different from the table within the logs. It should probably check and warn if the log is for a different base tune.

Very cool tool! Will chuck through some longer logs when...if I ever get some.

Re: Haruspex

Posted: Thu Jun 30, 2016 2:09 am
by johntramp
Yeah ideally it would process the raw log files rather than require exporting as csv. Basically it will work with the way I have set up my engine (eg AFR table has same dimensions as VE table), and if by chance yours is the same then it should hopefully work.

My AFR table uses the 'AP' macro, so that is what it is looking for when reading the table https://github.com/johnhowe/haruspex/bl ... pex.py#L62
There is a simple check that the data is AFR rather than lambda before converting, so by switching to 'LR' and giving lambda values should still work as expected https://github.com/johnhowe/haruspex/bl ... pex.py#L65

Re: Haruspex

Posted: Thu Jun 30, 2016 3:54 am
by Fred
I reckon the CSV aspect is OK, as you would often want to trim your log, just as we did last night, cutting off 90% of the front full of tuning changes.

Good to know it handles petrol AFR and lambda equally well :-) Reminds me why the macro is AP and not AF or AFR, too :-)

The big issue in my mind is that it was happy applying a generated diff from the log file to a completely different table input and telling the user to apply that table to their tune. I guess predicting the future based on intestinal matter isn't always accurate, but I reckon you could do better :-D

So, suggestions:

1) Ensure log values are *close* to the table values (they might be exactly the same, but should be within the values of the cell boundaries)
2) Ditto, for both VE and Lambda inputs.
3) JSON import/export for tables including decouple of the axes for VE and lambda
4) A readme.md with some of the tips/gotchas from this thread, updated as fixed :-)
5) Config file with filters etc that overrides app-defaults and documents what is possible

If you JSONised it, I'd give it to Rob right now to play with on his Miata.

Anyway, it's useful for developer types as is. Good work :-)

Fred.

Re: Haruspex

Posted: Mon Aug 22, 2016 4:51 am
by Fred
johntramp wrote:Yeah ideally it would process the raw log files rather than require exporting as csv.
Not sure if I realised this when I wrote the above post, but I did just realise it now, you meant to pull the
tables out of the log, right? I just meant to ensure that the data records in the log match the interpolated
data from the table you're supplied. You'd probably still want to sanity check this even if you pulled the
table out of the log (wrong table for eg). And if the tune is being modified, you'd want to discard everything
from before the final modification and ensure the user confirmed that they had supplied a table that was
obtained after the tuning found in the log, or confirm it yourself by checking the log data against it/them.

Checking for VE and Lambda is straight forward to do, and equally worthwhile as both things affect the end
result. In fact, VE is far more likely to have been modified in the log, too.