FreeEMS-Tuner Development Diary - Don't Post!

Aaron Barnes' wxPython based FreeEMS tuning tool. No longer maintained and out of date with the protocol requirements.
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

I have been talking to some guys in the local python user group, and it seems the use of python lists is what is slowing the app down so much.

Apparently switch to a native binary data type (the structs module) would give us a "thousand fold" increase :D

So thats the current job :D
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

My Toshiba laptop (only home computer at the moment (EEEpc died recently as well :roll:)) has died :roll: :evil: :indiff: :mad: :evil:

Will let you guys know how things go. Hard drive is ok thank god, but looks like either mobo, graphics card or ram is fuxxord
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

Have a new (old) laptop :D Acer Aspire 5000 widescreen, running Ubuntu 8.10.

Everything worked out of the box except wifi which only involved apt-getting a package

So, back into code!

Just pushed a commit (probably shouldn't have to master tho, but I did) that updates the serial protocol to match Fred's proposed changes :D
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

So.... got heaps done on the train this morning.

I am moving away from using python lists in the Tuner, and trying to stay with more binary suitable data types. It was suggested to me to use the Structs module, however it doesn't handle variable length packets :(

Sooo... I decided to convert all the code to use 8 bit strings and avoid looping as much as possible. Python 2.6/3.0 have a binary string data type, but I want to keep it 2.5 compatible.

Spent an hour on it this morning and pretty much have it almost converted. So far I have taken out twice as much code as I have added/modified, and removed almost every loop I have come across. The receive buffer processing code has gone from two monolithic functions, to two tiny ones :)

I have updated

Code: Select all

devtrunk
with my progress (although it hasn't been tested/prob wont run). Should have a working version tomorrow!! Tempted to work out in now at work, but it'll wait until tonight.
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

So pretty much finished the port.

Builds and sends packets fine. Even fixed a bug I didn't realise existed until I compared the output lol. Displays them in the gui grid fine too.

However, there is some bugs in the receive buffer parsing code. Its throwing a checksum error, but worse than that is it is not removing parsed bytes from the buffer. So going to look into this now. Won't push until I have fixed it.

Also, guy from the local python user group suggested that although the struct module doesn't handle variable length packets. I could just parse the header with it, discover the length of the payload and then parse out that. Problem with that is I would have to look up the rules for that packet to see if it was fixed length, etc, etc. So I think I will just stick with the string implementation I have, I think it is going to be a hell of a lot faster than it was anyways.

I have used the struct module in place of some perhaps dodgy int to 8bit conversion code. So now instead of from8bit() and to8bit(), we have shortFrom8bit(), intFrom8bit(), longFrom8bit(), shortTo8bit(), etc, etc.
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

Ok done a whole heap more testing, fixing.

Turns out the processed bytes weren't being remove from the buffer because I was trying to pass it by reference to the _processBuffer() method. Strings are immutable in python, so when they are modified you are creating a new string. Hope you can see where I am going with that...

Anyway, that is fixed now. Receives buffer beautifully, and processes and displays received packets no problem. Until a byte needs to be escaped that is. There is a bug causing an infinite loop in there I need to fix.

Found a great article regarding efficent string concatentation in python. If my new code doesn't produce the massive speed increase we require, I want to create a CLI for the comms/protocol code and do some tests using some of the alternative methods in that article, and some other ideas I have :)
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

Fixed the escaping bug, turns out it was an unescaping bug lol

Created a test packet with a payload of the bytes 0x00 - 0xFF, but that found a bug in the escaping code where the escape byte wasn't being escaped...
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

Fixed bug! Pushed to devtrunk.

Your turn Fred ;) testy testy!

I think there might be another bug tho, I have changed the request test packet to send the 0x00 - 0xFF bytes instead of "test". Sends find, Test response packet doesn't through any errors. However, I don't think its escaping correctly. Or maybe its escaping stuff twice.
Maybe someone could test the new code and have a look? My brain has turned to mush :P
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

No bugs fixed last night, but I did create a very simple CLI test framework and tested a couple of bugs.

So now I have a test framework to work against as I fix bugs, to make sure there aren't any regressions e.g. like it not handling 0xAA,0xAA properly anymore :P

I'll give the test framework a small tidyup and make it read from a json file so Fred can add his own test packets to it :)
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: FreeEMS-Tuner Development Diary - Don't Post!

Post by sry_not4sale »

Done heaps on the test frame work now :D

Now tests:

Code: Select all

    test_good,
    test_threeGood,
    test_doubleStart,
    test_oneBadByte,
    test_twoBadBytes,
    test_badByteBeforePacket,
    test_badBytesBetweenPackets
And fixed the receive code to actually pass them all ;)

It wasn't handling erroneous start bytes very well, so that is fixed. It also was doing a lot of copying of the buffer, which I have taken steps to reduce.

Changes pushed to devtrunk.

The testing interface is a bit messy, but it works. If anyone wants to make some improvements you are more than welcome!!!!
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
Post Reply