FreeEMS communications stress testing

A place for small project requests from the team to exist and grow into something fully fledged and useful to the masses.
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

FreeEMS communications stress testing

Post by EssEss »

Fred, Sean[0], and I were kicking around some ideas for stress testing the comms interface of a standards compliant freeems implementation. I decided to volunteer where I could to write this app.

I'm still reading and coming up with a plan, but this is some prelim direction (not in any order):
  • - cpp + boost + cmake == (hopefully) easy and painless cross-platform with your choice of compiler and ide on each
    - metrics -> what do you want to see ?
    - command line
    - should be a quick 'single-shot' test which sends malformed pkts/good pkts and validate correct device behavior for each
    - attempt to break a device's implementation in the fastest way possible. It's me against your implementation and I want to win. :D
I'm almost thinking this could be sort of generic where I simply reuse an existing harness (cppunit or equiv) and concentrate on robust multi-platform i/o libs (or a full blown freeems comm stack) that the tests use. I may be getting ahead of myself for the moment ...

what other input is there ? I can edit this list as/if it grows and get some priority and focus for initial implementation.
I'm really tired now, and the list is thin .. help me out for the time being.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

If you break the thing down into regions, at the least protocol and firmware, then the program could be called against, one or the other or both. Also, if you wanna future proof it, ensure that you can add more sets. EG, vanilla gets tested with proto and vanilla and aircraft spec gtes tested with proto and aircraft etc etc.

You'll want a fail fast otpion and a test everything you can anyway option.

OK, where to begin! I don't want to poison your mind, but I can think of lots of things that need testing, so I'll try to list some.

start.validheader.data.start.data.stop make sure its rejected/ignored.

check that unused header bits have no effect by sending same packet multiple times with same guts and different header

send bad checksums, and ensure it fails. iterate through all possible checksums for an otherwise identical packet, ensure only one makes it through

set length field and ensure packets fail when the length varies or when the packet is too small for the length field
check that firmware includes length fields with packets that require it.
check that firmware rejects packets that have no length field and require it.

set sequence and ensure it comes back the same

verify reset commands function, either adjust ram and reset and ensure its back to normal, or look at some clock variable to see if the reset occured.

modifying internal data:
adjust ram by whole block and check that it changed by retrieving before, sending, then retrieving again.
ditto flash
ditto by parts
ditto for each type of block by whole and parts, eg mess up a tables axis and ensure it complains, and make a valid change and ensure it doesnt complain

make sure all packets are responded to except for resets (check that they dont respond and do reset)

ensure datalog streaming can be turned on and off and set to specific types (right now only one)
ensure polled datalog packets work similarly

ensure modifying table data via table specific methods works correctly. N table types * 2 or 3 functions each.

ensure all possible error codes are generated, and all possible packets are sent/recieved.

gather metrics on speed, bandwidth streaming, bandwidth polled, time to wakeup after reset (right now, shockingly bad due to init code writing to flash, so good to test now, will to back to almost instant soon), turn off datalogging and measure delay from last byte of sent packet to first byte of reply for all packet types (burn commands will be slow to respond, how slow?).

categorise results and statistics

have a dirty quick basic test everything once option (enough to ensure operation is fairly normal) and a thrash it for longer and in more ways option.

check that max packet size comes back accurately and that oversize packets aren't accepted.

poorly escape bytes, flood with escaped bytes, each good type of escape, etc.

bug me about gettting internal error stats out of it to compare with your actions on a lower level basis such as bad checksums, bad escape bytes, etc.

include the iface version and firmware version in your output

confirm that echo packets work upto some max length

send with wrong byte format and try various things to get a response, record contents/settings for post analysis.

ensure hammering on the thing with packets just results in dropped packets and intermittent responses.

send unknown packets and odd packets

send updates with bad size bad offset, size that doesnt match data
send ram updates for blocks only in flash
send flash updates for blocks only in ram (these dont exist, i dont think)

send update to ram and send burn to flash command reset and retrieve to check for burn.

check unimpl to get unimpl error.

grab arbitrary memory if you want. it should work.

OK, I'm out of brainstorming ideas, apologies for the mess, but its mostly good stuff :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

Request packets that shouldnt have a body should fail if a body is included.

(22:42:53) johntramp: "It's me against your implementation and I want to win." you should be scared fredcooke
(22:43:02) Quan-Time: wow.. could be interesting
(22:43:31) fredcooke: i want him to win, if he can.

:-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

Resources:

Test packets: https://github.com/fredcooke/freeems-va ... st.packets
Packet IDs: https://github.com/fredcooke/freeems-va ... ketTypes.h
Error codes: https://github.com/fredcooke/freeems-va ... rDefines.h

Size data is difficult to deliver, but we need to do it somehow. One way would be to use your tool to probe sizes by iterating through with an offset of zero increasing size from 1 - failure point and recording it in a json or xml file. Or the same with offset from 0 to failure.

The file that can lead you to size and address details: https://github.com/fredcooke/freeems-va ... lsLookup.c

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

Also, I've been meaning to write two pieces of code for quite a while, one that takes fake ADC values instead of real ones, the other that generates an RPM and trigger points and outputs without any input. With the two combined and your application we could thoroughly test the entire app without a JimStim, with the exception of the integrity of the outputs.

I'll be writing the decoder faker soon so that Joseph Davis can use his PCB as an injector flow bench controller. Unsure when the other will get implemented, but you could probably do some math verification with datalogs and RPM input alone even if the ADCs were just floating.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

Re: freeems communications stress testing

Post by EssEss »

The only thing I don't agree with is the testing of commands .. I was hoping this was going to be a pure packet format verifier. The only commands I planned on using was to get capabilities and a lot of echo packets.

Something that exercises commands (outside of a tuner app) seems to be something above and beyond pkt formats and now gets into payload verification. Can we split that kind of stuff off into a secondary, 'higher tier', app ?

after work, i'll compose an initial plan and post it here, and get to work on it.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

Well, if you want, sure, what you're proposing is a good start, but I'm pretty confident (misplaced? prove me wrong!) that it's not needed, whereas the payload stuff is more likely to have some issues.

Tier 1 = packets/protocol/protocol payloads?
Tier 2 = firmware payloads
Tier 3 = firmware code by RPM/ADC faking/manipulation

I don't see why it can't live in the same app, but if you don't like the idea, it's your project, do as you wish :-)

Besides, I was just brainstorming anything I could think of that you could test via serial, and that seems to be fairly thorough. I like testing as much as you, so I'm excited to finally have the possibility of some on the horizon. The more the merrier. 100% coverage is a fine goal.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

Re: freeems communications stress testing

Post by EssEss »

yes, a 'tier 1' app was what I wanted to attempt first.
I'm going off to get a little project started .. got a late start tonight.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: freeems communications stress testing

Post by Fred »

EssEss wrote:yes, a 'tier 1' app was what I wanted to attempt first.
Great, I look forward to watching serial LEDs blink furiously.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
EssEss
LQFP112 - Up with the play
Posts: 244
Joined: Thu Sep 10, 2009 9:23 am
Location: Dayton, OH

Re: freeems communications stress testing

Post by EssEss »

https://github.com/EssEss/freeems-cst

it's a start.
overwhelmed, but super excited at the moment.
Post Reply