Over-The-Wire Unit Test App

A place for small project requests from the team to exist and grow into something fully fledged and useful to the masses.
Post Reply
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Over-The-Wire Unit Test App

Post by Fred »

I want to start unit testing the firmware in earnest! I can't do that in an efficient way with hand rolled binary packets, and writing this app myself would detract from my work on the firmware and project as a whole.

What The App Must Do
  • Send a hard reset to ensure a consistent state (though this shouldn't matter, perhaps make it an optional thing)
  • Disable log streaming before commencing (again, this shouldn't matter, but will ease the logic in this app, again, optional, if the app can filter packets)
  • Query information about the firmware that it is connected to and include them in the results
  • Send predefined data in appropriate packet wrappers to the device and correlate replies with this
  • Compare the reply with the defined expected reply
  • Summarise the results of all tests in some sort of output (console and/or gui and optionally file)
  • Read in individual test files or a test file with many tests in it and execute them all
  • Nice-to-have: Be able to enter data into some sort of UI to execute as a test live
Perhaps the reset and disable could just be manual buttons that you either press first or don't press first, before running the test(s)

CLI or GUI are both fine, perhaps even a tab in Tunix could handle this, unsure.

Test Setup Format
  • TestName = <arbitrary string>
  • TestDescription = <arbitrary string>
  • IDOfFunctionUnderTest = <0 - 65535>
  • ArgumentData = <string of comma separated hex values>
  • ExpectedReturnState = Error/OK
  • ExpectedReturnData = <empty or up to about 2k of whatever, including required error code if expecting an error>
If each is inside an xml or json structure then a set of them could just be catted together with an opening and closing tag and made into a test set. Such a set should have a title and description. A set of sets could be a test suite and have a title and description, as well as a intended version to test (git hash or release) such that the failure and/or success of the scripts makes sense.

The packet types/payload IDs are:

Code: Select all

#define requestInterfaceVersion     0x0000
#define requestFirmwareVersion      0x0002
#define requestHardSystemReset      0x000A 
#define requestUnitTestOverSerial   0x6666
#define requestDecoderName          0xEEEE
And the format of the data is defined by the ID under test, however the test app doesn't need to know about this.

So, anyone care to write this? I'd say it would take a decent dev less than a day to knock out. Longer with a GUI or in Tunix.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Over-The-Wire Unit Test App

Post by dandruczyk »

IDOfFunctionUnderTest = <0 - 65535>
ArgumentData = <string of comma separated hex values>

IS this a complete Packet in hex (header/sequence/data/checksum? If it is not, the software will need to know how to format the packet correctly and this introduces many places where errors could be induced, as various payload ID's have different payload content/layouts.

I'd prefer the test files have this as a full raw packet (in hex), thus eliminating all the special intelligence needed in the app to generate packets based on payload ID's (which would require code changes each time new packets are designed/developed/implemented, which is unfavorable), that the tool could basically send and gather the response and display/write them out, as well as run batches or repeating cycles of them.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Over-The-Wire Unit Test App

Post by Fred »

The point of having an app for it is to not have raw hand rolled packets :-) The packet format is basic, include an ID at the beginning, and all the data after it, checksum it, seal it and send. Take the reply, pull the NACK bit and payload data out and compare it with what was expected, display pass/fail and if fail what you got vs what you expected. I'll get you a screen shot of JUnit in action in a few mins.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Re: Over-The-Wire Unit Test App

Post by dandruczyk »

Fred wrote:The point of having an app for it is to not have raw hand rolled packets :-) The packet format is basic, include an ID at the beginning, and all the data after it, checksum it, seal it and send. Take the reply, pull the NACK bit and payload data out and compare it with what was expected, display pass/fail and if fail what you got vs what you expected. I'll get you a screen shot of JUnit in action in a few mins.

The packet payload format VARIES based on the payload ID since the software won't have any idea of it, how is it to assemble a packet? the header may be consistent, but the payload is payload ID specific.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Over-The-Wire Unit Test App

Post by Fred »

EDIT, no, the payload ID is fixed and is

Code: Select all

#define requestUnitTestOverSerial   0x6666
As posted in the first post in this thread.

The payload contents are verbatim what is provided to you. The response payload is anything, just compare it with what you were told to expect and produce output.

This isn't what I hoped to find, but it'll do:

Image

Something like this set:
  • Test failure! Expected NACK with error code 0x1234, got ACK with data 0x54,0x23,0xAA,0x66
  • Test failure! Expected NACK with error code 0x1234, got NACK with error code 0x5678
  • Test failure! Expected ACK with data 0x54,0x23,0xAA,0x66, got ACK with data 0x22,0x11,0x55,0x99
  • Test failure! Expected ACK with data 0x54,0x23,0xAA,0x66, got NACK with error code 0x5678
  • Test failure! Expected ACK with data 0x54,0x23,0xAA,0x66, got ACK with no data
  • Test passed!
or similar. Make sense now?

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Over-The-Wire Unit Test App

Post by Fred »

Hello people of FreeEMS land! This is another side project that needs doing! If anyone is keen, please post here and get started :-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply