Comms protocol post implementation reflection, improving it!

Official FreeEMS vanilla firmware development, the heart and soul of the system!
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

If you want to talk pure bandwidth savings, it's going to be no different using a size field in the header than a start/stop byte. If I'm not mistaken, the talk on the start/stop stuff I've seen has hinted as possibly more than 2 bytes total to delimit a single payload. Then, the size field is saving you bandwidth.

In the cases of static structures, you *could* omit the size field. That then requires extra code to handle packets from the ECU. (Remember we were talking about simplicity? :p .. or were we?)

Let's assume we have a serial link speed of 9600bps and 115200bps, and let's assume a datalogging packet is 100 bytes with no start/stop because it's a fixed size.

Normal packet, fixed (no start/stop) (98 bytes) (9600bps): 12.24 packets/sec
Normal packet, fixed (no start/stop) (98 bytes) (115200bps): 146.93 packets/sec
Normal packet, size field in header (100 bytes) (9600bps): 12 packets/sec
Normal packet, size field in header (100 bytes) (115200bps): 144 packets/sec

As we can see, aside from any overhead/not being able to utilize 100% of the line speed, the difference between a fixed packet and a packet using the header with the size declare is nominal.

You have two main sources of data that will be filling up your serial line, and that is datalogging and tuning. Pushing every single parameter you can for a datalogging packet, or updating entire fuel maps at a time are definitely no small operation. The great thing is that we know datalogging will be able to operate at a very acceptable speed, and fuel maps are a one-way push to the ECU, not interactive like datalogging updates, and neither happen simultaneously.

While the guesses I made for packet sizes are probably off, it's not hard to see that you're gonna be fine if you had a fixed header.

I won't sit here and tell you this is *the* way it needs to be done, but having a single, standard header for every single packet is definitely better than having it be a per-packet basis, and it won't be slower than using start/stop bytes.

EDIT: Just to clarify on a question you asked, instead of start/stop bytes you simply specify how long the payload is. If we know the header length (we do, because it's a fixed set of fields) then we simply look at the size field to determine how much data is left to read.

Example:
payload ID, size, data, checksum
[ 00 3D ][ 00 04 ][ FF FF FF FF ][ 7F ]

You have the payload ID, size field, and checksum. These are part of your header. The checksum isn't in the front, but it's a field we know is going to be there 24/7, so it's fixed.

The size field tells us how long the payload is. We read 4 bytes initially to get our payload ID and size. Here, we have enough data to do things like: see if a packet sent us enough/too much data. If the size was 0xFFFF for a packet that contains, at a maximum, 10 bytes of data, we'd know there was a problem and we could reconnect or send up a message box to alert the user.

After we read 'size' more bytes, we have our payload. Now we only need to read one more byte, and that's the checksum. Then we can calculate our own checksum for the packet (payload ID + size field + payload) to make sure everything came over the wire correctly.
User avatar
jbelanger
LQFP144 - On Top Of The Game
Posts: 387
Joined: Sat Feb 23, 2008 8:58 pm
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by jbelanger »

You're still missing Fred's point. There is a mode where the ECU is sending data log packets continuously without the PC constantly asking for it. In that case, it is possible for the PC to be connected to the ECU while the ECU is already sending data.

With the start/stop bytes, it is possible to sync to the data. Without it, I don't see how to do it and that was Fred's point also, I believe. In that case how do you propose to sync with the data stream if you don't have a repeatable pattern of bits/bytes somewhere in the packet? A payload length field will be of no use in that case. And the start/stop will also be of use is there is some interruption in the serial comm for whatever reason.

Jean
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

jbelanger wrote:You're still missing Fred's point. There is a mode where the ECU is sending data log packets continuously without the PC constantly asking for it. In that case, it is possible for the PC to be connected to the ECU while the ECU is already sending data.

With the start/stop bytes, it is possible to sync to the data. Without it, I don't see how to do it and that was Fred's point also, I believe. In that case how do you propose to sync with the data stream if you don't have a repeatable pattern of bits/bytes somewhere in the packet? A payload length field will be of no use in that case. And the start/stop will also be of use is there is some interruption in the serial comm for whatever reason.

Jean
Well, I wouldn't be streaming like that. If I *was* streaming like that, though, I suppose I'd probably do something like send a sync packet/message... something like 0xDEADBABE or 0xDEADBEEF. Send it every half a second or something, and you'd have a worst case delay of half a second when you wanted to see datalogging values.
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: Comms protocol post implementation reflection, improving it!

Post by jharvey »

Hmmm, do you mean a sync code like a barker code? I know those are common place in low bandwith noisy radio enviroments.
http://en.wikipedia.org/wiki/Barker_code
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

jharvey wrote:Hmmm, do you mean a sync code like a barker code? I know those are common place in low bandwith noisy radio enviroments.
http://en.wikipedia.org/wiki/Barker_code
Looking at the wiki page, I'd say that a barker code is pretty much analogous to my idea of a "sync" code. The barker code looks like it has a little more math behind it though, maybe due to the fact it's used for radio signals?
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by Fred »

Thanks Jean :-) I'm not sure if tobz was missing my point or it was my failure to articulate it properly, either way, it doesn't matter as it is clear now :-)

One of the design requirements from the outset of this serial side show was to be able to operate in a horrendously noisy environment or with bad hardware (both very common IRL) with only a drop in performance or a failure to operate at all. Spurious data that could still make it in with the barking code setup isn't acceptable to me really. I want to wait on average 50% of a packet before I KNOW where I am. The escape scheme (not my idea, much credit to FlappySocks for that) seems to me to be the only SURE way to do it while maintaining speed. In fact, that method already uncovered a few bugs due to it's inherent reliability. You'd have to put up a VERY good argument to make me want to drop the escape scheme.

Fixed headers on the other hand I am open to. It isn't overly complex right now, and the logic flow goes something like this :

look for start, discard and start buffering
look for start, escape or stop
for start, reset and start buffering again
for stop, seal the packets fate and go check stuff
for escape, un-escape the next byte and verify that it is a valid byte to be un-escaped

once you have this well defined chunk of data that was terminated cleanly by the start and stop pair you can process it any way you want.

The current setup means you should grab the last byte and sum the rest to see if the checksum is good, then you have a pretty much 100% guarantee that you have a good block of data. With that block you grab the first byte, and by the bit flags in that, grab various fields and chunks from the rest of it and parse them into usable format. Because you are already checking the bit flags, it is very easy to just grab the ones you want and find the start of the payload even with on/off-able length, addresses, etc.

I wrote the packet grabbing part in C that executes at several orders of magnitude faster than the data stream including generating some stats on it. I don't find that piece of C complex and it didn't take long to write. I'd say that complexity is definitely an acceptable cost for something so fundamental as solid comms. I'm not sure if you have played with megasquirt much but many of the ideas I'm working toward on this project are as a direct consequence of seeing megasquirt's weaknesses in the field. It's not nice when your engine suddenly starts bucking and going nuts because you burned garbage to the main table. I've seen this in person a few times, not cool.

Anyway, fixed headers, maybe, I need to get into the code, play with the tuner and get a real world feel for what would be best.

I very much appreciated your input as it is well considered and you have demonstrated a lack of defensiveness which is crucial around me as I (seem to) offend people too easily :-)

I know I have quite a bit of work still to do on the comms stuff soon, I'm unhappy with the way the code is at the moment on my side. I think Aaron is probably unhappy with the performance of the python too :-) We'll get there in the end. I hope you enjoy the ride as much as I have and will be.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by sry_not4sale »

Thought I would just make my thoughts on this public...

- I agree with Fred, I think we need a start byte at least to be able to determine where a header starts
- Also, at the moment I just add a length flag and byte to all packets I send (that have a payload). Not bother to check if that payload type needs a length field is just one less thing to do, for an extra 2 bytes saving?
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: Comms protocol post implementation reflection, improving it!

Post by jharvey »

For those that want to know more than most care to know, the barker code is a sync code, much like 232's first bit will be low (or high) the sync code is simply used to sync the clocks together, and has to be right to ensure your packet is right. Like most packets, it's common to have checksums, ect to confirm your data was recieve correctly. I seem to recall LTR is a fixed set of bits, but there are variable versions out there.

http://www.weavercomm.com/ltr.html

Setting up a barker code is simply a longer version of the start bit for 232. Personally, I don't see a need for this type of comms right now. In my oppinion, we don't need speed right now, we need it working, with the option of speed in the future. Set up something that allows you to download the fill memory map now, and go work on the RPM section.... Pressure, pressure, wipe crack, wipe crack.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by Fred »

You could do it without stop, just start/escape/length, but you then would have to read the bytes of the packet BEFORE it finishes AND before you checksum it = not solid IMO. I think you have to have the start and stop to be able to do it cleanly in a layered way.

I'll think about the length stuff more before making any further comments on it.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

Fred wrote:One of the design requirements from the outset of this serial side show was to be able to operate in a horrendously noisy environment or with bad hardware (both very common IRL) with only a drop in performance or a failure to operate at all. Spurious data that could still make it in with the barking code setup isn't acceptable to me really. I want to wait on average 50% of a packet before I KNOW where I am. The escape scheme (not my idea, much credit to FlappySocks for that) seems to me to be the only SURE way to do it while maintaining speed. In fact, that method already uncovered a few bugs due to it's inherent reliability. You'd have to put up a VERY good argument to make me want to drop the escape scheme.
Well, do you have a link to the theory behind this escaping scheme? I only understand it as start/stop markers. You make it sound like it's a set of bytes injected every so often in a packet.

Also, where does the existing system handle fault-tolerance of spurious data? My changes wouldn't affect whether or not the system is fault-tolerant in that sense. That would be entirely separate, mostly related to checksum'ing the packets.

So long as you're checking the checksums, and rejecting bad packets, you should only lose performance. The ECU isn't going to know about bad packets; only in the sense that it may be requested to send a packet a second or multiple times until the receiver gets data it thinks is good. There should never be a need to fail to operate.

If you're looking to deal with horrendously noisy environments, I'd (also) be looking into a different layer 1 strategy like RS-422, for differential signaling. FTDI's FT232RQ chip can handle RS-422/485 so one of their cables or a custom box could be made. I was thinking of a USB type A plug into a little box with an Ethernet jack. CAT5 cable is already twisted and cheap so it'd be trivial to hook up to the unit. This may be overdoing it, but at roughly $8 FTDI chip/RJ45 jack + PCB runs and external components for the USB UART IC... it'd be relatively cheap for a rock-solid connection.

Some things to think about, maybe. :P
Post Reply