Comms protocol post implementation reflection, improving it!

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by sry_not4sale »

Our escaping scheme is just for converting start/stop bytes found inside a packet.
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by Fred »

tobz wrote:Well, do you have a link to the theory behind this escaping scheme? I only understand it as start/stop markers. You make it sound like it's a set of bytes injected every so often in a packet.
It is quite simple :-)

If you are going to have start and stop markers then you can't let them appear in the body as part of the data. Thus any that would be stop or start or escape bytes are subtracted from 0xFF and that is used intead, but it is prefixed with an escape byte such that you can know to unescape it. After that it is just a marker scheme. But without that it can't mark anything.
Also, where does the existing system handle fault-tolerance of spurious data? My changes wouldn't affect whether or not the system is fault-tolerant in that sense. That would be entirely separate, mostly related to checksum'ing the packets.
Sure, but you don't want to use data out of the packet before checking the checksum or it might be totally wrong. Thus you need to know where the boundaries are before you dig into the header looking for size/etc. If you layer the reception then you know the size before you check the header because you know when you have a full packet.

So long as you're checking the checksums, and rejecting bad packets, you should only lose performance. The ECU isn't going to know about bad packets; only in the sense that it may be requested to send a packet a second or multiple times until the receiver gets data it thinks is good. There should never be a need to fail to operate.
If you're looking to deal with horrendously noisy environments, I'd (also) be looking into a different layer 1 strategy like RS-422, for differential signaling. FTDI's FT232RQ chip can handle RS-422/485 so one of their cables or a custom box could be made. I was thinking of a USB type A plug into a little box with an Ethernet jack. CAT5 cable is already twisted and cheap so it'd be trivial to hook up to the unit. This may be overdoing it, but at roughly $8 FTDI chip/RJ45 jack + PCB runs and external components for the USB UART IC... it'd be relatively cheap for a rock-solid connection.
Agreed, but the system should work with the LCD and that is rs232 physical layer. Anyone with a spare 20usd will put a usb adapter on the board and be done with it.
Some things to think about, maybe. :P
The more the better :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

Fred wrote:
tobz wrote:Well, do you have a link to the theory behind this escaping scheme? I only understand it as start/stop markers. You make it sound like it's a set of bytes injected every so often in a packet.
It is quite simple :-)

If you are going to have start and stop markers then you can't let them appear in the body as part of the data. Thus any that would be stop or start or escape bytes are subtracted from 0xFF and that is used intead, but it is prefixed with an escape byte such that you can know to unescape it. After that it is just a marker scheme. But without that it can't mark anything.
Also, where does the existing system handle fault-tolerance of spurious data? My changes wouldn't affect whether or not the system is fault-tolerant in that sense. That would be entirely separate, mostly related to checksum'ing the packets.
Sure, but you don't want to use data out of the packet before checking the checksum or it might be totally wrong. Thus you need to know where the boundaries are before you dig into the header looking for size/etc. If you layer the reception then you know the size before you check the header because you know when you have a full packet.
If you're looking to deal with horrendously noisy environments, I'd (also) be looking into a different layer 1 strategy like RS-422, for differential signaling. FTDI's FT232RQ chip can handle RS-422/485 so one of their cables or a custom box could be made. I was thinking of a USB type A plug into a little box with an Ethernet jack. CAT5 cable is already twisted and cheap so it'd be trivial to hook up to the unit. This may be overdoing it, but at roughly $8 FTDI chip/RJ45 jack + PCB runs and external components for the USB UART IC... it'd be relatively cheap for a rock-solid connection.
Agreed, but the system should work with the LCD and that is rs232 physical layer. Anyone with a spare 20usd will put a usb adapter on the board and be done with it.
Some things to think about, maybe. :P
The more the better :-)

Fred.
Right, the only problem is simply how you achieve synchronization of the stream. If it's synchonrized, you don't need a marker. You know the header structure, and you have the neccessary information to parse continuously. It's only once that stream is interrupted/corrupted that problems happen, and thus your escaping + start/stop marker scheme or my sync packet idea allow the receiver to resynchronize.

It sounds like either way is going to work. I'm obviously a little partial to my way, but they'd both work fundamentally. I'd still suggest a size field in the header so receivers can utilize their memory a little more efficiently.

What about offering RS-232 *and* RS-422/485? Maxim has a nice IC for RS-422/485, the MAX489, and it comes in extended temperature ranges and is relatively cheap, around $3. Add the ability to pop that on the board and utilize it, or let people just use the RS-232 off of the MCU.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by Fred »

tobz wrote:Right, the only problem is simply how you achieve synchronization of the stream. If it's synchonrized, you don't need a marker. You know the header structure, and you have the neccessary information to parse continuously.
I'm not so sure. what if you have a continuous stream of data, sure, if its clean you can parse this, but what if it IS interrupted, how do you know and when do you know. Then that stream stops and there is a gap. Instead of a new signal, then there is some noise, but it is backed by a genuine signal, how do you know where the noise ends and genuine starts?
It's only once that stream is interrupted/corrupted that problems happen, and thus your escaping + start/stop marker scheme or my sync packet idea allow the receiver to resynchronize.
Yes, but to be 100% reliable (or as close to it as possible) you need to do this for each and every packet. Furthermore, if you don't escape then you have a risk (given the quantity of data and randomness of it, it's a certainty) that you will find your sync code in the stream. If your sync code was to work with decent reliability, it would need to be quite long, at which point it's worse than the 1% bandwidth overhead of escaping - and less certain. Plus, if it is long you have to look for multiple bytes in a sequence and buffer your state in the mean time etc. Far worse than looking for a single byte and having only a few states IMO.
It sounds like either way is going to work. I'm obviously a little partial to my way, but they'd both work fundamentally.
I'm not certain that yours works under ALL circumstances. I can see what you are saying and I agree that in a clean environment you *could* make it work, but in a dirty uncertain one with varying timings I'm not convinced. I'll take this opportunity to remind you that the escape scheme was not my baby. I just chose the values to use as nice obvious ones in a stream of hex. Stu thought it up and has used it in the past successfully. It's good to banter about it though as being challenged forces you to be sure of what you are saying and think it over. It's been a good thing to break me back into working on it :-)
I'd still suggest a size field in the header so receivers can utilize their memory a little more efficiently.
Sound like Aaron is already doing this and I'm not anti it myself. I will say though, in the first line of defense you only have to hold two full packets of data to capture things properly. After that you know exactly what you have and can allocate memory to suit based on captured size alone without even looking at header OR type. Once I've thoroughly reviewed the serial code and come to understand it again I'll pass further comment. For now though, just know that it's not ruled out :-)
What about offering RS-232 *and* RS-422/485? Maxim has a nice IC for RS-422/485, the MAX489, and it comes in extended temperature ranges and is relatively cheap, around $3. Add the ability to pop that on the board and utilize it, or let people just use the RS-232 off of the MCU.
I think the default card has got this on it anyway, but on a different than default channel. Does that support full duplex? My current scheme on rs232 does :-) It can receive data while sending it and parse it and deal with it and reply and only interrupt the stream where absolutely necessary. Pretty cool I think, but I would :-)

Check out the schematic for the TA card, it's in the download section of the main www.diyefi.org site somewhere.

Thanks again for your input, it's nice to talk about these things in detail, the more you do, the better and more concrete the solution will be.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by sry_not4sale »

Fred wrote:
tobz wrote:Right, the only problem is simply how you achieve synchronization of the stream. If it's synchonrized, you don't need a marker. You know the header structure, and you have the neccessary information to parse continuously.
I'm not so sure. what if you have a continuous stream of data, sure, if its clean you can parse this, but what if it IS interrupted, how do you know and when do you know. Then that stream stops and there is a gap. Instead of a new signal, then there is some noise, but it is backed by a genuine signal, how do you know where the noise ends and genuine starts?
It sounds like tobz header structure would contain some sort of unique start byte or structure to make it distinguishable
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

Well I can see what your concerns are. You're right, the existing way should let the receiver know where they are during a packet, not between it. My way would be periodic. Both of our ways are still prone to noise as far as I can see. The existing way just gives the receiver a chance to resync faster.
sry_not4sale wrote:It sounds like tobz header structure would contain some sort of unique start byte or structure to make it distinguishable
Nope, the header is not unique in any way. The "resync" packet would be for syncing against.


As far as the MAX489, yes, it supports full duplex transmission. Just did some more reading, though, and the MAX491 is full-duplex but supports up to 2.5Mbit speeds, so I'd probably go for that for headroom in the future.

http://www.rs485.com/downloadfiles/ssc232m.pdf

That link shows an RS-232 "extender" circuit, I'd simply be dropping in an FT232RL for the SP232E, more or less, for USB<=>RS-422/485. Doable in the size of a key fob dongle sorta thing, I'd say, although I'd ideally like to see something like an injection molded dongle.

For pricing, the FT232RL is ~$4.50, the MAX491EESD is ~$4.90, and the RJ-45 jack I'd be looking at is ~$2.70. Add in about $0.50 for the USB type A plug, $0.15 for a ferrite bead, and say $1.00 for resistors/caps, plus the cost of PCBs, and you end up at a relatively cheap USB<=>RS-422 dongle. Around $15 in parts, plus I'd say $5 per PCB if you did a decent-sized run of them.


Thoughts?
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by Fred »

Sorry for the delay on the reply here. I was offline for 3 days!! wow.
tobz wrote:Well I can see what your concerns are. You're right, the existing way should let the receiver know where they are during a packet, not between it. My way would be periodic. Both of our ways are still prone to noise as far as I can see. The existing way just gives the receiver a chance to resync faster.
Yeah, noise can screw up a packet, but it has no show of making past the multiple layers of checks without being caught. Anyway, seems like we have reached the end of this conversation :-)
As far as the MAX489, yes, it supports full duplex transmission. Just did some more reading, though, and the MAX491 is full-duplex but supports up to 2.5Mbit speeds, so I'd probably go for that for headroom in the future.
Thoughts?
Yep, the TA card has this already, but it uses the same UART module inside, so you need the same sort of robustness and state logic regardless. The state kept track of with my way is very minimal. You just know the buffer is being filled or not, then you know that it needs processing. In the middle you know if you are escaping something etc. Very simple straight forward stuff - important in my eyes :-)

Anything developed for the UART module should work equally well on all physical layers, and if not it's not really suitable. The existing code will work with that just fine.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
tobz
TO220 - Visibile
Posts: 14
Joined: Sat Apr 25, 2009 10:28 am
Location: Rhode Island, United States
Contact:

Re: Comms protocol post implementation reflection, improving it!

Post by tobz »

Yup, I'm just about ready to post my schematic for a USB-to-RS-422 adapter over in the general hardware section. :D
Post Reply