Serial protocol pre-design discussion - give us your input!

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

I've just released 0.4 of the serial spec :

https://sourceforge.net/project/showfil ... _id=610714

Newly finalised are :
  • 8 bit CRC agreed on.
  • A header bit defines whether payload type is from the protocol or firmware set
  • Strings will be terminated with the null byte
Yet to be resolved are (yes, I know it is longer) :
  • CRC semantics - should the header be included in the main check?, CRCed separatey?, or header not checked
    jbelanger wrote:Also, it would be a good idea to protect the header part of the message with it's own checksum (or another more robust method).
    thebigmacd wrote:The reason I reserved 4 bytes in the CRC field is I like having a separate CRC for each field in the datastream
    Please discuss.
  • CRC semantics - should the CRC be done on the pre escaped data, or post escaped data?
    Please discuss.
  • Flow control - hardware, software, ignoring incoming requests when busy - requirements for each etc.
    http://microcontrollershop.com/download ... E64man.pdf page 12, they discuss using normal pins for hardware control and the jumpers that connect them. I found other references to hardware control via pin toggling too so my assumption that this was done manually with bit banging in software was correct. The plug itself is connected to 8 solderable holes on the board. Labeled W 7 - 10. These are usually used for bridging the connections for loopback use I guess. We can use them to hook some CPU pins and do our flow control. I don't see a need for two directional control. The PC should be able to take anything thrown at it. The FreeEMS on the other hand really needs to be able to say "I'm busy, don't send" some of the time. Because of this I think that one extra pin for FreeEMS to control the PC is enough. Can anyone comment on this? Basiclly I expect it will work like this : the pc can send requests any time except when told it can't. It will be told it can't when the FreeEMS is busy handling a request. So, at boot pc is able to send. pc sends request, FreeEMS recieves the end of that request, disables the interrupt, turns the stop control line to stop, deals with the request, sends a block of ack or data back and then reenables reception of data where the process repeats.
    Image
  • Stretchy ints, yes or no?, if yes, how?, if no, how big for numbers?
    I have Stu's example code and need to analyse it and discuss what I find.
  • Appendix A payload format details - Interface version request packet size/contents
    I think an empty packet with correct payload type etc is fine, discuss.
  • Appendix A payload format details - Firmware version request packet size/contents
    I think an empty packet with correct payload type etc is fine, discuss.
  • Include extensive "basic" vars in part 1 spec for compatibility with simple devices that can't cope with flash protocols? (would require detailed thinking...)
    Undecided, but bit flag for protocol/firmware use of payload type field allows this, discuss.
  • Acknowledgements :
    Any PC -> Device packet can contain an acknowledgement request, the returned packet has a header bit that says whether it is positive or negative. If positive the packet will either be empty, or contain the requested data. If negative the packet will be empty, or contain data that will be read as a X bit number where X is however many bits come through, discuss.
  • Special protocol packet types :
    What is there, errors, special conditions, etc etc, name some please if you can.
  • Packet size (PC -> Device) :
    We need a request for maximum packet size OR we need to write our data down straight away and complain afterwards if it was bad. Or we need to have a fixed max packet size on the protocol, this is bad. Discuss.
  • I'd like to propose a packet type that requests a block of data either with a page value, address and length, or just an address and length, or possibly two, one for each. I don't want memory access for writes, but reads seems useful and harmless.
    Please discuss.
  • The CRC where used should also be escaped incase it ends up being the stop start or escape byte.
    thebigmacd wrote:I don't know the best way to use this so I left it reserved. It likely needs to be a value of 128-255 because we wouldnt want to inadvertently produce a CRC with a !, #, or $ in ascii code.
    Please discuss.
Please Stu, Shameem, Marcos, Jean, Abe, Keith, etc discuss these outstanding things such that we can start to get initial implementations of this protocol out and running asap.

Thanks for your efforts so far all parties!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
shameem
LQFP112 - Up with the play
Posts: 135
Joined: Thu May 01, 2008 6:30 pm
Location: Ann Arbor, MI
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by shameem »

Well i can start things off by showing off my ignorance -

CRC:
The way i remember from ethernet - the termination character comes after CRC frame - http://docs.hp.com/en/98194-90053/ch02s03.html
Whether to have CRC for header - it depends on which is worse i guess - corrupt header or corrupt CRC. Corrupt header accepted due to corrupt CRC is highly unlikely (but statistically possible i guess) - good header rejected due to bad CRC very likely to happen. Adding CRC for header only adds to overhead - if inadvertent writing to firmware is a concern then maybe it could be precluded by the hardware asking for confirmation from the host before writing firmware.

Flow control - the easy way to do is none. In my mind - the host can send all the requests it wants - the controller doesnt have to reply (if there is no flow control). And when the controller finally has time to respond it can either take care of all the pending requests or clear the buffer and tell the host to start over. The host should have preset timeouts to avoid waiting on the controller indefinitely.
csrow
TO92 - Vaguely active
Posts: 2
Joined: Mon Jul 07, 2008 9:58 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by csrow »

I'm sorry I haven't had the time to read all 16 pages of previous posting here but I would like to make couple of comments.

Most communication protocol are done in layers. That is in both physical terms and in control terms. By having separate layers, you can change without affecting other layers.

1) Separate RS-232 to a physical connection layer definition with link control definition. That way you can change it to a USB or Bluetooth connection (or even a wifi support later on). Do not use hardware pins for data layer flow control. Not too many ports or operating system support the correct RS-232 data flow using the hardware control signals.

2) Next definition needed would be the one layer up of packet definition. This is where the size, type, error checking, and the flow control would come in. For now you can envision just point to point connection but later on maybe you would like to have one-to-many connection (eg. a PDA and a laptop connected at the same time?)

3) And at the top would be where the meat of the data information exchange definition would come in. A higher level flow control as well as timing control would fall here.

You can make these three layers as simple as possible but they are all required to have a robust communication channel running.

Sorry if the above idea has already been discussed before.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

Hi, welcome along :-)

Thanks for your input! I just thought I'd point out that there is no real need to read the entire thread in this instance. The top post on this page has the link to 0.4 spec and all (known) remaining things to discuss listed. Feel free to be highly critical of anything that seems to have been "decided" on as there is always room for changed based on well founded criticism.

With regards layers, can you explain in some more detail? The single restriction is that on the controller side we will be sending via the standard SCI UART module. Thus anything compatible with that will work regardless of what we do. Layers to me equals overhead which we really want to avoid. Currently I have some rudimentary and temporary functions to send a string by length and pointer, by pointer and null byte, to send each of char, short, long, and to send a block of memory by pointer and length. The implementation behind those could be changed without affecting the rest of the program, but I can't see it happening except possibly for CAN which may require different semantics anyway.

With regards flow control, most good USB-UART converters have hardware support built in : http://www.ftdichip.com/Products/FT232R.htm AFAIK most drivers allow you to set/unset the pin and check the other pin with polling thereby putting the "burden" of the flow control on the user application. This doesn't require OS support or any special support on the hardware accept to be able to turn on and off the pins through the driver.

What I had in mind for "flow control" was for the signal to basically mean "I'm not listening, don't bother *trying* to send" such that the user program doesn't waste it's time putting out commands that will never get answered and having to resend them. If a particular medium of communication doesn't have this, the devices on the other end could just be well behaved and send a packet, wait for some interval before assuming that the packet was lost and sending another. In that wait period it would assume that the ecu had received the packet and was "not listening" and working on responding.

Finally, thank you for taking the time to format, punctuate and spell check (if you needed to) your post. It's getting more rare to see reasonable language use on the internet and I for one appreciate it!

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
johnd
DIP8 - Involved
Posts: 23
Joined: Tue Jun 24, 2008 8:21 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by johnd »

maybe me being thick --or just not on same wavelength
why are you even talking about serial port tuning when nearly every lap top made now has only USB port connection.
I know you don,t need the speed ,but using USB serial leads is never good and another place for comms to fall over .
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

johnd wrote:maybe me being thick --or just not on same wavelength
why are you even talking about serial port tuning when nearly every lap top made now has only USB port connection.
I know you don,t need the speed ,but using USB serial leads is never good and another place for comms to fall over .
John, exactly the same protocol will be used regardless of whether a user implements a physical USB interface or a physical RS232 interface. The underlying data stream is just serial data to be interpreted and treated in a specific way which does not depend on the medium of physical transfer. I hope that clarifies it and doesn't make it more muddy :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
csrow
TO92 - Vaguely active
Posts: 2
Joined: Mon Jul 07, 2008 9:58 pm

Re: Serial protocol pre-design discussion - give us your input!

Post by csrow »

Fred,

I would advise the protocol to not to rely on RS-232 hardware handshake signals for the low level flow control. Most RS-232 ports out there only utilize TX-RX pins. Others are usually hard wired.

As for the layers, think of it as having command level to control the connection while allowing datalog data flow to continue. That would be the most simple two level set up.

1) How to synchronize the communication link. - You would need a way to get the attention of the other side so that both side can be on the same level.

2) Packet sequence numbering is a fairly simple way to self check that both sides are in sync.

3) For command packet, a simple checksum would be fine for checking for errors.

In RS-232 communication over a short distance, the biggest problem would be intermittent cable port connection which will show up as packets of data coming and going.

A single byte start and stop flag is fine but those flags will also show up in data sequences too. That's where a higher level protocol would be needed.

Additional structure will actually help in simplifying the communication coding by having a state flow of the protocol. You will just have to define how many different states are needed and how to transition between the states.
y09223
TO220 - Visibile
Posts: 6
Joined: Sat Oct 25, 2008 2:57 am

Re: Serial protocol pre-design discussion - give us your input!

Post by y09223 »

I've just come across FreeEMS - my thoughts exactly! Even though I have extremely limited time due to a wee one, I'd like to contribute what I can.

I've started by reading through the Serial Protocol 0.5 Draft (11 Nov 2008), and some things really jump out at me. Not having been part of the initial definition, I hope this will be a fresh, and impartial, take on this definition. What follows below is based on the premise that memory and instruction cycles are precious resources in an embedded system, so save them everywhere possible. I apologize in advance for the length of this post.

The message-based, packetized format and checksum are both good. An 8-bit checksum should work nicely.

I am, however, having a little trouble with the header. Though it may not take much code or cycles to encode or decode a variable length header, it has to be done for every packet, so the total expense can be substantial. Potentially significant savings in both code and cycles can be had by simplifying and fixing the header length, and moving certain tasks around. Here's what I'm getting at:

Byte Data
---- ----
0-1 2-byte Header Flags
Bit 15 ASCII/Binary flag (0=ASCII, 1=Binary)
Bit 14 ACK/Respond Req'd flag (0=Resp NOT req'd, 1=Resp req'd)
Bit 13 ACK (1=Message GOOD, 0=Message BAD or ACK not req'd)
Bit 12 NAK (1=Message BAD, 0=Message GOOD or ACK not req'd)
Bit 11-0 Reserved for future use
2-3 2-byte Payload ID
Bit 15 Command/Response (0=Command, 1=Response)
Bit 14-11 Reserved for future use
Bit 9-0 Payload ID
4-7 4-byte Payload Length (32-bit binary)

As presently defined in the v0.5 spec, byte 1 bits 5-7 are "firmware implementation specific." It's generally a bad idea to allow definitions to float like this; if it's not defined, reserve it until it's defined.

As best I can tell, the need and use of Ack/Seq Number byte is undefined in the spec. Also, in reading through the forum, I could find no obvious argument in favor of it.

Payload:
Each packet should contain exactly one payload. Given a payload length of 2^32, there should be no need for multi-packet sequences. By the same logic, a string payload should contain one item; if multiple character blocks are to be sent, each block should be in it's own packet. Besides, null-terminated blocks take time to encode and decode.

Payload Type:
It is unclear to me what the difference between a "protocol" and "firmware" payload type is. Why have two separate sets of payload definitions?

Addressing Behaviour:
Addressing does not seem to be necessary at the message packet level. The simplest communication scheme, all communication will be between FreeEMS and a host computer via RS232 for development, tuning, etc. The next level of complexity would be to add an RS232-USB chip/converter (see http://www.ftdichip.com). The converter takes care of all the USB ugliness, yet still appears as a simple RS232 port. This is all point-to point, so there is no need for an addressing scheme. If a CAN or LIN bus is to be added to the EMS, CAN and LIN addressing can be easily implemented by defining one of the Payload ID reserved bits to indicated a CAN or LIN bus, with the Payload ID containing the address. The EMS would then simply pass the payload to the bus/address specified, minimizing parsing and processing overhead.

UART specific items:
STX and ETX characters can appear in a binary data packet. Even with an escaping scheme, the escape sequence can appear in the binary data packet. This is a 30-40 year old problem, with a lot of good solutions out there. But, ALL solutions require additional code (memory) and processing (cycles). The most compact, efficient, and robust solution is to make all transfers binary, and to use hardware handshaking and flow control. The penalty with this approach is that the full communication interface requires 7 wires. Wires (aka I/O ports) can be saved by using I2C or SPI to USB or RS232 converters, but that shouldn't change the messaging protocol definition. Should it?

John
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial protocol pre-design discussion - give us your input!

Post by Fred »

Thanks for your input, there is some good stuff there, John.

Some of those things are issues.
Some of them require a re-looking at.
Some of them are under control.

I'll have to respond in detail later as I have to go to work now. Just acknowledging your message for the moment :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
y09223
TO220 - Visibile
Posts: 6
Joined: Sat Oct 25, 2008 2:57 am

Re: Serial protocol pre-design discussion - give us your input!

Post by y09223 »

I'm not sure if this is the right place for this, but I saw a comment earlier in this thread about datalogging. The earlier comment was to use either USB or SD storage.

There are three bits of code that need to be written to implement a data logging solution: the data storage format, the file structure, and the mass storage stack. If the data message format is robust enough, it can do double duty as the data logging format, minimizing the amount of coding necessary. Having some form of file structure is mandatory for easy access to the data. The best value (return vs. code/cycles) is FAT16.

The USB protocol and USB mass storage class are quite large pieces of code, and require quite a bit of processing - processing that could better be used on 'carputations'. SD cards have two modes: 4-bit parallel and serial. The serial mode is of particular interest - it's basically SPI. The code for this is quite small, and gives you a LOT of bang for the buck. If I remember correctly, the Freescale processor for this project has both SPI and DMA, so dumping data to an SD card shouldn't involve too much overhead.

To read the SD card on a PC, $2 (no kidding!) at MicroCenter buys you an SD to USB reader/writer.
Post Reply