Good idea. Thanks for posting it up! :-)thebigmacd wrote:I got the "bank" idea from OBDII. This could be handy for V8s without a crossover pipe. Even if banks arent supported for a long time, we could just always use bank0 for everything until multiple banks are supported.
Serial protocol pre-design discussion - give us your input!
Re: Serial protocol pre-design discussion - give us your input!
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
-
- LQFP112 - Up with the play
- Posts: 205
- Joined: Thu Apr 10, 2008 5:51 pm
Re: Serial protocol pre-design discussion - give us your input!
See here for standard ANSI/VT100 terminal escape sequences
http://www.termsys.demon.co.uk/vtansi.htm
It has simple support for device type reporting etc with standard byte sequences. There are reserved ASCII bytes for data demarcation as well
http://www.asciitable.com/
The codes we are interested in include:
Whether these specific codes are used or not isn't critical, but it's a good example of the type of data handling codes required for a structured datastream.
http://www.termsys.demon.co.uk/vtansi.htm
It has simple support for device type reporting etc with standard byte sequences. There are reserved ASCII bytes for data demarcation as well
http://www.asciitable.com/
The codes we are interested in include:
Code: Select all
Binary Oct Dec Hex Abbr PR[a] CS[b] CEC[c] Description
-----------------------------------------------------
000 0000 000 0 00 NUL †^@ \0 Null character
000 0001 001 1 01 SOH â ^A Start of Header
000 0010 002 2 02 STX â‚ ^B Start of Text
000 0011 003 3 03 ETX ⃠^C End of Text
000 0100 004 4 04 EOT â„ ^D End of Transmission
000 0101 005 5 05 ENQ â… ^E Enquiry
000 0110 006 6 06 ACK ↠^F Acknowledgment
...
001 0000 020 16 10 DLE â ^P Data Link Escape
001 0001 021 17 11 DC1 â‘ ^Q Device Control 1 (oft. XON)
001 0010 022 18 12 DC2 â’ ^R Device Control 2
001 0011 023 19 13 DC3 â“ ^S Device Control 3 (oft. XOFF)
001 0100 024 20 14 DC4 â” ^T Device Control 4
001 0101 025 21 15 NAK â• ^U Negative Acknowledgement
...
001 0111 027 23 17 ETB â— ^W End of Trans. Block
...
001 1001 031 25 19 EM â™ ^Y End of Medium
...
001 1011 033 27 1B ESC â› ^[ \e[f] Escape[g]
001 1100 034 28 1C FS ✠^\ File Separator
001 1101 035 29 1D GS â ^] Group Separator
001 1110 036 30 1E RS âž ^^ Record Separator
001 1111 037 31 1F US ⟠^_ Unit Separator
Keith MacDonald
Control Engineering (Systems) Technologist
Control Engineering (Systems) Technologist
-
- LQFP112 - Up with the play
- Posts: 205
- Joined: Thu Apr 10, 2008 5:51 pm
Re: Serial protocol pre-design discussion - give us your input!
So I thought about it some more today. This might be a long post 
I have developed a protocol framework that you can adapt if you like. The basic ideas behind my framework are:
Friendly; the structure must be human-readable on a terminal.
Flexible; data that is unchanged since last packet/startup is not required to be repeated.
Configurable; the protocol must support subscription for datalogs.
Integrity; every packet must have a simple crc.
I broke the packets down into three types: synchronize, command, and data.
Rules:
(1) The packet begins with a character identifying the type of packet. This character will be !, #, $, or ?.
(2) The data fields in the packet are separated by the [tab] ascii character
(3) Two [tab] characters in a row means the field has not changed since last update. Move to next field
(3) The last field in all packets is a CRC
(4) Any character after the CRC is ignored until a new packet begins
The "Synchronize" Packet ($xx ...)
From PC: this packet is used to set the time in the ECU, or start the datalog count from 0
From ECU: this packet is used to separate datalog frames
d (byte) = device id (0 = default = ecm, 1 = tcm, 2 = abs, etc)
b (byte) = bank id (0 = default = average)
id (bcd) = frame id
dd-mm (text) = day and month
yyyy (text) = year
xxxx (4 bytes) = crc 1 byte per field, $00 counts as a field
The "Data" Packet (#xx ...)
From PC: this sets values in the ECU
From ECU: this is part of a datalog stream or a response to a command or query
d (byte) = device id (0 = default = ecm, 1 = tcm, 2 = abs, etc)
b (byte) = bank id (0 = default = average)
name (text) = name of the variable
value (bcd or text) = value of the named variable
unit (text) = units of the named variable
xxxx (4 bytes) = crc 1 byte per field, #00 counts as a field
The "Command" Packet (!xx ...)
From PC: used to determine if a value exists in the ECU, to modify the datalog subscription, and/or to change the unit the value is sent in
From ECU: used to echo the subscription status or units of a value
d (byte) = device id (0 = default = ecm, 1 = tcm, 2 = abs, etc)
b (byte) = bank id (0 = default = average)
name (text) = name of the variable
log? (bool or text?) = set whether value is part of datalog
unit (text) = set the unit to send the value in
xxxx (4 bytes) = crc 1 byte per field, #00 counts as a field
Useage
Sample data stream:
The nice thing is you could dump this raw serial stream to a file and it's already tab-separated. A feature of ignoring any data after the crc until the next packet character is that you could throw a CR/LF into the stream after every packet and it would display nicely in hyperterminal! It would look exactly like my code snippet. Might have to play around with the tab lengths in the terminal when the numbers get long, but it would be pretty slick.
More to come!

I have developed a protocol framework that you can adapt if you like. The basic ideas behind my framework are:
Friendly; the structure must be human-readable on a terminal.
Flexible; data that is unchanged since last packet/startup is not required to be repeated.
Configurable; the protocol must support subscription for datalogs.
Integrity; every packet must have a simple crc.
I broke the packets down into three types: synchronize, command, and data.
Rules:
(1) The packet begins with a character identifying the type of packet. This character will be !, #, $, or ?.
(2) The data fields in the packet are separated by the [tab] ascii character
(3) Two [tab] characters in a row means the field has not changed since last update. Move to next field
(3) The last field in all packets is a CRC
(4) Any character after the CRC is ignored until a new packet begins
The "Synchronize" Packet ($xx ...)
From PC: this packet is used to set the time in the ECU, or start the datalog count from 0
From ECU: this packet is used to separate datalog frames
Code: Select all
$00 1 14-MAY 2008 xxxx
^^^ ^ ^ ^ ^
$db id dd-mmm yyyy crc
b (byte) = bank id (0 = default = average)
id (bcd) = frame id
dd-mm (text) = day and month
yyyy (text) = year
xxxx (4 bytes) = crc 1 byte per field, $00 counts as a field
The "Data" Packet (#xx ...)
From PC: this sets values in the ECU
From ECU: this is part of a datalog stream or a response to a command or query
Code: Select all
#00 AI.TPS 25 % xxxx
^^^ ^ ^ ^ ^
#db name value unit crc
b (byte) = bank id (0 = default = average)
name (text) = name of the variable
value (bcd or text) = value of the named variable
unit (text) = units of the named variable
xxxx (4 bytes) = crc 1 byte per field, #00 counts as a field
The "Command" Packet (!xx ...)
From PC: used to determine if a value exists in the ECU, to modify the datalog subscription, and/or to change the unit the value is sent in
From ECU: used to echo the subscription status or units of a value
Code: Select all
!00 AI.TPS NO % xxxx
^^^ ^ ^ ^ ^
!db name log? unit crc
b (byte) = bank id (0 = default = average)
name (text) = name of the variable
log? (bool or text?) = set whether value is part of datalog
unit (text) = set the unit to send the value in
xxxx (4 bytes) = crc 1 byte per field, #00 counts as a field
Useage
Sample data stream:
Code: Select all
$00 0 14-MAY 2008 xxxx : initial synch, values from ECU, bank 0
# ai.tps 0 % xxxx : throttle position in unit %
# ai.map 32 kPa xxxx : manifold air pressure in unit KPa
# ai.iat 68 F xxxx : intake air temp in unit F
# ai.clt 195 F xxxx : coolant temp in unit F
# av.rpm 900 RPM xxxx : crank speed in RPM
# av.adv 12 deg xxxx : advance in degrees
# av.aft 14.7 :1 xxxx : air/fuel target in proportion
$01 xxxx : same frame, switch to ECU, bank 1
# av.ve 45 xxxx : volumetric efficiency
# av.pw0 2.5 ms xxxx : injector pulse width in unit ms (without opening time)
# av.afr 14.7 :1 xxxx : air/fuel ratio in proportion
$02 xxxx : same frame, switch to ECU, bank 2
# av.ve 45 xxxx : volumetric efficiency
# av.pw0 2.5 ms xxxx : injector pulse width in unit ms (without opening time)
# av.afr 14.7 :1 xxxx : air/fuel ratio in proportion
$10 xxxx : same frame, switch to TCM, bank 0
# bo.tcc off xxxx : torque convertor clutch solenoid in default unit (on|off)
$00 1 xxxx : next frame, values from ECU, bank 0
# ai.tps 0 xxxx : throttle position
# ai.map 35 xxxx : manifold air pressure
# ai.iat 68 xxxx : intake air temp
# ai.clt 195 xxxx : coolant temp
# av.rpm 904 xxxx : crank speed
# av.adv 12 xxxx : advance
# av.aft 14.7 xxxx : air/fuel target
$01 xxxx : same frame, switch to ECU, bank 1
# av.ve 45 xxxx : volumetric efficiency
# av.pw0 2.5 xxxx : injector pulse width (without opening time)
# av.afr 14.5 xxxx : air/fuel ratio
$02 xxxx : same frame, switch to ECU, bank 2
# av.ve 45 xxxx : volumetric efficiency
# av.pw0 2.5 xxxx : injector pulse width (without opening time)
# av.afr 14.8 xxxx : air/fuel ratio
$10 xxxx : same frame, switch to TCM, bank 0
# bo.tcc off xxxx : torque convertor clutch solenoid
$00 2 xxxx : next frame, etc...
# ai.tps 0 xxxx
# ai.map 36 xxxx
# ai.iat 68 xxxx
# ai.clt 195 xxxx
# av.rpm 892 xxxx
# av.adv 12 xxxx
# av.aft 14.7 xxxx
$01 xxxx
# av.ve 45 xxxx
# av.pw0 2.5 xxxx
# av.afr 14.7 xxxx
$02 xxxx
# av.ve 45 xxxx
# av.pw0 2.5 xxxx
# av.afr 14.7 xxxx
$10 xxxx
# bo.tcc off xxxx
$00 3 xxxx
# ...
# ...
More to come!
Keith MacDonald
Control Engineering (Systems) Technologist
Control Engineering (Systems) Technologist
-
- LQFP112 - Up with the play
- Posts: 205
- Joined: Thu Apr 10, 2008 5:51 pm
Re: Serial protocol pre-design discussion - give us your input!
CRC stuff
The reason I reserved 4 bytes in the CRC field is I like having a separate CRC for each field in the datastream
I don't know the best way to use this so I left it reserved. It likely needs to be a value of 128-255 because we wouldnt want to inadvertently produce a CRC with a !, #, or $ in ascii code.
How to ask an ECU if a value exists
This is a lot of unnecessary steps, but it is intended to illustrate the functionality of the protocol
Naming Stuff
This is from BACnet:
av = analog variable
bv = binary variable
ai = analog input
bi = binary input
ai.tps = analog input - throttle position sensor
bi.armnos = binary input - nitrous system armed
bv.warmup = binary variable - warmup mode
av.ve = analog variable - volumetric efficiency
This is where standardization takes a lot of work
The reason I reserved 4 bytes in the CRC field is I like having a separate CRC for each field in the datastream
I don't know the best way to use this so I left it reserved. It likely needs to be a value of 128-255 because we wouldnt want to inadvertently produce a CRC with a !, #, or $ in ascii code.
How to ask an ECU if a value exists
This is a lot of unnecessary steps, but it is intended to illustrate the functionality of the protocol
Code: Select all
PC>
!00 BV.STRM NO xxxx : pc tells ecm to stop sending datastream (this is used if datastream is sent from same UART)
!00 AI.IAT K xxxx : pc asks for Intake Air Temp in degrees Kelvin, ignore subscription status
ECM>
$00 -1 xxxx : ecm responds with a synch packet with a negative number, means response, do not include in datalog
# AI.IAT xxxx : ecm responds with a blank AI.IAT packet, after a synch this means the value does not exist
PC>
!00 AI.MAT K xxxx : pc asks for Manifold Air Temp instead, in degrees Kelvin, ignore subscription status
ECM>
$00 -1 xxxx : ecm responds with a synch packet with a negative number, means response, do not include in datalog
# AI.MAT K xxxx : ecm responds with a blank AI.MAT packet in degrees K, after a synch this means the value exists but it does not recognize Kelvin
PC>
!00 AI.MAT xxxx : pc asks for AI.MAT, no units specified, this should return the value in whatever units the ecm is set to
ECM>
$00 -1 xxxx : ecm responds with a synch packet with a negative number, means response, do not include in datalog
#00 AI.MAT 68 F xxxx : ecm responds with AI.MAT in degrees F
PC>
!00 AI.MAT YES F xxxx : pc tells ecm to subscribe AI.MAT in datastream, in degrees F
ECM>
$00 -1 xxxx : ecm responds with a synch packet with a negative number, means response, do not include in datalog
#00 AI.MAT 68 F xxxx : ecm responds with AI.MAT in degrees F. this is to confirm the value will be included
PC>
!00 BV.STRM YES xxxx : pc tells ecm to restart datastream
This is from BACnet:
av = analog variable
bv = binary variable
ai = analog input
bi = binary input
ai.tps = analog input - throttle position sensor
bi.armnos = binary input - nitrous system armed
bv.warmup = binary variable - warmup mode
av.ve = analog variable - volumetric efficiency
This is where standardization takes a lot of work
Keith MacDonald
Control Engineering (Systems) Technologist
Control Engineering (Systems) Technologist
-
- QFP80 - Contributor
- Posts: 41
- Joined: Tue Apr 29, 2008 1:47 pm
- Location: Hampshire, UK
- Contact:
Re: Serial protocol pre-design discussion - give us your input!
A question that might be worth investigating: Are there any pros or cons to having a human readable protocol for this project?
The ECU is going to be kicking out a fair bit of data in real time, and we may want that to be at a very high resolution. Plain text, is going to push the overheads up considerably. String handling on a processor can be very taxing, not to mention the extra volume of data that will have to be sent.
The ECU is going to be kicking out a fair bit of data in real time, and we may want that to be at a very high resolution. Plain text, is going to push the overheads up considerably. String handling on a processor can be very taxing, not to mention the extra volume of data that will have to be sent.
-
- LQFP112 - Up with the play
- Posts: 205
- Joined: Thu Apr 10, 2008 5:51 pm
Re: Serial protocol pre-design discussion - give us your input!
I doesn't have to be human readable as long as the variable identifiers can be standardized. There are five things that make my framework human-readable: using a display character as a frame/packet header, using a tab character as a field delimiter, using a string as a variable ID, using BCD for the numeric values, and throwing line breaks in between packets.FlappySocks wrote:A question that might be worth investigating: Are there any pros or cons to having a human readable protocol for this project?
The ECU is going to be kicking out a fair bit of data in real time, and we may want that to be at a very high resolution. Plain text, is going to push the overheads up considerably. String handling on a processor can be very taxing, not to mention the extra volume of data that will have to be sent.
The packet header can't really be eliminated, so why not use an ASCII character while we are at it.
The field delimiter can be eliminated at the expense of having fixed length var_ids, values, and units. If variable, why not use a tab for a delimiter so it is somewhat decipherable.
The variable IDs are tricky...how does one add more info without breaking compatibility with code forks etc, without using strings. If numbers are used there will have to be a very good centralized table available online.
As far as the actual value encoding, if you use BCD combined with a units field you can eliminate an entire section of protocol needed to define how to transfer definitions of word length, min, and max values etc. that are required when binary values are used.
As far as line breaks, no you don't need to send them.
If any characteristic of my idea were to be retained, I would suggest it be the feature of skipping fields that haven't been changed. In certain circumstances this could condense this:
Code: Select all
#00_AI.RPM_10000_RPM_xxxx
Code: Select all
#_AI.RPM___xxxx
Carry on

Keith MacDonald
Control Engineering (Systems) Technologist
Control Engineering (Systems) Technologist
Re: Serial protocol pre-design discussion - give us your input!
Firstly, thanks for posting your ideas up! I appreciate it and the more material for discussion the more well rounded our solution will be in the end.
Having said that, I agree with what Flappy said.
The overhead from tab delimiting is like 50 - 100% increase in size.
Converting everything to BCD on the fly is a factor of 5? 10? a lot more than just sending the value.
I'm a big fan of the EMS having what it has and knowing what it is, and the PC software also knowing what is there. MT achieves this with massive INI files. MTX achieves this in code with easy to add defs and modules (for a dev). I think it should be assumed that if you can fork/branch/heavily mod the firmware that patching the cross platform open source pc interface to match should not be a big deal. I think they should come as matched pairs either in config+firmware or code+firmware.
We just won't be able to achieve good speed doing it that way IMO. Once the thing is up and running, I really don't think there is much to gain by having it readable. To retrieve a value from 500 options manually would be a pain even if you could read it. Additionally I think there is a lot to lose also in the way of speed and potentially robustness.
Additionally, the contents of the code should not deal in unit swapping, they should either be in SI or arbitrary internal units with a known conversion in the outside world.
However :
I like the idea of using standard ascii chars for our special chars provided that it can be shown that my escape/start/stop bits have no advantage over any other random char used.
Some symbol as start and a newline for for end (unix single char, not windoze both chars(which made sense on a type writer, but not in a pc :-p )) and some other sensible ones for other purposes could make a quick scan of what is coming through recognisable at all. Whether there is value in this is dubious though at the end of the day IMO.
Keep up the discussions :-)
Flappy, I'm still keen to hear your comments on my initial comments if you have time.
Admin.
Having said that, I agree with what Flappy said.
The overhead from tab delimiting is like 50 - 100% increase in size.
Converting everything to BCD on the fly is a factor of 5? 10? a lot more than just sending the value.
I'm a big fan of the EMS having what it has and knowing what it is, and the PC software also knowing what is there. MT achieves this with massive INI files. MTX achieves this in code with easy to add defs and modules (for a dev). I think it should be assumed that if you can fork/branch/heavily mod the firmware that patching the cross platform open source pc interface to match should not be a big deal. I think they should come as matched pairs either in config+firmware or code+firmware.
We just won't be able to achieve good speed doing it that way IMO. Once the thing is up and running, I really don't think there is much to gain by having it readable. To retrieve a value from 500 options manually would be a pain even if you could read it. Additionally I think there is a lot to lose also in the way of speed and potentially robustness.
Additionally, the contents of the code should not deal in unit swapping, they should either be in SI or arbitrary internal units with a known conversion in the outside world.
However :
I like the idea of using standard ascii chars for our special chars provided that it can be shown that my escape/start/stop bits have no advantage over any other random char used.
Some symbol as start and a newline for for end (unix single char, not windoze both chars(which made sense on a type writer, but not in a pc :-p )) and some other sensible ones for other purposes could make a quick scan of what is coming through recognisable at all. Whether there is value in this is dubious though at the end of the day IMO.
Keep up the discussions :-)
Flappy, I'm still keen to hear your comments on my initial comments if you have time.
Admin.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
-
- QFP80 - Contributor
- Posts: 41
- Joined: Tue Apr 29, 2008 1:47 pm
- Location: Hampshire, UK
- Contact:
Re: Serial protocol pre-design discussion - give us your input!
Sure. Just wanted to stay quiet for a bit and see what others said.Admin wrote: Flappy, I'm still keen to hear your comments on my initial comments if you have time.
I'd say fixed size. 8 or 16 bit. What are the chances of data corruption? Can anyone theorise?Admin wrote: [*]IMO CRC should follow the payload and be just before the stop byte, the header should contain 1 bit (fixed size) or 2 bits (00 = off, 01 = 8bit, 10 = 16bit, 11 = 32bit). I tend to think fixed size is the way we should go and that multiple sizes is overkill. I also tend to think that doing as mtx_man suggested and using a rolling sum during sending is a good idea. That would work most simply with 8 bits, but 16 bits could use a convention such that if only one byte was left, it was either assumed to be high or low in the 16 to add and padded with zeros or ones above or below that. If we want to put it in the header, that means running through all the data BEFORE sending and calculating it in the hope that it doesn't change while we are sending. If it is a large packet, this will not work out well.
I like the idea of having a command that requests it. Maybe we should have two version numbers; Part 1 and Part 2 of the protocol? Or just a Part 2 version, which also covers any changes in part 1.Admin wrote: [*]Should the protocol carry a version number of it's own, or make it available? Or should we rely on a basic external to the normal protocol scheme of using a particular "command" to request the firmware version number (a byte per part, so max of 255.255.255) and some sort of other unique identifier or similar? Should that be part of level one protocol? I think yes, a version (3 part, or maybe 4 for future proofness) and a name/product identifier string field request too, and possibly one other.
My thinking is, the communication part of the software should be self contained. If I were programming this in say Java or C#, I would wrap it up into a class, and have it run in it's own thread. I could then say to it, Here is a bit of data - send it. Or I could say, Here is a bit of data, send it, but if it's lost, I dont care.Admin wrote: [*]What about responses as opposed to acknowledgments? i.e. If you ask for xyz, do you need to know that it is THE one you asked for, or just any xyz packet? Could the ack number be dual purpose? as in a bit in the header says "initiator" or "responder" and the ack byte or word is either what is being responded with, or the identifier for a request. perhaps if that bit is 1 for a request the request itself could come back bearing the 1 whereas if it is zero and it is a request. Does this protocol need to be symmetrical?? I think not as the EMS will be responding to data or providing it without being asked (in the case of event notification or data logging continuously being turned on) and the PC (or dash cluster or xyz) will be the one asking for X telling it to do Y, or receiving the unasked for data and processing it. Apologies if this wasn't clear...
If the "Comms Class" has been instructed to send some data, and it's been flagged as important, then it must take care of that process in one of two ways (To be decided). 1) buffer all "important" data, and keep resending it until an acknowledgment is received. AND/OR 2) wait a preset time for an acknolegment, and if it's not received, fire an exception event.
My thinking there was that all devices must respond to ping requests, as part of the standard. Its a simple thing to implement, and gives the device the ability to know if anything is connected to it. Maybe we dont have Ping, and just have version number requests.Admin wrote: [*]Ping seems like a good idea. Would you say "processor reset request" etc are an appropriate part of the part 1 protocol discussion or part 2?
Going back to the "Comms Class" thing. A programmer thats in charge of say the coolant sensor, might not care about how the communications is done. All he wants is to get his coolant readings and pass them on. If he feels that his measurements are important enough, then he can just flag it as such, and passing it on to the Comms Class without knowing or caring how that process works.Admin wrote: [*]It's nice that every packet CAN have crc and ack, but none are forced to. I like this.
No magic here. Just made some hex numbers up. AA BB etc are a bit easier to spot in a debug trace than 81 or 12 etc.Admin wrote: [*]Are your escape scheme codes commonly used, or arbitrary? if they are arbitrary, wouldn't CC be a better choice than BB ? maximum clarity of the patterns. What abour 0xFF for XORing for the same reason, ie, notting the value. I suppose it doesn't matter though, or perhaps there is some reason I haven't seen yet.
Real time data = live stream of rapidly changing data. Could be called a logging packet.Admin wrote: [*]You say that not all devices support all data etc, and in principle I agree, but in practice, the one that all devices would need is likely to end up more complex to handle than it used to be, and more powerful for it. IE a logging packet (is this what is meant by "real time data" ?) should use a mask that defines what each bit is based on the total sequence of bits etc so that it can be user trimmed to include and exclude various parameters. A detailed knowledge of what the data is will be required on any device wishing to use that data because of it's variable form. Using a fixed form would simplify it, but limit it in the future and limit flexibility with regards logging speed/verbosity. Just speaking out loud here, shoot holes in it as you wish![]()
If I made an LCD gauge that showed just RPM, I would need to implement Part 1, and extract only the parts of Part 2 that related to RPM. All other Part 2 data will be dropped.
Re: Serial protocol pre-design discussion - give us your input!
Firstly, I had a couple of personal responses about this thread :
1. "Way over my head" : There may be a lot of writing, but it's not that complex really, and if something is too complex, say so. One aim would be for this to be easy to understand in code. It seems fairly straight forward to me so far, but share your thoughts, or, don't be shy, ask questions. Even just asking questions can sometimes expose flaws in something. If it doesn't make sense to you, perhaps it doesn't make sense at all :-)
2. "I'm not interested" : This is pretty disappointing to be honest. IMO this is one of the critical foundations that we are building upon. It needs to be RIGHT for a number of reasons. Getting it wrong could mean unpleasant interfaces (code and UI), poor performance, corruption problems, etc etc. We really need to get this right, and the sooner the better.
3. "I've not had any problems with serial comms" : Me neither, BUT, many many have for many many reasons. When I click "burn" I want to KNOW that the burn has stuck without re fetching the data manually. This could be accomplished with an automated burn and retrieve function that sends the data, then asks for it back and compares, or it could be done with an "ack" and checksum that lights a green "ok" indicator up when good, or red "burn failed" if not. Next to the "ok" could be a "verify" button to make 110% certain. Or even a setting in the software that does both on every burn. (I'd use that!)
Would you prefer Flappy, Stuart, or Stu? (or any other nick name? I'm a fan of nick names :-) )
I checked out the ASCII for those double letter hex codes and none are standard codes but none are common chars or control chars. At the opposite end, if we use FF to XOR them (keeps it a double banger after XORing) 0xAA gives a 0x55 which is 'U', BB gives 0x44 which is 'D', 0xCC gives 0x33 which is '3', 0xDD gives 0x22 which is '"' and the rest give control chars. For this reason, I vote 0xAA for start, 0xBB for escape and 0xCC for stop with XOR via 0xFF (to produce 0x55 0x44 and 0x33). Unless we forget being able to see them in a stream and go with Keith's plan of using some visible start char, and a newline for end?
If we did want to have a console mode for simple quick by hand testing I think it would be best to do the swap n switch system and have dual protocols. If we do that, the "query" needs to be compatible with both such that you can tell first up which you are dealing with and/or the version of it etc. Some aspects of Keith's "readable" protocol might be a good idea here if we go ahead and do 2. I personally don't like the idea of having two sets of code to do one job though. But, I do like the idea of a simple access mode for testing and/or basic devices that want to be dumb. Perhaps the simple mode could give access to only the core values and none of the extra/intermediate stuff in some fixed format?
Could you provide C code for stretchy integers / "BER compressed integers" so we can either put mtx_man's mind at rest or avoid using them. (trying to get some decisions made and work towards an end).
Could you also comment on Karsten's ptp idea?
As far as datalogging goes, I would prefer that all messages were recorded. That way if you have a log and you were tuning while taking it you can see where your changes went in and which behaviour matches with which tune. This could be done in raw form, or translated into a special line in the log indicated by a flag field being on or off that means the rest of the values signify something else.
Fixed 8 or 16 CRC seems good, someone will know how to calc the chances of failure for each, but I'd say combined with parity bits (which I know how to use on our core now) It should be fairly robust with even the most simple system.
With regards ack vs response and encapsulation I understand encapsulation, I'm just not sure what you mean. It sounds like you send a packet and the exception thing occurs if it doesn't get to the ECU AND/OR? get processed by the ECU, perhaps it needs two stages of response :
example A :
PC sends request for VE table
ECU receives request
ECU returns ack ("I got your request") with unique ID on it.
> ECU returns "operation successful" response with VE data with same unique ID on it
> ECU returns nothing (timeout occurs at sender if they care)
example B :
PC sends burn command for value already cached in memory and being used
ECU receives request
ECU returns ack ("I got your request") with unique ID on it.
ECU performs burn
> if successful ECU returns "operation successful" response with same unique ID on it.
> if burn failed ECU returns "operation failed" response with same unique ID on it.
Or would it be better to have only the response with a success/failed bit and contents or no contents. I guess the time frame should be short enough even at high load to not warrant a separate ack to response. Ack can be part of a response or stand alone. Is that what you intended?
My point wasn't about handling the contents, rather about providing the facility for ID for response and ID for ack and enough of them such that we don't ever get roll back to start.
What about flow control. Can this be done correctly in software? Or does it have to use up two more gpio pins? What about the physical side of that, do most/all USB/rs232 adapters handle hardware flow control if we go that way? It seems to me that to KISS in the ECU we need to handle requests serially.
ie :
receive request
block further messages
send ack
send response
unblock further messages
or similar. If we do this, 8 bits for a message number will be fine because 234 will never come until 233 is serviced (success or fail). A message number could be reserved for autonomous data such as logging packets that keep flowing constantly when enabled. say 0 = autonomous, cycling through 1 - 255 for responses. In that case, logging packets could be of two types 0 ID or response ID with the same exact contents. The response ID is something that should surely exist as part of part 1.
I'm just looking to clarify it all and make the documents totally explicit to the finest detail such that there is no ambiguity of language used etc.
Excuse the long post. Lets start to get these ideas into that document (I don't mind doing it if you don't have time, but don't want to step on your toes by editing the doc against your wishes...) and reducing the number of things up for contention.
Admin.
1. "Way over my head" : There may be a lot of writing, but it's not that complex really, and if something is too complex, say so. One aim would be for this to be easy to understand in code. It seems fairly straight forward to me so far, but share your thoughts, or, don't be shy, ask questions. Even just asking questions can sometimes expose flaws in something. If it doesn't make sense to you, perhaps it doesn't make sense at all :-)
2. "I'm not interested" : This is pretty disappointing to be honest. IMO this is one of the critical foundations that we are building upon. It needs to be RIGHT for a number of reasons. Getting it wrong could mean unpleasant interfaces (code and UI), poor performance, corruption problems, etc etc. We really need to get this right, and the sooner the better.
3. "I've not had any problems with serial comms" : Me neither, BUT, many many have for many many reasons. When I click "burn" I want to KNOW that the burn has stuck without re fetching the data manually. This could be accomplished with an automated burn and retrieve function that sends the data, then asks for it back and compares, or it could be done with an "ack" and checksum that lights a green "ok" indicator up when good, or red "burn failed" if not. Next to the "ok" could be a "verify" button to make 110% certain. Or even a setting in the software that does both on every burn. (I'd use that!)
Would you prefer Flappy, Stuart, or Stu? (or any other nick name? I'm a fan of nick names :-) )
I checked out the ASCII for those double letter hex codes and none are standard codes but none are common chars or control chars. At the opposite end, if we use FF to XOR them (keeps it a double banger after XORing) 0xAA gives a 0x55 which is 'U', BB gives 0x44 which is 'D', 0xCC gives 0x33 which is '3', 0xDD gives 0x22 which is '"' and the rest give control chars. For this reason, I vote 0xAA for start, 0xBB for escape and 0xCC for stop with XOR via 0xFF (to produce 0x55 0x44 and 0x33). Unless we forget being able to see them in a stream and go with Keith's plan of using some visible start char, and a newline for end?
If we did want to have a console mode for simple quick by hand testing I think it would be best to do the swap n switch system and have dual protocols. If we do that, the "query" needs to be compatible with both such that you can tell first up which you are dealing with and/or the version of it etc. Some aspects of Keith's "readable" protocol might be a good idea here if we go ahead and do 2. I personally don't like the idea of having two sets of code to do one job though. But, I do like the idea of a simple access mode for testing and/or basic devices that want to be dumb. Perhaps the simple mode could give access to only the core values and none of the extra/intermediate stuff in some fixed format?
Could you provide C code for stretchy integers / "BER compressed integers" so we can either put mtx_man's mind at rest or avoid using them. (trying to get some decisions made and work towards an end).
Could you also comment on Karsten's ptp idea?
As far as datalogging goes, I would prefer that all messages were recorded. That way if you have a log and you were tuning while taking it you can see where your changes went in and which behaviour matches with which tune. This could be done in raw form, or translated into a special line in the log indicated by a flag field being on or off that means the rest of the values signify something else.
Sure, but then there are three (or more) ways to go about this. 1. know about datalog packets and the mask used to make them variable in nature and the version of the code/protocol 2. know the command sequence to get just that value out by itself. 3. extract it from a log packet of standardised simple data in the secondary protocol or similar. 4 ? Only 3 seems to be free of version specific stuff and complexish knowledge. Then again, if you are driving gauges through CAN or similar, your controller is quite smart anyway, and throwing in a new version or config shouldn't be a big deal anyway.Real time data = live stream of rapidly changing data. Could be called a logging packet.
If I made an LCD gauge that showed just RPM, I would need to implement Part 1, and extract only the parts of Part 2 that related to RPM. All other Part 2 data will be dropped.
This is a good idea, "are you there?, (and if so, what are you?)". The firmware number could be checked against known versions in a table from config/memory and if it's not known, just not work. I'm ok with 24 bits for version numbering on the code itself. do we need a 4th byte for anything else or not? So can we flesh this command out now? give it some structure etc...Maybe we dont have Ping, and just have version number requests.
Fixed 8 or 16 CRC seems good, someone will know how to calc the chances of failure for each, but I'd say combined with parity bits (which I know how to use on our core now) It should be fairly robust with even the most simple system.
With regards ack vs response and encapsulation I understand encapsulation, I'm just not sure what you mean. It sounds like you send a packet and the exception thing occurs if it doesn't get to the ECU AND/OR? get processed by the ECU, perhaps it needs two stages of response :
example A :
PC sends request for VE table
ECU receives request
ECU returns ack ("I got your request") with unique ID on it.
> ECU returns "operation successful" response with VE data with same unique ID on it
> ECU returns nothing (timeout occurs at sender if they care)
example B :
PC sends burn command for value already cached in memory and being used
ECU receives request
ECU returns ack ("I got your request") with unique ID on it.
ECU performs burn
> if successful ECU returns "operation successful" response with same unique ID on it.
> if burn failed ECU returns "operation failed" response with same unique ID on it.
Or would it be better to have only the response with a success/failed bit and contents or no contents. I guess the time frame should be short enough even at high load to not warrant a separate ack to response. Ack can be part of a response or stand alone. Is that what you intended?
My point wasn't about handling the contents, rather about providing the facility for ID for response and ID for ack and enough of them such that we don't ever get roll back to start.
What about flow control. Can this be done correctly in software? Or does it have to use up two more gpio pins? What about the physical side of that, do most/all USB/rs232 adapters handle hardware flow control if we go that way? It seems to me that to KISS in the ECU we need to handle requests serially.
ie :
receive request
block further messages
send ack
send response
unblock further messages
or similar. If we do this, 8 bits for a message number will be fine because 234 will never come until 233 is serviced (success or fail). A message number could be reserved for autonomous data such as logging packets that keep flowing constantly when enabled. say 0 = autonomous, cycling through 1 - 255 for responses. In that case, logging packets could be of two types 0 ID or response ID with the same exact contents. The response ID is something that should surely exist as part of part 1.
I'm just looking to clarify it all and make the documents totally explicit to the finest detail such that there is no ambiguity of language used etc.
Excuse the long post. Lets start to get these ideas into that document (I don't mind doing it if you don't have time, but don't want to step on your toes by editing the doc against your wishes...) and reducing the number of things up for contention.
Admin.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Re: Serial protocol pre-design discussion - give us your input!
Here is my response:
4. "Too much to read". Also a proposed solution to stay constructive: Organize the options, arguments, opinions, etc in a dedicated place (a simple structured doc or a wiki) and keep forum thread(s) separately from this doc. To make one read through the whole discussion to understand where we stand is asking too much, which may be what discourages people.
Alex
4. "Too much to read". Also a proposed solution to stay constructive: Organize the options, arguments, opinions, etc in a dedicated place (a simple structured doc or a wiki) and keep forum thread(s) separately from this doc. To make one read through the whole discussion to understand where we stand is asking too much, which may be what discourages people.
Alex
Legal disclaimer for all my posts: I'm not responsible for anything, you are responsible for everything. This is an open and free world with no strings attached.