Serial Communication Take 2

Official FreeEMS vanilla firmware development, the heart and soul of the system!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial Communication Take 2

Post by Fred »

Firstly, excuse my typing, it could be a little broken as I tried to remove my left index finger in the kitchen earlier... it could be worse, last time I was trying to work on this project and cook I nearly burned the place down...
EdG wrote:in the case of just having runtime information out of the ECU, you're going to want it with as little overhead as possible (small 'packets')
I disagree. There is just no way that you are going to want one variable at a time, and even if you did, you would need a pause in between to know which byte was which. If the bytes are one after the next, where is the type and where is the value? Hence the packetising in the first place that Stu was nice enough to draw up for us some months back. Just for example, MS uses about 112 bytes for the basic stuff. I just added up the current basic variables and they come to 27 * 4 bytes total. Regardless, at 100 bytes, 14 bytes of start, stop, header, checksum, escape is not really that significant, esp when it guarantees you accurate data. Then you'd also want some sort of time stamp etc in there too. So call it 100 for now for the purpose of conversation. Past that, same goes for debug, you are going to want say "all the runtime vars" or "all the counters" or "all of the latencies" etc. At the 100 mark you are going to want data integrity for the packet, and the cost of it is low enough IMO. In fact, the larger the packet, the lower the cost of it at the end of the day, so sending in little bits is probably a bad idea and will increase overheads a bunch.
For now, it appears we want the serial comms to:
Download flash block data for programming.
Code debugging of variables.
Runtime system information (for display/datalog/tuning)
The last two are effectively the same.

We want to :
Send data to memory while tuning
Tell it to burn memory to flash when tuned
Send data straight to flash for config and sensor calibration
Get logging/realtime/debug/etc live data out
Get previous config and tune out
Tell it to reset

And a million other things :-)
I'd say keep that in mind & engineer the serial comms to do just that, and as efficiently as possible.
That is the goal :-) As with everything it is a compromise though right. What you see as best and what I see are bound to differ :-) The trick is in coming to some agreement in the middle ground and getting on with it.
Making the serial link part of a higher level network, where the ECU is one of many addressed devices, to my mind, isn't necessary. Certainly not at this point.
I'm half inclined to agree and half inclined to disagree. Certainly it's not a big thing to add later as it only affects the IF in dealing with a packet or ignoring it. Without that all packets will be dealt with.
If that becomes obviously necessary at some later time, when other 'nodes' are present, then extend or rewrite the serial protocol then, though I suspect it may never be necessary.
But it is a certainty that there will be a datalogger, and a display attached, and maybe a wideband and extra PC and IO extender etc. I don't think it will be that difficult to cover those bases now, we just have to be very careful to make sure I don't muddy the line between protocol and application. As it stands I think I may have already and that may require a bit more work to redefine what is mandatory and what is application dependent.
If you had, say 5 different devices in the car, networked via CAN & wanted to plug your serial cable into any one of them to tweak your ECU tune, then fair enough. For now, KIS :)
Well, I've realised that to "just forward" a packet onto CAN will be tricky, SO, I've pushed that out to the tuning authors and other CAN authors to deal with by saying that you can send a freeems serial packet with ID = CAN fwd and the first X bytes are the can ID and parameters and the balance is data to be sent or similar.
I agree with the point about the ESC start assuring you of a start condition, but there will only ever be a transmitter & receiver on the serial and I think it's an unnecessary overhead.
The trouble is, if you ditch it, you pickup other overheads like multiple false starts etc. If you can almost guarantee that a start IS a start you will waste less time getting synced in.

It isn't my idea, it's Stu's idea (a fellow POHM (http://www.diyefi.co.uk)) but I do like it and I think it should stay unless there is a compelling reason not to have it. After all, whatever is in all of our best interests is what I want to have.
That's just my take on it. In short, I'd go for less overhead at this point, rather than more.
Well, let's look at it :

start - gotta have it unless you use time and quiet periods to determine start.
flags - probably want some for various things regardless of design
source - only for promiscuous serial use
dest - only for promiscuous serial use
ack - only for some packets, but flagging it in and out seems more trouble than it's worth
payload ID x 2 - gotta have it
payload Length x 2 - gotta have it
escapes (x3 on a typical 100 byte packet) - if you have start, you probably want this, plus, low cost, and extra error checking
checksum - gotta have it
stop - not strictly required as you could base on length and checksum etc but keeps the code isolated and clean.

If you ditch the escape/start/stop scheme, you instantly push a bunch more logic down into the lowest layer. If you keep them, the only things the low layer must know about are those three bytes and what they mean. I have it setup to check the checksum when the stop byte comes in so that if flagged it is definitely a goer, but that could be moved out to main loop flag handling too.
As for CAN, is there some standard you intend to be compatible with? I found references to J1939. Is anyone looking at that right now?
I've looked at it, but I don't like it and think it would be a headache to try to align with that. I also don't see any value in trying to fit in with that on a data level. However, if we keep the ability of the serial side open to do almost anything when forwarding CAN packets then we can communicate with whatever is out there using the EMS only as a bridge. For personal CAN use, I think either custom, or aligned with the MS spec in some fashion is the best idea. I'm not sure what the MS spec is like though, it could be that it's a dirty standard that I don't want to align with. Either way, the CAN hardware and network of local devices feature is a nice thing to have, and there is after having thought about it quite a lot, no reason to think about that now at all. The only thing we should think about is having the ability to pass messages on verbatim using a special call.
('CAN' isn't the easiest thing to search the forum for!)
Try Controller Area Network :-)

I trust you saw your name in that doc?

Thanks again for your input :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial Communication Take 2

Post by Fred »

Well, this may have been obvious, but I've not tried to do anything like this before being a Java dev by trade, however when I had a play today with the flagged header field idea it is actually very very easy. So, I'll adjust the documents to reflect my new thoughts and create the code to handle it and then we can have small headers for things that don't require large ones and large ones for things that do.

EdG should be pleased.

As far as I can think there is nothing to stop the header excluding the address fields and defaulting to a 1:1 connection in that case. So, from the above we could end up with a minimal setup that looks like this :

start
flags
payload ID
payload Length
escapes
checksum
stop

Meaning 11 bytes overhead for a 100 byte message, it still doesn't seem worthwhile when the full thing is 14 however we could also ditch the length and have a payload ID that had a fixed length and format. In fact thinking about it now, the length is implicit in every payload format anyway. So is it just me or is the length actually redundant information? go to the right block based on the ID field and lookup the length and check it.

If length is ditched or made optional too then it comes down to

start
flags
payload ID (2)
escapes
checksum
stop

which for an empty request packet is a grand total of precisely 6 bytes from end to end.

Thoughts?

On another topic, it seems that full duplex uart may be a bit tricky. The way it is working now is one or the other. No chance for receiving while sending. I'm starting to think request response and no async (at least not on the primary channel) is a good idea. The only thing async would have been the logging stream which could easily be polled with a 6 - 10 byte empty request packet at quite a high rate.

Thoughts?

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: Serial Communication Take 2

Post by jharvey »

Hmmm, how live can the EMS out data be? At 56Kbits/s we have about 5.6kByte/s across. So one packet of say 6 bytes is about 1k updates/s, and with 10 sensors (RPM, MAP, ect) it's still 100 updates per second, and better if you prioritize things like RPM by sending it twice or more for one cycle of data. So display wise, I'd say that's fast enought.

Hmmm, at what RPM can you get a chunk of data out, for each revolution? For a single sensor with a 6 byte packet, at 56K were not far from 56kRPM.

56kb/s (byte/10b) (packet/6byte) (60s/min) (1 rev/packet) = 56kRPM

Right so if you wanted to see the fuel trim or spark advance for each cycle, you could get it for each cycle if the engine is less than 56kRPM. Not to bad I'd say. Or how many packets for one rev at say 12KRPM.

1min/12Krev (60s/min) (56kb/s) (bytes/10b) (packet / 6 byte) = 4.6 packets / rev at 56Kbs and 12KRPM.

Also not so bad.

I think 19.2 is one of the most common data rates for 232, however with these short distances 56k or better should be easy enough to obtain. Hmmm, now fast can a dumb little PIC or equiv process data. Perhaps the limiting factor isn't the EMS side but the graphical display side.

One thought about the ASCII, I've inserted and sniffed traffic before with a computer and two serial ports. I've done this both to capture data, to process later, and for on the fly debugging. I really don't have much faith in the wire. I've seen funny things happen for funny reasons. I still think some kind of human readability is a good idea. After all were not sending movies across this data link, and I don't see bandwidth as a huge hold up.

I also won't argue that point much, I just want to see it running, and I'm hoping the more complicated approach isn't going to be too big a bite. After all if you don't fail next time, where would we be if you were successful at removing your finger? At some point we need it running. Perhaps I'm a bit eager, perhaps I'm trying to KISS. Any how keep it up, I know I can't make any where as much progress on this as you are. So I encourage not to not argue with me, and spend more time coding. I just wanted to toss some thoughts out there for your use if you like. I'm sure I'll be happy with what ever you pump out.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial Communication Take 2

Post by Fred »

It's 3am. I get up at 5:30am. I promise, I'm working hard on this :-)

GCC is giving me hell, or I don't understand C properly.

Anyhow, tonight I thought of an idea that might make you happy. Debug packets. IE, a normal packet with a payload of arbitrary text. Tahts about as ascii as its gunna get.

as for speed, the speed is 115200. This can be configurable via the interface if need be.

just taking the variables isn't enough, you need the inputs and outputs and there are quite a few.

also, I don't trust the wires either, hence the complexity, to ENSURE that the info you get is solid.

bed for me, stuffed if I know what is going on with this block of code...

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: Serial Communication Take 2

Post by jharvey »

A quick thought sprung up by the comment in the spec that notes all versions might not be compatible with each other. Perhaps we should have a compatibility note in the soft. For example the tuning soft might note under help what version of freeEMS was available when the software was compiled. Or perhaps the soft can check the freeEMS version via serial, and toss up a message if it's not what it expects.

Oh and get back to work, (whip crack,,,, whip crack)

Just a thought.
User avatar
jharvey
1N4001 - Signed up
Posts: 1607
Joined: Tue Jun 10, 2008 5:17 pm

Re: Serial Communication Take 2

Post by jharvey »

Serial spec notes 8 data and then parity. You may find the serial port in 8-n-1 may require 7 data bits with the 8th as parity.

http://en.wikipedia.org/wiki/Parity_bit

I seem to recall it worked that way, and wikipedia seems to re-enforce my memory. However, I haven't used parity for some years. I could go either way with parity, I think the checksumming is more than enough checking.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial Communication Take 2

Post by Fred »

You are talking about the interface version request. Also, what the text says is that some devices will carry a subset, not an incompatible set. There will always be version differences. Two types. Code version with behavioural bug fixes but identical serial/data interface, and interface changes which require a tuning tool change.

As for parity, you can have 7, 7+1, 8, 8+1, etc. We need 8 data, and I want 1 parity! It works very well thus far :-)

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
EdG
TO220 - Visibile
Posts: 5
Joined: Wed Oct 15, 2008 7:57 pm

Re: Serial Communication Take 2

Post by EdG »

Yes, I like the reduced header size :) - it didn't occur to me that the 'length' was superfluous. Either start/end or start/length only is required.

What's the issue with duplex? As you say it isn't really required for now as the processing of messages is sequential, but I feel uncomfortable to think the hardware cant receive a command while sending, like you say, a datalog stream for example.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Serial Communication Take 2

Post by Fred »

I'm sure it can, I just had a bug, and I'm not sure I want two concurrent serial processes occurring anyway. It could get messy.

As for the length, that is not why it's superfluous, you still need to know it to ensure you got no extra zero bytes, but you must know it anyway when you define the payload format itself. So you can correlate the payload id with the length and then check it that way.

For some things it is still required though, eg random length style packets.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
sry_not4sale
LQFP144 - On Top Of The Game
Posts: 568
Joined: Mon Mar 31, 2008 12:47 am
Location: New Zealand, land of the long white burnout
Contact:

Re: Serial Communication Take 2

Post by sry_not4sale »

This part of the spec is confusing me:
Acknowledgments

When a device wishes to receive confirmation that what it asked for was actioned appropriately
and indeed whether the outcome was successful or not it should set the header bit for
acknowledgement valid/required, and populate the acknowledgement field with the next value in
the sequence being used.
If I an requesting an acknowledgement, do I increment the ack/sequence field twice?? Or just increment it once as normal...
Owner / Builder: 1983 Mazda Cosmo 12at (1200cc 2-rotor turbo) coupe [SPASTK]
165hp @ 6psi standard - fastest production car in japan Oct 82
Post Reply