FreeEMS Serial Protocol

FreeEMS topics that aren't specific to hardware development or firmware development.
wilbertcr
DIP8 - Involved
Posts: 16
Joined: Tue Jul 08, 2014 7:44 pm

Re: FreeEMS Serial Protocol

Post by wilbertcr »

You mean interface as in API? What set of files and functions would you say compose the interface? Where can I get them?
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS Serial Protocol

Post by Fred »

You already know what it is. You have decoded the packet down to a chunk of data which is variable. You've been told that that chunk is defined through structs and how they're copied. You've been told where how they are copied is defined. You've been shown which files contain this info. You've even stumbled upon the piece of code that does the copying. Did you even look at those links? Tell me what you find there in relation to "datalog".
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
wilbertcr
DIP8 - Involved
Posts: 16
Joined: Tue Jul 08, 2014 7:44 pm

Re: FreeEMS Serial Protocol

Post by wilbertcr »

Thanks.
wilbertcr
DIP8 - Involved
Posts: 16
Joined: Tue Jul 08, 2014 7:44 pm

Re: FreeEMS Serial Protocol

Post by wilbertcr »

Yes, I read them. It is still unclear. It's ok if you don't want to tell me what's on the packet, just don't be a dick about it.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS Serial Protocol

Post by Fred »

I'm absolutely not being a dick about it. And I AM telling you exactly what's in the fucking packet! Multiple times! You have to read what I wrote again, and do as I said. Once you've done as I said, and not read comms*.c/.h at all, because you don't need to, come back and tell me what you found, and what you think it means. We'll go from there...
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
wilbertcr
DIP8 - Involved
Posts: 16
Joined: Tue Jul 08, 2014 7:44 pm

Re: FreeEMS Serial Protocol

Post by wilbertcr »

I'll ask my professor next week and see what he says. Thank you.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS Serial Protocol

Post by Fred »

Ask your professor? Is this some kind of joke?

The data within that packet consists of chunks of data in sequence. Which chunks, how much of each chunk, and what each chunk consists of are CLEARLY defined in the above-linked files.

If you need to ask your professor for help understanding that, you need to pursue a career in a different field of endeavour.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
wilbertcr
DIP8 - Involved
Posts: 16
Joined: Tue Jul 08, 2014 7:44 pm

Re: FreeEMS Serial Protocol

Post by wilbertcr »

I'm used to looking at RFC's. Protocols normally have something along these lines in their RFCs:
TFTP-Protocol.png
Is it possible for you to define your protocol in this format?

Thank you.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: FreeEMS Serial Protocol

Post by Fred »

Mate, the protocol you've already understood! The guts is a *variable* data format defined as chunks of data by address and location. In the firmware address is given by reference, so you don't have to do any funky calculations. Consider the bit you posted above to the in the category "n bytes, data" as in the above image. We're not discussing the protocol anymore, we're discussing the data. The data is variable! :-D What else is there to say? You want it fucking spoon fed? FINE!

Code: Select all

numberOfChunks: 3,
logChunks: {
    [0] = {
        address: &CoreVars0,
        size: sizeof(CoreVar)
    },
    [1] = {
        address: &DerivedVars0,
        size: sizeof(DerivedVar)
    },
    [2] = {
        address: &KeyUserDebugs,
        size: sizeof(KeyUserDebug)
    }
},
three blocks, use all three, use all of each in sequence.

ONE:

Code: Select all

typedef struct {
	/* Calculated and averaged from ADC0 readings */
	unsigned short IAT;   ///< Inlet Air Temperature           : 0.0 -   655.35       (0.01 Kelvin (/100))
	unsigned short CHT;   ///< Coolant / Head Temperature      : 0.0 -   655.35       (0.01 Kelvin (/100))
	unsigned short TPS;   ///< Throttle Position Sensor        : 0.0 -   102.398438   (0.0015625 % (/640))
	unsigned short EGO;   ///< Exhaust Gas Oxygen              : 0.0 -     1.99996948 (0.0000305175781 lambda (/32768))
	unsigned short MAP;   ///< Manifold Absolute Pressure      : 0.0 -   655.35       (0.01 kPa (/100))
	unsigned short AAP;   ///< Atmospheric Absolute Pressure   : 0.0 -   655.35       (0.01 kPa (/100))
	unsigned short BRV;   ///< Battery Reference Voltage       : 0.0 -    65.535      (0.001 Volts (/1000))
	unsigned short MAT;   ///< Manifold Air Temperature        : 0.0 -   655.35       (0.01 Kelvin (/100))

	/* Calculated and averaged from ADC1 readings (Subject to change! http://issues.freeems.org/view.php?id=190) */
	unsigned short EGO2;  ///< Exhaust Gas Oxygen              : 0.0 -     1.99996948 (0.0000305175781 lambda (/32768))
	unsigned short IAP;   ///< Intercooler Absolute Pressure   : 0.0 -   655.35       (0.01 kPa (/100))
	unsigned short MAF;   ///< Mass Air Flow                   : 0.0 - 65535.0        (raw units from lookup)

	/* Calculated from MAP and TPS history */
	unsigned short DMAP;  ///< Delta MAP kPa/second or similar
	unsigned short DTPS;  ///< Delta TPS %/second or similar

	/* Calculated from engine position data */
	unsigned short RPM;   ///< Revolutions Per Minute (Calced) : 0.0 - 32767.5        (0.5 RPM (/2))
	unsigned short DRPM;  ///< Delta RPM (Calced)              : 0.0 - 32767.5        (0.5 RPM/Second (/2))
	unsigned short DDRPM; ///< Delta Delta RPM (Calced)        : 0.0 - 32767.5        (0.5 RPM/Second^2 (/2))
} CoreVar;
TWO:

Code: Select all

typedef struct {
	/*  */
	unsigned short LoadMain;       ///< Configurable unit of load, same scale as VE by default
	unsigned short VEMain;         ///< Divide by 512 to get 0 - 128%
	unsigned short Lambda;         ///< Divide by 32768 to get Lamda 0 - 2.0

	// TODO remove these:
	unsigned short AirFlow;        ///< Top half of the equation - intermediate, take this out of derived vars into own special struct with other intermediates
	unsigned short densityAndFuel; ///< Bottom half of the equation - ditto, or, just ditch. This is dev/debug code, really...

	unsigned short BasePW;         ///< Raw PW before corrections divide by PW_TICK_FACTOR for milli seconds
	unsigned short ETE;            ///< Engine Temperature Enrichment percentage correction divide by 327.68 for 0 - 200%
	signed short TFCTotal;         ///< Transient fuel correction PW (+/-) divide by PW_TICK_FACTOR for milli seconds

	unsigned short EffectivePW;    ///< Actual PW of fuel delivery, before per channel corrections, divide by PW_TICK_FACTOR for milli seconds
	unsigned short IDT;            ///< PW duration before fuel flow begins, before per channel corrections, divide by PW_TICK_FACTOR for milli seconds
	unsigned short RefPW;          ///< Reference electrical PW what each cylinder is based on, used for all general decisions, divide by PW_TICK_FACTOR for milli seconds

	unsigned short Advance;        ///< Ignition advance (scaled degrees / ANGLE_FACTOR = degrees)
	unsigned short Dwell;          ///< Dwell period, divide by PW_TICK_FACTOR for milli seconds
} DerivedVar;
THREE:

Code: Select all

typedef struct {
	// To be improved MAJORLY:
	unsigned char tempClock;       ///< Incremented once per log sent
	unsigned char spareChar;       ///< Unused at this time.

	// All flags! Pair keeps things sane for hacky apps that think everything is 16 bit.
	unsigned char coreStatusA;     ///< Duplicated, migrate here, remove global var
	unsigned char decoderFlags;    ///< Various decoder state flags
	unsigned short flaggableFlags; ///< Flags to go with our flaggables struct.
	// counter flags once counter mechanism implemented

	// These things should only exist once in memory, and should be grouped in a struct, perhaps this one
	unsigned char currentEvent;          ///< Which input event was last to come in
	unsigned char syncLostWithThisID;    ///< A unique identifier for the reason behind a loss of sync
	unsigned char syncLostOnThisEvent;   ///< Where in the input pattern it all went very badly wrong
	unsigned char syncCaughtOnThisEvent; ///< Where in the input pattern that things started making sense
	unsigned char syncResetCalls;        ///< Sum of sync losses and state clears
	unsigned char primaryTeethSeen;      ///< Free running counters for number of input events, useful at lower RPM
	unsigned char secondaryTeethSeen;    ///< @copydoc primaryTeethSeen

	// Likewise these too
	unsigned char serialOverrunErrors;         ///< Incremented when an overrun occurs due to high interrupt load, not a fault, just a fact of life at high RPM
	unsigned char serialHardwareErrors;        ///< Sum of noise, parity, and framing errors
	unsigned char serialAndCommsCodeErrors;    ///< Sum of checksum, escape mismatches, starts inside, and over/under length
	unsigned short inputEventTimeTolerance;    ///< Required to tune noise rejection over RPM TODO add to LT1 and MissingTeeth

	// TODO move this up above with the other flags post OLV 0.0.3 release
	unsigned short flaggableFlags2; ///< Flags to go with our flaggables2 struct.

	// replace highest first to avoid hassles for offset based dave/mtx...
	unsigned short zsp9;  ///< Spare US variable
	unsigned short zsp8;  ///< Spare US variable
	unsigned short zsp7;  ///< Spare US variable
	unsigned short zsp6;  ///< Spare US variable

	// TODO move these:
	unsigned short blendAlphaNPercent;  ///< TODO migrate to correct place once OLV/EMStudio are flexible...
	unsigned short speedDensityAirFlow; ///< TODO migrate to correct place once OLV/EMStudio are flexible...
	unsigned short alphaNAirFlow;       ///< TODO migrate to correct place once OLV/EMStudio are flexible...

	// Do we want these recorded at log assembly time, or at recording of ADC time, or at calculation of vars (core and/or deriv) or at enabling of scheduling, or all of the above?
	unsigned short clockInMilliSeconds; ///< Migrate to start of all large datalogs once analysed
	unsigned short clockIn8thsOfAMilli; ///< Migrate to start of all large datalogs once analysed
	unsigned char ignitionCuts;  ///< Up to 8 reasons to not schedule ignition
	unsigned char injectionCuts; ///< Up to 8 reasons to not schedule injection
} KeyUserDebug;
For the bit flags you'll have to dig further. Spoon fed. PS, I used the above links, ONLY. NOTHING ELSE. It was *already* spoon fed. In the real world RFCs are not the only thing you'll have to deal with. I had to deal with a moron recently who gave no written indication of how a piece of software was supposed to work, at all. I got the tasks done anyway, though.

Now you can push this data into your app and display it. Easy.

You could experiment with sending body-less reqs for versions etc and parsing the strings out of the replies, too.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
malcom2073
LQFP112 - Up with the play
Posts: 211
Joined: Tue May 01, 2012 4:17 pm
Location: Shrewsbury PA
Contact:

Re: FreeEMS Serial Protocol

Post by malcom2073 »

First off, Hi! Glad to see someone else interested in writing software that talks to FreeEMS! Do you mind sharing (Either here, or in another post) what your intentions and interests are with this? It would help us to guide you in the right direction once you get the fundamentals down.

Fortunately, much like the real world, you often have others around you who are willing to help you through things that may at first seem difficult so you can focus on the core of your project :)

Let me break some of this down for you, it was a bit of a headache getting my head around it in the beginning as well.
AA 01 01 91 00 62 B7 CB BA 0D 03 6B 41 80 04 47 04 69 01 37 B7 CB 00 0C 00 0D 00 06 00 00 00 00
00 00 00 00 00 00 04 47 38 00 80 00 01 32 84 54 00 91 53 33 00 00 00 BC 0E 36 0E F2 00 00 47 0C D6
00 00 00 00 D0 00 00 00 00 00 00 00 00 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 35 19 A8 CF 00 00 F3 CC
As per a document I'm assuming Fred already sent you (If he did not, ask and I'll see if I can find it again), you figured out
0xAA : Start of packet
0x01 : 7th bit is on. I know Bit 0 - Has Length, Bit 1 - Is Negative Acknowledgment, Bit 2 - Has Sequence. What are Bit 3, 4, 5,6 and 7?
0x0191 : Basic Datalog packet???
0x00 : Mode Stop
You got two things wrong there, 0x01 means the 0th bit is on, not 7th which is HAS_LENGTH This means that the two bytes after the payload (again, re the protocol document) 00 62 is the length. This means everything else except your checksum and stop byte is the packet itself.

Now that you have the packet read, you should de-escape the bytes. Once again that is documented in the protocol document which bytes mean what. Once all that wonderful magic is finished, you now have a complete datalog packet.

The datalog packet format itself is not documented anywhere that I know of, so you have to read through the firmware source files Fred linked to get a feel for what order the bytes are in. I broke it down here: https://github.com/malcom2073/emstune/b ... r.cpp#L155 but be advised, this is subject to change as the datalog format will in the future be configurable.

Basically, each DataField object constructor allows you to give it a name, description,byte offset, size (in bytes) of the value, divisor, and offset. ((value / divisor) + offset). You should be able to glean from the firmware source, and that .cpp file the basic values you can pull out of datalogs (at the moment). You make it as modular as possible, to allow the datalog format to change on the fly (as I have, easy to reconfigure the offsets/value order)

I highly recommend idling on IRC and seeing if you can catch me, irc.freenode.net, #freeems, #freeems-dev, and #freeems-ot (this kind of talk would be in -dev I believe) if you have more troubles.
Post Reply