“Beauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye.” - Miss Piggy

Pic 3

Hooovahh's Blog

CAN Part 9 - CRCs and Incrementing Counters (Bucket Filling Technique in XNet)

07/08/2017 09:16 PM

As this blog series gets longer these intros become less meaningful.  The intent of the into is to summarize all the information covered in the previous blog posts, but by now we've basically crammed a year or so of practical CAN knowledge into 8 blog posts.  As a result trying to summarize it two sentences really practical.  Just know lots of CAN stuff has been talked about, and this post is going to build on top of the previous knowledge of reading and writing raw CAN frames, as well as the in depth XNet discussed in Part 6.  In this blog post we are going to cover the incrementing counter and CRC that sometimes takes place in automotive communications, as well as ways of satisfying these requirements using hardware timing on XNet devices.

Incrementing Counters

In automotive based CAN, it is not uncommon to have a rolling counter on some signals in a CAN Frame.  The purpose is to ensure the integrity of data being sent, by incrementing a value with every transmission of the frame.  Then any device on the bus can read all frames being transmitted, and can determine if the device is still talking, and still performing the task of incrementing the counter, which may give some insight into the status of the device.  If a listener on the CAN bus still sees frames coming in, but they are not incrementing, then I can know that some part of the device is still functioning and I might be able to better understand the devices status.  Here is an example of an incrementing counter:

Time 0.000 ID 0x10 Payload: 0x00 00 00 00 00 00 00 00
Time 0.010 ID 0x10 Payload: 0x00 00 00 00 00 00 00 01
Time 0.020 ID 0x10 Payload: 0x00 00 00 00 00 00 00 02
Time 0.030 ID 0x10 Payload: 0x00 00 00 00 FF 00 00 03
Time 0.040 ID 0x10 Payload: 0x00 00 00 00 FF AB 00 00
Time 0.050 ID 0x10 Payload: 0x00 00 00 00 00 AB 00 01
Time 0.060 ID 0x10 Payload: 0x00 00 00 00 FF AB 00 02

Here we see that the last byte in the payload counts from 00 to 03, and then resets back to 00.  The other bytes of the payload may change or they may not, but that last byte must be incrementing every time it is sent, and in our example that is every 10ms.

Being a listener on a CAN bus, and determining if another device is still talking isn't too difficult to do in any API.  For most APIs you can just read all frames, pull out the counter value and compare it to the previous counter value that was read.  For XNet things can be a bit easier since we can perform a buffered read on Signals or on Frames getting all values that have been sitting in the buffer.

But if we are to simulate a device on the bus, and are required to increments a value, then the timing of our software is going to be much more critical.  Most devices looking for an incrementing counter have a relatively short timeout and may go into a faulty state if our software doesn't write the incrementing values in a timely manor.  It is not uncommon to have timing requirements of calculating the new CAN frame value, and sending a new one out every 10ms.  In Windows the reliability of timing is always a struggles.  In any general purpose operating system many other application might be trying to get the CPUs to perform their work.  Having a 10ms timing requirement is possible in software timing, but will likely have lots of jitter sometimes maybe taking 20ms, or more on overloaded systems.

So what's the solution?  Well some CAN hardware have the ability to automatically retransmit CAN frames on the hardware level, which doesn't require software to continually perform functions to keep the transmission going.  When we explored the XNet API in Part 6 we saw that some of the modes like Single Point allowed us to perform the write once, and then the database would handle how often the data would actually be written.

If we use a write mode like Single Point then we can be sure the data will go out every 10ms since it is hardware timed, but what we be able to do is change the value with every frame going out.  For this level of control what we are going to need is Queued mode.

Concept of Filling a Bucket

If we want to use XNet to allow for control over every frame going out, and have the timing be hardware controlled we are going to need something like this example illustration:

At the start of the application start we want to write several frames into a buffer which will be sent one at a time by the hardware.  The amount of frames put into this buffer doesn't matter too much, we just want to ensure that the buffer doesn't reach zero elements before we are able to put more in.

At 10ms the first frame goes out, then 20ms the next then 30ms the next.  This timing of transmission is handled by the hardware, but this only works if there are still more items in the buffer to send.  If there are no more frames in the write queue then the transmission will stop.  This means as the bucket is being emptied we need to make sure we fill it back up with another write function.

Now we see that at 40ms the next frame goes out, but what our software needs to do is ensure the bucket (or queue) never gets emptied completely, and adds some more frames to be sent out.  We need to keep track internally of the last value we wrote back at Time = 0 and start writing more adding one to the last written.  In our example we last wrote 01, so the next write needs to start at 02, so we write 02, 03, 00, and 01.  This is what I call the bucket filling technique, where the frames are being written one at a time, but we add new frames to be written every so often.  The timing of our write call isn't critical at all and in our example we first added 6 frames, so we need to write more within 60ms since that is how long it will take for the queue to be emptied.

Now that we have the concept lets look at how this appears in LabVIEW

Basic XNet Bucket Example

Here is a basic example which closely follows the images from earlier.  Here we are sending 8 bytes on a periodic CAN frame, with the first byte being a counter which goes from 0 to 3 back back to 0.  Bytes 2 through 8 come from the Payload control.  The VI will add 10 frames to the queue, and then periodically check to see how many are in the queue.  If the number of pending items is less than 10 this VI adds 10 more.  One downside of this is that a value change on the Payload may not be seen on the CAN bus until the remaining pending items are sent.  There are some more advanced techniques like flushing the buffer on a value change of the other payload bytes but then there is more work to know what the counter value should be.

Payload CRC

As discussed in Part 2 the CAN frame has a very basic CRC built in which ensures that the message is received properly.  In some automotive applications an additional CRC can be added to ensure the data being received is correct and from a device that is still communicating.  In all instances I've seen this implemented this is done with the last byte of the payload.  The particular algorithm for generating the CRC can vary, but the SAE J1850 standard uses a polynomial of 0x1D, so we will use that for this example.

With this in mind we have another reason we may need to calculate the data for each frame.  If we have an incrementing counter, and we have a CRC, then the CRC is going to change with every frame sent, since the counter will change with every frame sent.  In addition to this the payload may change, and the CRC will need to change when that occurs too.  Luckily there's an example for that too.  This is very similar to the previous example except the last byte is used for the CRC, and the CRC is calculated using a VI found in the XNet Tools package which I originally found on the NI forums here.

This is pretty similar to the last example except there we take bytes 1 through 7, perform a CRC, and then use that as the last byte in the payload.

Non-XNet Solution

So lets say you don't have XNet hardware, and can't push frames into a queue that get sent out on hardware timing.  Does this mean you can't perform an incrementing counter and CRC?  No it just means that the timing needs to be done in software, and as a result a decent amount of jitter will be seen.  Your code will typically sit in a small while loop waiting for some amount of time to go by, and then send the next frame.  On Windows you can expect jitter on the order of 10ms or so.  You may get lucky and have no jitter, but because Windows is a general purpose operating system you can never guarantee timing.  If your requirement is to send out a frame every 100ms or more software timing will probably work just fine.

Part 10

I've been trying to put out a new CAN blog post about every two weeks, but I'm starting to run out of ready made content.  As a result Part 10 might not come out for a while.  I'd like the next part to be on XCP and CCP or CAN flashing ECUs using XCP or UDS, but these topics will take time to research, and the content just isn't ready.  Alternatively I have one more XNet trick up my sleeve that I wouldn't mind talking about if I can get NI's permission to talk about an undocumented feature.  

I guess this is me saying good bye for now.  If you have any CAN and LabVIEW related questions feel free to post them over on the Automotive Network subforum over at Forums.NI.com.

CAN Part 8 - ISO 15765, KWP2000, ODB, and UDS (Automotive Diagnostic Command Set)

07/05/2017 02:28 AM

In the earlier parts of this blog we CAN hardware, software, APIs, databases, CAN Signals and CAN Frames.  In this part we are going to talk more about one of the software techniques that has been developed to get around some of the technical limitations to CAN to help allow for other types of data, with larger payloads, to be sent and received.

Standard CAN Example

As defined by the CAN 2.0 spec, a standard or extended CAN Frame consists of up to 8 bytes of payload, in 1 byte increments.  This is a hard limitation and cannot be changed, without changing to a new specification like CAN FD which also requires new hardware from multiple devices.  But often times when we start getting into automotive ECU development we find ourselves wanting to use CAN in ways that maybe it wasn't intended to be used for.  For example the CAN data we've been talking about so far has primarily been periodic information.  A device may send out the status of an analog input every 100ms by sending out a single frame that contains this signal information.  This information is great for looking at values that may change rapidly over time, and having a waveform of all data at small time increments like that is very useful.

Engine Trouble Codes Example (DTC)

But lets say for a moment that I want to use CAN for something else.  Let say for instance my check engine light has turned on in my car and I want to know why.  An automotive technician will plug in a device on the CAN bus and it will ask the ECU what is wrong.  The ECU will then respond and tell the scanner what the list of things are that are currently wrong, and a list of things that were wrong but aren't any more, to give some history of issues to the technician.  These trouble codes are known as Diagnostic Trouble Code (DTC), and aren't periodically sent, and instead only are sent when another device asks it to.  DTCs are the kind of data that should not be sent every 100ms because over the life of the ECU there is a small chance that a device that wants to know what the DTCs are, is even listening.  So this type of data is sent only on the event that it was requested.  Standard and Extended CAN data can be sent this way but there is another issue with DTCs.

The list of potential DTCs any ECU can have set could be in the hundreds.  And we cannot know how many are set, or which ones are set, until the ECU tells us.  This causes the response from the ECU to vary the amount of data it is going to send back depending on other conditions, and it will likely need to be able to send back more than just 8 bytes of data which a CAN frame is limited to.

ISO 15765 Multi Frame Messaging

This is where the ISO 15765 protocol comes in.  The protocol is used in places where more than 8 bytes of data need to be received, or more than 8 bytes of data needs to be sent as a response.  It is also used in places were the data that needs to be sent or received is not the kind of data that should be blindly broadcasted on a network autonomously.

How does it work?

The basics of the 15765 are pretty simple.  A device will send out a single frame telling another device that it has something it wants to say.  Now if the thing it wants to say is small in size, it can be all contained in that one frame.  But often times we need to send more data so the sender will wait for the receiver to get the data, and then respond.  In the response the receiver will state how many frames the sender can send at once, and how much time can be between each frame.  Older CAN devices had a small buffer size, and wouldn't be able to receive data too fast.  So the sender is supposed to receive this one frame, known as a flow control, and then send back more data in with the timing specified.  This helps prevent the device receiving the data from being overloaded, since the sender will only send the specified number of frames requested.  Once all of the data gets received, the receiving device will act on it, and then send a reply back.  Just like the request, the response can also be a single frame, or multiple frames.  If it is multiple frames then it will send the first frame, and then the original sender device will send back a frame stating how many frames it can receive, and what timing should be between them.  These steps are a bit confusing so here is a flow diagram explaining it, generated using the NI State Diagram Toolkit.  The state machine starts on the green node, and ends on the red one.

Looking at this flow diagram things might be a bit confusing, so lets break it down a bit more and talk about the various components and checks that go into using raw frames for performing 15765 queries.


So in Part 2 we first talked about CAN Frames and mentioned the payload limitation, as well as the fact that each frame has an ID.  Well when using ISO 15765 there are only two IDs ever used.  One ID is for the request and one ID is for the reply.  The ID performing the request is known as the Command Receive Object (CRO), and the ID for the response is known as the Data Transmission Object (DTO).

This means that all requests from other devices on the bus will have the same IDs, and the same priority.  This also means that the reply sent back will go to all devices on the bus, and it is the the responsibility of the device to look at the payload sent back to determine if the reply it found was intended for that device, or if the this response was to another devices request that was issued.  On a normal automotive CAN bus these 15765 messages aren't going out often, and as mentioned with our DTC example is something that is useful for debugging but isn't intended to be part of normal operation.  As a result this issue of multiple requester at once is probably a rare case and in most situations every reply you see to an ISO 15765 message, is going to be from a request your device issued.

The CAN IDs used in 15765 can vary from device to device, and will be specific to your device.  It is common to have different request and response IDs for each device on the bus so this information needs to be looked up in the documentation for the device you are trying to talk to.  In the following examples we will use CAN ID 0x70 for the request and 0x7F for the response.

Simple Example

The following is a simple example where we send a small amount of request data, and we receive a small reply following the state machine from above.

Write Single Frame

At the start of our state machine we have a state where we either write a Single Frame if our payload is less than or equal to 7 bytes, or we will write the first frame in many if it is more.  Lets start with writing the single frame and how it looks.  Lets say we intend on writing the following 3 bytes of data 0x22 00 02, when we do this using 15765 our Single Frame we send will look like this:

--> ID 0x70 Payload 0x03 22 00 02 00 00 00 00

The padding of the frame to 8 bytes isn't always required and from now on will not be included but some hardware may expect it.  The rest of the payload is pretty simple.  The first byte represents the number of bytes in our transmission.  Because this first byte always says how many bytes to expect, it leaves us with only 7 bytes of usable payload.

Some messages don't require a response, and if we are sending a Single Frame, and we don't care about the response we are done.  If we do care about the response we then go and start reading frames looking for the response in the Read Reply state.

Read Reply

So in our example we have successfully sent one frame which contained all the data we wanted to send.  The next step is to look for a reply.  The only reply we care about, is one on the CAN ID associated with a response and in our case is 0x7F.  All other frames can be ignored.  If too much times goes on and we haven't seen a frame with ID 0x7F we go to Done where we can throw an error.  This timeout is usually on the order of 1 second but can vary depending on hardware.  Assuming we do see a frame with ID 0x7F the next step is to identify what kind of frame it is.

Process Read (Single Request)

So we sent a frame, and we just got one back that is in response to what we sent.  Now lets identify what kind of frame it is in Process Read state.  Just like the first step in our state machine, the reply could be a single frame taking 7 or less bytes, or it may need to be multiple frames.  For simplicity lets say the reply is also 7 bytes or less and consists of the following data, 0x01 02 03 04.  As a result the reply may look something like this:

<-- ID 0x7F Payload 0x62 04 01 02 03 04

The first byte of the response is 0x62 because the first byte of the request was 0x22, and the response needs to be the same as the request, but with 0x40 added.  This helps us know the response we found was to a request we issued.  If the first byte was anything else we know the message was not for us.  The only other valid first byte that is for us is 0x7F which is a negative response which we'll get into later.

So 0x62 means the reply is for us, and 0x04 means we have 4 bytes of data to receive, and then after that we have the 4 bytes of data.  So in this example we sent the three bytes of data 0x22 00 02, and we received four bytes of data 0x01 02 03 04.  Using another protocol on top of 15765 known as UDS (ISO 14229), this example data corresponds to asking the device to read an identifier (memory address) 0x00 02, and the device returning the memory at that address which was 0x01 02 03 04

--> ID 0x70 Payload 0x03 22 00 02 00 00 00 00    #Request to read data at address 0x00 02
<-- ID 0x7F Payload 0x62 04 01 02 03 04              #Response data at that address 0x01 02 03 04

 Negative Response

If we got a negative response from the device it would look something like this:

<-- ID 0x7F Payload 0x7F 22 11

The 0x7F means this is a negative response, to the request 0x22, and the status of the response is 0x11.  There can be many custom response codes but there are a set of them that are mostly standard.

Negative Response Table
Response CodeResponse
0x10General reject
0x11Service not supported
0x12Sub-Function not supported
0x13Incorrect message length or invalid format
0x14Response too long
0x21Busy repeat request
0x22Conditions not correct
0x24Request sequence error
0x25No response from sub-net component
0x26Failure prevents execution of requested action
0x31Request out of range
0x33Security access denied
0x35Invalid key
0x36Exceeded number of attempts
0x37Required time delay not expired
0x38 - 0x4FReserved by Extended Data Link Security
0x70Upload/Download not accepted
0x71Transfer data suspended
0x72General programming failure
0x73Wrong Block Sequence Counter
0x78Request correctly receive, but response is pending
0x7ESub-Function not supported in active session
0x7FService not supported in active session

Looking at the state machine the only other option is needing to reissue the request.  If this happens the sender should send the request again.  One other negative response that doesn't end the state machine is if the device asked for more time 0x78.  If this negative response is seen  the receiver should reset the timeout value and continue looking for the next positive, or negative response.

More Complicated Example

The following example is a bit more complicated and will involve sending a larger amount of request data, and we receive a larger amount of data in a reply, again using the state machine from earlier.

Write First Frame

The example listed above is a very simple one.  We send a single frame, and we get a single frame.  but 15765 is more useful when sending large amounts of data.  So lets walk through an example of sending more data.  If we want to send more than 7 bytes of data we still need to start by sending the First Frame and then we will send more when we are ready.  Lets say for instance we want to write the following 17 bytes of data 0x2E 00 02 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D, our First Frame will look like this:

--> ID 0x70 Payload 0x10 11 2E 00 02 00 01 02

Here the first two bytes are used telling the receiver what the transmission will contain.  The second nibble in the first byte of the payload being a 1 is telling the receiver that there is more data than just this one frame.  This leave the remaining lower nibble, and 2nd byte to be used to tell the receiver how many bytes to expect and in our case it is 0x00 11 or decimal 17 which is the full payload we need to send.  Since the first two bytes of the payload are taken by the number of bytes to be sent, this leaves us with only 6 bytes to send usable data through, and so we take the first 6 bytes of our 17 bytes and send it.

Reply is Flow Control

So after sending the first frame, we will wait reading new frames just like the previous example, but we expect to see a flow control frame.  This is a frame that tells us how many frames the device can read at once, and how much time needs to be between each one.  The reply will look something like this:

<-- ID 0x7F Payload 0x30 05 0A

Here the 0x30 tells us this is a flow control message, 0x05 means the device can accept the next 5 frames before needing to reissue the flow control.  This helps so that we don't overload the device and we know that we are sending data to a device that is still listening.  And 0x0A is the separation time needed between each frame.  The units of this are in milliseconds from 0 to 127, and in microseconds from 100 to 900.  This doesn't need to be precise and is a lower bounds, so the device should be fine if we send them a little slower than that, but we shouldn't send it faster than requested.  A zero for the number of frames means just send them all, and a zero for the separation time means no delay between frames is needed.

Write Chunk of Data

After we sent the first frame, and we received the flow control response, we are now able to start sending chunks of data.  We need to abide by the flow control rates, and number of frames.  In this case we are only going to need to send 2 more frames and the device said we can send up to 5, as long as we separate them by 10 milliseconds.  Because of that our next two frames we write will look like this:

--> ID 0x70 Payload 0x21 03 04 05 06 07 08 09
--> ID 0x70 Payload 0x22 0A 0B 0C 0D

The first byte in the payload here goes from 0x21 and increments to 0x2F then back around to 0x21 with each frame sent in this chunk.  In our case we only needed to send 2 frames.

Read First Frame Reply

So we sent some data, waited for the device to reply, then we were able to send the rest of our payload.  Now we need go into perform similar work as we did in the first example.  We continue to read data looking for a reply.  Just as in the earlier example the reply may tell us to re-transmit a request, wait longer for a reply, or get the reply.  The reply can be a single frame as in the previous example, or we can have a multi frame reply, just like our multi frame request.  We've already seen a single frame reply example so lets look at the multi frame reply.  In this example lets assume the device wants to reply with the following 21 bytes of data 0x6E 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14.  This will start with the first few bytes of the payload as before:

<-- ID 0x7F Payload 0x10 15 6E 01 02 03 04 05

Here we see the second nibble telling us this is the first frame of many because it is a 1, and the first nibble and second byte is 0x00 15 telling us there are 21 bytes total to receive.  Then the remaining 6 bytes is the first 6 of the reply.

Write Flow Control

So we've sent our request, and we got the first frame of the reply, but now we need to tell the device how many frames it can send us, and how much time can be between each frame.  In most cases our hardware can just read as many frames as possible since they will sit in our hardware buffer, and we don't really need any time between.  But just for an example we will tell the device we can only read 4 frames at a time, and we need 20 milliseconds between each:

--> ID 0x70 Payload 0x30 04 14

After we send this the device is free to send up to 4 frames with 20 milliseconds between them.  To tell the device to send all frames with no delay replace the last two bytes to 0.

Read Consecutive

So we sent a request, and we got the first 6 bytes of the reply, and we sent the flow control telling the device to send the rest which will look like this, which mirrors how the request message sent multiple frames earlier:

<-- ID 0x7F Payload 0x21 06 07 08 09 0A 0B 0C
<-- ID 0x7F Payload 0x22 0D 0E 0F 10 11 12 13
<-- ID 0x7F Payload 0x23 14

After this it is the responsibility of our software to concatenate all of the data into the request and response.  Here is the full transaction that takes place:

--> ID 0x70 Payload 0x10 11 2E 00 02 00 01 02     #First frame of multiple in request
<-- ID 0x7F Payload 0x30 05 0A                             #Flow control telling us how to send the request
--> ID 0x70 Payload 0x21 03 04 05 06 07 08 09     #More data in request
--> ID 0x70 Payload 0x22 0A 0B 0C 0D                 #More data in request
<-- ID 0x7F Payload 0x10 15 6E 01 02 03 04 05     #First frame of multiple in reply
--> ID 0x70 Payload 0x30 04 14                               #Flow control telling how we can receive data
<-- ID 0x7F Payload 0x21 06 07 08 09 0A 0B 0C   #More data in response
<-- ID 0x7F Payload 0x22 0D 0E 0F 10 11 12 13    #More data in response
<-- ID 0x7F Payload 0x23 14                                    #More data in response

(If you made it this far congratulations, that was a very large wall of text in those two basic examples.)

Other Protocols

The ISO 15765 just defines how to read and write multi-frame data, but doesn't define what that data is.  The Unified Diagnostic Services (UDS), Keyword Protocol 2000 (KWP2000), and On-Board Diagnostic (OBD) and three software layers that each sit on ISO 15765 to perform operations like reading and clearing DTCs.

Automotive Diagnostic Command Set (ADCS)

The NI toolkit ADCS is basically the software implementation of ISO 15765, KWP2000, OBD, and UDS for NI-CAN and NI-XNet devices.  This is fine if you have these hardware options, but the software to implement this in any other language is laid out before you.  So I decided to take the ADCS toolkit, and replace the core calls to the NI DLL driver, which is locked down, and replace it with my G implementation.  I posted this idea on the Idea Exchange.  I also ran into several bugs with the toolkit and getting support wasn't easy, so I eventually posted the basic G implementation of the toolkit for NI-CAN hardware here.  With the G implementation before us, all that is needed to this functionality in other hardware is to just replace the CAN Read, and CAN Write functions with the hardware API of your choice.  This means that ISO 15765, and really the ADCS functionality is now available for any CAN hardware in LabVIEW.  I've created a set of code and wrappers and inlcuded them in the Hooovahh CAN ISO 15765 package, which works with the CAN Drivers package for 3rd party devices from earlier.

So with this newly created G code and palette, what steps are needed to read the DTCs on an ECU, and then clear the DTCs?  Take a look at the following example.

Here we open the first Vector port, initialize the protocol based on the request and reply ID, read the DTCs, Clear DTCs, then close the references  There are some default values on some subVIs but the values used are the most common.  All of the ISO15765 protocol, and the UDS protocol that sits on top of it is all available in the G implementation inside those VIs.  These protocols and APIs work with any KVaser, Vector, or Intrepid CAN hardware device, since it uses the Hooovahh CAN Drivers package from earlier to perform the raw CAN frame reading and writing.  The hardware drivers aren't wrapped into a nice hardware abstraction class, but they probably should be, I just didn't get around to it.  Regardless the point of this code is not to give a working solution for your project, but instead to demonstrate how ISO 15765 operates, and demonstrate that it can be used on any CAN hardware as long as you can read and write raw CAN frames.  This means adding support for other hardware types like NI-CAN, NI-XNet, or any other device can be done without too much extra work.

Part 9...

In this blog series we've covered a lot of information.  From hardware, to APIs, to CAN Frames, CAN Signals, databases, advanced XNet features, and now a software layer on top of CAN to help facilitate multi frame data, and event based communication.  What else could we possibly talk about?  The subjects are now going to get a bit more specific.  So next we will have a quick discussion on the availability of what I call the bucket filling problem in XNet, and how to handle functions like CRC, and incrementing counter signals.

CAN Part 7 - Logging CAN and other File I/O

07/05/2017 02:31 AM

Throughout this series we've talked about how to read and write signals and data and so forth.  But we haven't yet talked about formats of saving CAN data to a file.  Since CAN data is generally a complicated data structure consisting of multiple data types, saving it to a file isn't always easy, and you may come up with something that works, but is non-standard and requires non-standard tools.  In this series we will talk about reading and writing CAN data as TDMS files and BLF files, as well as talk about other common operations like reading HEX, and S record files, which will be important for later blog posts.  This code and examples can be installed by installing the Hooovahh CAN File IO package, which will be referenced in this post, and is available on the downloads page for this blog series.


The TDMS file itself is a very interesting topic, and one that I've presented on in the past at NI Week.  It is best understood as a file format that is similar to Excel.  You have a file (like a workbook) with multiple groups (like a worksheet) with channels (columns in a table) and samples in those channels (rows in a table).  TDMS also has all kinds of interesting features like properties.

So how can CAN data be stored into a TDMS file?  Well luckily NI came up with a standard that flattens raw CAN frames into an array of bytes, and can then read the data back turning it into frames.  When a TDMS file uses this standard format for CAN data, some of NI's tools like DIAdem can read the TDMS file and recognize it has CAN data in it.  Luckily NI provides some examples on how to go from CAN frames to a TDMS file and it just requires one channel within one group.  The basic example can be found in the CAN Input Stream to TDMS Logfile.vi example, which gets installed with the XNet drivers.

Like many examples done by NI they only go so far, and are usually not the most efficient.  That's why I've cleaned up their code, and added a few features like logging CAN IDs as a separate channel for filtering purposes.  After installing the Hooovahh CAN File IO package, you'll have a subpalette devoted to reading and writing to a TDMS file.  For writing first use the Configure TDMS channel for write, providing all the information you can.  Then call the Write CAN frames to log to the TDMS file.

The result will be a TDMS file will has CAN frames data flattened, and written to the channel specified.  Using my API there will also be a second channel in the TDMS file recording the CAN IDs of the frames logged.  Using this we can do more efficient reads, finding only the frames that have a desired CAN ID.  Just like the write process, use the Configure TDMS Channel for Read.  You can also use the List CAN Channels to find what channels in a TDMS file contain CAN frames.  Then perform a Read CAN Frames.  The Read CAN Frames is polymorphic and allows for reading N frames from a given offset, reading frames that match a specific set of IDs, or reading a set of frames from a set of signals in a database or DBC file.  The result can be a pretty effective method of reading CAN frames in a file given the information you know about what you want to read.  Reading only a subset of the data at a time is also useful for when your log file is very large.  Included in the package is an example showing how to read CAN data from a TDMS file, allowing for graphing of signals, given a time window, and IDs to read.


NI has their standard for logging CAN data to a TDMS, but since TDMS files aren't as common outside of NI software, another option is the BLF file.  BLF is a file format primarily used by Vector and their CANalyzer software.  CANalyzer has the ability to log, and replay BLF files which is a binary file, in a custom Vector format that is partially documented.  Luckily Vector also provides a Windows 32 bit DLL for reading and writing to BLF files.  Because the format is more closed there aren't as many features of the BLF API.  Still with the API you can perform writing operations, as well as reading N frames from a given offset.

With both APIs sharing some of the same functionality, there are a couple of included examples that allow for converting from one format to the other.  This is just a read from one API and a write to the next.  This means that other software tools like CANalyzer can be used to process CAN data read by LabVIEW.  This also means that CAN logs created by CANalyzer can be converted and then processed by software written in LabVIEW.

You'll notice that both TDMS and BLF data only read and write the raw frames of a CAN log.  Things like frame to signal conversion still is done by the software after the read operation.  Luckily tools like the ones I've already covered in Part 5 can perform the conversion for you.

Reading Hex and S Record Files...really?

Yes really.  I know this might seem like the kind of thing that doesn't belong, but believe me it does.  Included in the CAN File IO package is a set of tools for reading and parsing HEX and S Record files.  HEX and S Rec. files are ASCII text files, that are used in defining what data should be put into what memory address on an embedded device.  HEX files are common in things like Arduinos that take the C++ code, and compile it down to the functions the hardware need to perform.  Since HEX and S Rec. files are used on all kinds of different hardware with various architectures (8-bit, 32-bit, etc.) this means that there are slight variations on the format used in these files.  The formats of these files is clearly defined in the Wikipedia articles that describe them.  Just like other examples, this all started from an NI example, this time included in the ECU Measurement and Calibration toolkit.  The example was very inefficient, with lots of unnecessary operations, and optimizations in reading it differently.  After improving it the result is a VI that takes a file path, of any HEX or S Rec. file type, and will return an array of addresses, and the array of byte data that should be written to those memory address on the device.

If you've made it this far you might be asking yourself, "What in the world does this have to do with CAN?"  Well my friend, the automotive CAN bus has been used for several years now to be able to reprogram ECUs in your vehicle.  Your car already has a CAN bus for communication, and the OEMs just figured they could use that for reprogramming hardware in the car.  If software in an ECU has a critical bug found, you can take your car to a dealership, and they will reprogram it.  Of course if that ECU is buried behind the engine getting to it physically will be very difficult.  So instead a technician can plug into a diagnostic port under the dash, and a tool can reprogram the ECU without having physical.  There are two basic methods of performing what is called CAN Flash, which is flashing an ECU's firmware using CAN.  We want to go over a few examples in the next section of CAN flashing, but only once we've understood how to parse and read the firmware, which are HEX and S Rec. can we hope to program the ECU.  So you see it does relate to file I/O operations and CAN.

Part 8...

So now that we know how to read and write CAN data to files, and now that we can read HEX and S Rec. files, we can look into CAN flashing ECUs.  But what protocols are used for CAN flashing?  What are the common steps in flashing?  How does one even send multi bytes of data to a CAN device?  In the next part we can answer these things in preparation for CAN flashing examples.  But for now Part 8 is going to be dedicated to the ISO 15765 protocol used to read and write multiple frames of data for other protocol standards like KWP2000, UDS, and OBD.

CAN Part 6 - In Depth XNet

06/20/2017 11:54 PM

Okay so we've covered a lot of information in this series of posts so far.  From hardware to the different variants of CAN, and how it compares to other communication protocols.  We've talked about the raw data sent and received.  We've also talked about NI's offerings with NI-CAN and NI-XNet, along with several 3rd party APIs.  And we've also covered the conversion from raw frames to signals, and back.  In this post we're going to try to cover the more advanced features of the XNet API, and the different XNet sessions that can be used hardware.  XNet is different from all the other CAN APIs mentioned so far and so using it can sometimes be confusing at first.  But it is by far the most flexible and powerful of any mentioned in this series.

More than Just CAN

So the XNet API was designed to work with several communications types.  At the moment the API works with the CAN bus, the LIN bus, and Flexray.  The idea is that each of these APIs (and potentially others) all need an open, read, write, close, and database management functions.  Sure some things are going to be specific to one bus type, but by using this simplified API, anyone familiar with XNet can use it with other hardware types.  So while this blog series will focus on the CAN features of XNet, just know that many of the things talked about here apply to LIN, Flexray, and potentially others in the future.

Base XNet Palette

From looking at the XNet palette, NI has tried to make a very simple looking API, but hide away some of the more advanced features so that new users can get started quickly.  While still allowing for more low level control to those that need it.  First lets look at the base palette.

The Create Session, Read, and Write are all polymorphic functions that can adapt to different functions.  And the Session Node is a property node that can perform other useful functions on the session reference returned by the Create Session.  

Session Properties

Using the Session Node, you can read or write properties of the session, but you should be aware that writing to some properties, will effect all sessions on that hardware's port.  For instance one thing you can do with CAN is turn on and off a 120 Ohm resistor.  Back in Part 1 of this series we talked about how for a proper CAN network you should have two 120 Ohm resistors, one on each end of the bus, between CAN High and CAN Low.  The high speed XNet CAN hardware can turn on or off this resistor using software which is super handy for setups that may or may not need it.  This type of setting is clearly going to be turned on or off on the physical port.  And if you have multiple sessions using that single port, turning on or off the resistor will turn it on or off for all sessions using that port.  

The list of properties available are pretty extensive and some are network specific.  All are pretty well documented in the context help, and most have examples showing how their functions work.

XNet Session Types

There are 12 main session types for the CAN bus, and a conversion session.  Each of these sessions can be created using the Create Session polymorphic VI.  The conversion session was mentioned in Part 5 of this series, and is basically a way to convert from Signals to Frames and Frames to Signals but doesn't rely on any actual hardware.  There are 3 session types for Signal Input, 3 for Signal Output, 3 for Frame Input and 3 for Frame Output.  Luckily the read and write session types are similar and just work in reverse.  This leaves us with only 6 other session types that we will go over in more detail.  Each of these session types has at least one example installed with the API and can be found by going to Help >> Find Examples.

Frame In Stream

This session type works like all other APIs we've talked about so far as a CAN read.  If you want to read you call the read, and you'll get all the frames since the last read.  As the frames are seen by the hardware they will sit and wait in a hardware buffer until your software reads them from the buffer.  When you perform a read, all frames waiting in the hardware buffer get returned as an array.  This session type is useful when you want to view every frame coming in and either evaluate them or log them.  Here is a nice illustration made by NI.  

Notice that all frames came in earlier, and a single read returns them all at once.  The read function also has optional inputs which can specify things like reading a set number of frames, and having a timeout on waiting for those frames.

Frame In Single Point

The Single-Point session type is most useful on frames that you want to know the most recent value for, but don't really care about each individual frame.  When starting this session type you must specify an array of Frames that will be read and returned in the order requested.  These frames identifiers come from the database selected, which will be talked about later.  Since so many CAN devices will send out data periodically, it isn't always important to evaluate every frame.  Lets say a device sends out the device status every 10 milliseconds.  If our software is looking to react to when the status changes, then maybe a single point read is all we need to do since we will likely be reading at a much slower rate than the device is sending it.

The caveat here is that if the status changed twice between reads, we won't know it from the single point read, since that only returns the most recent value.  Additionally we may have to look at the timestamp data to ensure that we are still getting new values for the frame.  If the device sending the frame turns off, or stops sending it, any subsequent reads will return the newest values received for a frame.  For this session type you must specify all frames you expect to read at the start.  And if a frame hasn't been seen yet, the read will return the default value for the frame.

Here we see that the first read returns data for both IDs 0xC and 0xE even though 0xE hasn't been written to the bus yet.  Indicating this is the fact that the timestamp is zero.  The second read will again return the last value of 0xC and 0xE but in this case 0xE has changed twice but looking at this data we can't know this.

Frame In Queued

When creating a Frame In Queued session, a frame must be specified from the database to monitor, and can't just be done on the whole bus, like Frame In Stream, or on an array of frames like Frame In Single Point.  So if your software wants to perform a read, and return all frames from one specific ID then you can use the Queued session type.  This is most useful when a particular ID contains information that should be evaluated on every frame, but other IDs aren't as important.  One place this is useful is for when an incrementing counter is used.  Maybe a device will increment a number with every frame sent.  For this one ID we may want to read all frames to ensure the number does increment, but for all other IDs maybe the Single-Point read is more appropriate.  Since XNet allows us to open multiple sessions our application might work best with one Frame In Queued session, and one Frame In Single Point session.  Below is another NI illustration.  Note that in this case two sessions are opened, one for the 0xC and one for the 0xE IDs, with two separate reads, one for 0xC and one for 0xE

Signal In Single-Point

The Signal In Single-Point session is useful when you want to read a signal, but not a frame.  This read will not return the frame but instead will return the engineering units as a double, that have been scaled from the raw frame using the database as mentioned in Part 5.

This session type share many of the similarities and caveats as the Frame Single-Point mentioned earlier.  The Single-Point means a read will return the most recent value for each signal and multiple changes occurring between reads won't be detected, but unlike the Frame read the Signal read does not return a timestamp, so there is no way of knowing if the device is still transmitting new data or not using this session type alone.  All signals desired to be read must be specified on the Create Session.  And if a Signal hasn't been seen yet the default value for the signal will be returned when a read is performed.

Signal In XY

With Signal XY we read the engineering units of a signal, but this time we can read it for every value that the port sees.  This will also mean that if a signal hasn't been seen, it will return an empty array for that index.  Just like the other Signal session types, all signals intended to be read must be specified in the Create Session before performing the read.  Since frames come in at different rates, the times on each signal will correspond to when that signal's frame was read.  As a result all signals from the same frame will have the same number of samples, and same times for each sample.

Signal Waveform

This session type is like combining the Single-Point and the XY.  All samples for all signals specified will be returned.  But all signals read will have the same number of samples, the same start time, and the same delta X time between samples.  And the value for each signal at each time, will be as if a Single-Point reading was taken at that time.  Just like a Single-Point read there is no way to know if a signal is still being updated, or the signal value hasn't changed, unless the value changed between indices of the waveform.

The Other 6 Session Types

The last 6 Session types are similar to the first 6 except they are for writing signals and frames, instead of reading them.  For the most part they work the way you'd expect.  Single-Point sets a Signal or Frame, and will automatically send out the value periodically, until a new write is called at which point the value will be updated but still sent out regularly.  With the rate of transmission set by the database.  Signal Out XY can write a new value with each periodic write of the signal, with timing defined by the database.  Signal Out Waveform can update all signals with each change in time, but the periodic rate is still set by the database.  Frame Out Stream works like all other CAN APIs and will write all frames as fast as it can to the bus writing it once.  And the last one I want to spend a bit more time on later and that is the Frame Out Queued which makes a queue where each transmission sends a new frame, at the rate specified by the database.

Lots of Session Types I Know

There's lots of session types.  And at first this can seem a bit overwhelming.  But the good news is you can open multiple session types to each interface, so you can combine the strengths of one type, with the weaknesses of another.  Maybe you want Signal In Single-Point for reading the majority of the signals, but a Frame In Stream to log all raw frames, Frame Out Single-Point for most frames, and a Frame Out Queued for a couple special ones.  All of these can be open and running at once to make for some pretty flexible ways of sending data and receiving data.  The goal of all of this is to help offload the resources from the software you write, to the XNet hardware.  Sure a single Frame In, and Frame Out session can do all you need if you want to perform software timing on re-transmits, and software scaling from frames to signals, but a more efficient way is to have the hardware deal with timing and conversions.


So I've mentioned a few times how databases help with the timing of transmitted data.  In Part 5 we talked about the frame and signal conversion, and how signals are data embedded in the 8 byte payload of a CAN frame.  With a database in place, we can define how the conversion takes place.  This is somewhat like how DAQmx has virtual channels which can be saved which define how to read and scale raw voltage into a signal with engineering units.  If a DBC file is provided, the simplest way to use it is to import it into the XNet Database editor.  If one isn't provided, then this database can create be created from scratch using the XNet Database editor which is pretty useful at visualizing the frames and how the data gets converted.  But what if I don't have a DBC, and the database needs to be created based on some other document specifications like a custom spreadsheet?  How do I get and use this scaling information?  Well the XNet API can do all those things for you, but it has a bit of a learning curve.

Creating a database manually can be done using the XNet Database Editor pretty easily.  Doing it programatically is not as easy but luckily there are several examples included.  Open up the Editing Database with IO Control.vi example from the Help >> Find Examples.

Database Levels

The database consists of several levels.  The first level is the file or database.  A database doesn't need to be stored as a file and can be just a database in memory but all changes are lost if that database isn't saved.

Within the database you have between 0 and N clusters.  A cluster is for a particular protocol, but you can have multiple clusters with the same protocol.  This can be useful if an application has multiple CAN buses that normally use different DBC files.  Each file can be associated with a cluster and the result is a single database file that defines the application.

Within the cluster there is the frame, which we covered in Part 2 of this series.  And within a normal frame we can have between 0 and 64 signals which were covered in Part 5 of this series.  Using the example mentioned earlier we have the ability to make Databases, Clusters, Frames, and Signals, as well as remove Signals, Frames, and Clusters from an existing database.

Through looking at the code that edits the Database you'll see the functions needed are a few polymorphic VIs like Create Object and Delete Object, and you'll see lots of property nodes.  These are all found on the Database palette of XNet.

When dealing with remote targets like an embedded cRIO or cDAQ, some functionality is missing from the normal Windows experience.  Things like importing and exporting a database aren't supported.  But instead what can be done is a database can be imported on the host OS, then deployed using the IP address of the remote target.  The rest of the editing of databases on the RT target seem to work as expected.  If you want to be able to perform an import of a DBC from an RT target feel free to kudo my idea on the Idea Exchange board.

If you've made it this far into this blog series I commend you.  XNet itself is a very large topic, and one that a multi-week class could be taught on.  But moving forward we will be done talking about all the various NI and non-NI APIs for talking to hardware.

Part 7...

In the previous parts of this blog we talked about hardware options, software options for NI and non-NI hardware, reading and writing CAN Frames, as well as database management and scaling with CAN Signals.  In Part 7 we will be talking about how to read and write CAN data to various file standards.  As well as look at some other CAN File I/O functions.

All Posts