Return to Developer Blog

Developer Blog

27/02/2017 by Magnus Carlsson

Special converter conditions

KVR_DevBlog_HorizLogo_Final_Black

This is the final post in a 3-part series about using the Converter Library (kvlclib) in CANlib SDK:

  1. Writer formats and properties (1 of 3)
  2. Converting to plain ASCII (2 of 3)
  3. Special converter conditions (3 of 3)

In the second part of this blog series, we created a simple program to convert a kme50 log file to plain text. We will now continue and look at the three special conditions that can arise when converting a log file; overrun, data truncation, and change in output filename:

Overrun
can occur during logging with a Memorator if the bus load exceeds the logging capacity. This condition can be detected via our converter by calling isOverrunActive(). Once the overrun has been noted, we need to manually reset the state in our converter using resetOverrunActive().

Data truncation
occurs when the selected output converter can’t write all bytes in a data frame to the output file. This can happen if CAN FD data is extracted to a format that only supports up to 8 data bytes, e.g. KME40. This condition is reset by a call to resetStatusTruncated().

New output filename
is a change in the output filename that occurs when the converter creates a new file. This can e.g. be the result of splitting output files (which we will look into briefly later).1 The call to isOutputFilenameNew() will return True when the last converted event resulted in the creation of a new output file. This condition will automatically reset during the next call to convertEvent(). We can get the name of the newly created output file by calling getOutputFilename().

Let us now create a function that, beside of doing the actual conversion, also looks for the above conditions and prints some informational messages to the user.

def convertEvents(cnv):
    # Get estimated number of remaining events in the input file. This
    # can be useful for displaying progress during conversion.
    total = cnv.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF
            # is reached
            cnv.convertEvent()
            if cnv.isOutputFilenameNew():
                print('New output filename:', cnv.getOutputFilename())
                print('About %d events left...' % cnv.eventCount())
        except kvlclib.KvlcEndOfFile:
            if cnv.isOverrunActive():
                print('NOTE! The extracted data contained overrun.')
                cnv.resetOverrunActive()
            if cnv.isDataTruncated():
                print('NOTE! The extracted data was truncated.')
                cnv.resetStatusTruncated()
            break

Extending the converter program we had in our previous article with this new convertEvents function we now end up with the following.

#06_convert_kme_check_condition.py
from canlib import kvlclib


INPUT_FILE = 'mylog.kme50'


def trySetProperty(cnv, property, value=None):
    # Check if the format supports the given property
    if cnv.format.isPropertySupported(property):
        # If a value is specified, set the property to this value
        if value is not None:
            cnv.setProperty(property, value)

        # Get the property's default value
        default = cnv.format.getPropertyDefault(property)
        print('  %s is supported (Default: %s)' %
              (property, default))

        # Get the property's current value
        value = cnv.getProperty(property)
        print('    Current value: %s' % value)
    else:
        print('  %s is not supported' % property)


def convertEvents(cnv):
    # Get estimated number of remaining events in the input file. This
    # can be useful for displaying progress during conversion.
    total = cnv.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF
            # is reached
            cnv.convertEvent()
            if cnv.isOutputFilenameNew():
                print('New output filename:', cnv.getOutputFilename())
                print('About %d events left...' % cnv.eventCount())
        except kvlclib.KvlcEndOfFile:
            if cnv.isOverrunActive():
                print('NOTE! The extracted data contained overrun.')
                cnv.resetOverrunActive()
            if cnv.isDataTruncated():
                print('NOTE! The extracted data was truncated.')
                cnv.resetStatusTruncated()
            break


# set up formats
out_fmt = kvlclib.WriterFormat(kvlclib.FileFormat.PLAIN_ASC)
in_fmt = kvlclib.ReaderFormat(kvlclib.FileFormat.KME50)

# set resulting output file name taking advantage of the extension
# defined in the format.
out_file = "myresult." + out_fmt.extension
print("Output filename is '%s'" % out_file)

# create converter
cnv = kvlclib.Converter(out_file, out_fmt)

# Set input file and format
cnv.setInputFile(INPUT_FILE, kvlclib.FileFormat.KME50)

# allow output file to overwrite existing files
trySetProperty(cnv, kvlclib.Property.OVERWRITE, 1)

# we are only interested in the first channel
cnv.setProperty(kvlclib.Property.CHANNEL_MASK, 1)

# add nice header to the output file
trySetProperty(cnv, kvlclib.Property.WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize
# the width of the data output to 8 bytes
trySetProperty(cnv, kvlclib.Property.LIMIT_DATA_BYTES, 8)

convertEvents(cnv)

# force flush result to disk
cnv.flush()

Running this program on a specially prepared kme50 file we now get the following output:

Output format is 'Plain text'
Output filename is 'myresult.txt'
Input filename is 'mylog.kme50'
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
Converting about 71253 events...
New output filename: myresult.txt
About 71252 events left to convert...
NOTE! The extracted data contained overrun.

As can be seen in the output, our input file contained over 7000 events and as a result the output file created was over 2MB. We can also see that the specially prepared kme50 file contained overruns. Let us now split the output files by also setting the property SIZE_LIMIT.

# split output files into max 1 MB files
trySetProperty(cnv, kvlclib.Property.SIZE_LIMIT, 1)

#07_convert_kme_check_split.py
from canlib import kvlclib


INPUT_FILE = 'mylog.kme50'


def trySetProperty(cnv, property, value=None):
    # Check if the format supports the given property
    if cnv.format.isPropertySupported(property):
        # If a value is specified, set the property to this value
        if value is not None:
            cnv.setProperty(property, value)

        # Get the property's default value
        default = cnv.format.getPropertyDefault(property)
        print('  %s is supported (Default: %s)' %
              (property, default))

        # Get the property's current value
        value = cnv.getProperty(property)
        print('    Current value: %s' % value)
    else:
        print('  %s is not supported' % property)


def convertEvents(cnv):
    # Get estimated number of remaining events in the input file. This
    # can be useful for displaying progress during conversion.
    total = cnv.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF
            # is reached
            cnv.convertEvent()
            if cnv.isOutputFilenameNew():
                print('New output filename:', cnv.getOutputFilename())
                print('About %d events left...' % cnv.eventCount())
        except kvlclib.KvlcEndOfFile:
            if cnv.isOverrunActive():
                print('NOTE! The extracted data contained overrun.')
                cnv.resetOverrunActive()
            if cnv.isDataTruncated():
                print('NOTE! The extracted data was truncated.')
                cnv.resetStatusTruncated()
            break


# set up formats
out_fmt = kvlclib.WriterFormat(kvlclib.FileFormat.PLAIN_ASC)
in_fmt = kvlclib.ReaderFormat(kvlclib.FileFormat.KME50)

# set resulting output file name taking advantage of the extension
# defined in the format.
out_file = "myresult." + out_fmt.extension
print("Output filename is '%s'" % out_file)

# create converter
cnv = kvlclib.Converter(out_file, out_fmt)

# Set input file and format
cnv.setInputFile(INPUT_FILE, kvlclib.FileFormat.KME50)

# split output files into max 1 MB files
trySetProperty(cnv, kvlclib.Property.SIZE_LIMIT, 1)

# allow output file to overwrite existing files
trySetProperty(cnv, kvlclib.Property.OVERWRITE, 1)

# we are only interested in the first channel
cnv.setProperty(kvlclib.Property.CHANNEL_MASK, 1)

# add nice header to the output file
trySetProperty(cnv, kvlclib.Property.WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize
# the width of the data output to 8 bytes
trySetProperty(cnv, kvlclib.Property.LIMIT_DATA_BYTES, 8)

convertEvents(cnv)

# force flush result to disk
cnv.flush()

The output from our final run is shown below, we now got three output files, where the first two are about 1MB.

Output format is 'Plain text'
Output filename is 'myresult.txt'
Input filename is 'mylog.kme50'
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
  PROPERTY_SIZE_LIMIT is supported (Default: 0)
    Current value: 2
Converting about 71253 events...
New output filename: myresult-part0.txt
About 71252 events left to convert...
New output filename: myresult-part1.txt
About 41473 events left to convert...
New output filename: myresult-part2.txt
About 11695 events left to convert...
NOTE! The extracted data contained overrun.

This was all for this time, if you have any questions, comments or suggestion for future blog articles, please contact us directly at support@kvaser.com.

Footnotes

1 Splitting of output files is controlled by the writer properties KVLC_PROPERTY_SIZE_LIMIT and KVLC_PROPERTY_TIME_LIMIT.


This article has been updated. To view the original, click on the box below.

Original Article

KVR_DevBlog_HorizLogo_Final_Black

This blog uses now deprecated functions, see blog “Improved API in Python canlib v1.5” for more information.

This is the final post in a 3-part series about using the Converter Library (kvlclib) in CANlib SDK:

  1. Writer formats and properties (1 of 3)
  2. Converting to plain ASCII (2 of 3)
  3. Special converter conditions (3 of 3)

In the second part of this blog series, we created a simple program to convert a kme50 log file to plain text. We will now continue and look at the three special conditions that can arise when converting a log file; overrun, data truncation, and change in output filename:

Overrun can occur during logging with a Memorator if the bus load exceeds the logging capacity. This condition can be detected by calling kvlcIsOverrunActive(). Once the overrun has been noted, we need to manually reset the state using kvlcResetOverrunActive().

Data truncation occurs when the selected output converter can’t write all bytes in a data frame to the output file. This can happen if CAN FD data is extracted to a format that only supports up to 8 data bytes, e.g. KVLC_FILE_FORMAT_KME40. This condition is reset by a call to kvlcResetStatusTruncated().

New output filename is a change in the output filename occurs when the converter has created a new file, which can be the result when splitting output files (which we will look into briefly later).1 The call to kvlcIsOutputFilenameNew() will return true when the last converted event resulted in the creation of a new output file. This condition will automatically reset during the next call to kvlcConvertEvent(). When a split has occurred, we can get the new output filename by calling kvlcGetOutputFilename().

Let us now create a function that, beside of doing the actual conversion, also looks for the above conditions and prints some informational messages to the user.

def convertEvents(kc):
    # Get estimated number of remaining events in the input file. This can be
    # useful for displaying progress during conversion.
    total = kc.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF is reached
            kc.convertEvent()
            if kc.IsOutputFilenameNew():
                print("New output filename: %s" % kc.getOutputFilename())
                print("About %d events left to convert..." % kc.eventCount())
        except kvlc.KvlcEndOfFile:
            if kc.IsOverrunActive():
                print("NOTE! The extracted data contained overrun.")
                kc.resetOverrunActive()
            if kc.IsDataTruncated():
                print("NOTE! The extracted data was truncated.")
                kc.resetStatusTruncated()
            break

Listing 8: Defining a function that does the conversion and also check for special conditions.

import canlib.kvlclib as kvlc


def trySetProperty(converter, property, value=None):
    # Check if the format supports the given property
    if converter.format.isPropertySupported(property):

        # If a value was specified, set the property to this value
        if value is not None:
            converter.setProperty(property, value)

        # get the property's default value
        default = converter.getPropertyDefault(property)
        print(" PROPERTY_%s is supported (Default: %s)" %
              (property['name'], default))

        # get the property's current value
        value = converter.getProperty(property)
        print("	Current value: %s" % value)
    else:
        print(" PROPERTY %s is not supported" %
              (property['name']))


def convertEvents(kc):
    # Get estimated number of remaining events in the input file. This can be
    # useful for displaying progress during conversion.
    total = kc.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF is reached
            kc.convertEvent()
            if kc.IsOutputFilenameNew():
                print("New output filename: %s" % kc.getOutputFilename())
                print("About %d events left to convert..." % kc.eventCount())
        except kvlc.KvlcEndOfFile:
            if kc.IsOverrunActive():
                print("NOTE! The extracted data contained overrun.")
                kc.resetOverrunActive()
            if kc.IsDataTruncated():
                print("NOTE! The extracted data was truncated.")
                kc.resetStatusTruncated()
            break


# set output format
fmt = kvlc.WriterFormat(kvlc.FILE_FORMAT_PLAIN_ASC)
# the name of the formatter is fetched using kvlcGetWriterName() internally
print("Output format is '%s'" % fmt.name)

# set resulting output filename taking advantage of the extension defined in
# the format. (Uses kvlcGetWriterExtension() under the hood.)
outfile = "myresult." + fmt.extension
print("Output filename is '%s'" % outfile)

# create converter
kc = kvlc.Kvlclib(outfile, fmt)

# Set input filename and format
inputfile = "mylog.kme50"
print("Input filename is '%s'" % inputfile)
kc.setInputFile(inputfile, file_format=kvlc.FILE_FORMAT_KME50)

# allow output file to overwrite existing files
trySetProperty(kc, kvlc.PROPERTY_OVERWRITE, 1)

# add nice header to the output file
trySetProperty(kc, kvlc.PROPERTY_WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize the width
# of the data output to 8 bytes
trySetProperty(kc, kvlc.PROPERTY_LIMIT_DATA_BYTES, 8)

convertEvents(kc)

# Delete converter, flush result to disk and free memory
kc.deleteConverter()

Listing 9: Simple program to convert a kme50 log file to plain text output.

Running the program in Listing 9 now gives the following output:

Output format is 'Plain text'
Output filename is 'myresult.txt'
Input filename is 'mylog.kme50'
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
Converting about 71253 events...
New output filename: myresult.txt
About 71252 events left to convert...
NOTE! The extracted data contained overrun.

As can be seen in the output, our input file contained over 7000 events and as a result the output file created was over 4MB. Let us now split the output files by also setting the property KVLC_PROPERTY_SIZE_LIMIT.

# split output files into max 2 MB files
trySetProperty(kc, kvlc.PROPERTY_SIZE_LIMIT, 2)

The output from our final run is shown below, we now got three output files, where the first two are about 2MB.

Output format is ’Plain text’
Output filename is ’myresult.txt’
Input filename is ’mylog.kme50’
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
  PROPERTY_SIZE_LIMIT is supported (Default: 0)
    Current value: 2
Converting about 71253 events...
New output filename: myresult-part0.txt
About 71252 events left to convert...
New output filename: myresult-part1.txt
About 41473 events left to convert...
New output filename: myresult-part2.txt
About 11695 events left to convert...
NOTE! The extracted data contained overrun.

This was all for this time, if you have any questions, comments or suggestion for future blog articles, please contact us directly at support@kvaser.com.

Footnotes

1 Splitting of output files is controlled by the writer properties KVLC_PROPERTY_SIZE_LIMIT and KVLC_PROPERTY_TIME_LIMIT.

Author Image

Magnus Carlsson

Magnus Carlsson is a Software Developer for Kvaser AB and has developed firmware and software for Kvaser products since 2007. He has also written a number of articles for Kvaser’s Developer Blog dealing with the popular Python language.

RSS Copy Link

Get The Latest news feeds from our RSS