Category Archives: Methods & Tips

Workflows, problems, solutions.

Decoding Serial Busses from Digital Oscilloscope

Introduction

 Using a Digital Oscilloscope was a premium tool in the past; although professional equipment is still a high cost if considered for amateur use, the cost of equipment has fallen the last years allowing more people to get these instruments.

 

In today’s world of microcontrollers and IoT, it is not uncommon to debug serial low-speed buses, like RS232, SPI and TWI (I2C). Hobbyists and professionals need to decode manually each bit knowing the protocol. This is natural to do if you are at the first debugging stages where you need to check electrical integrity on a new design, but it gets tedious if you need to concentrate on the protocol. Purchasing a decoding module for these oscilloscopes increase the cost and many times is not an option for a one off project.

 

The good thing with such instruments is that they usually provide some connectivity with serial, ethernet or USB connection. Usually apart from any web based control you have the option to program the instrument through the SCPI interface. You may use any language you want if you want to access the instrument as long as you have installed the relevant VISA libraries/drivers like National Instrument’s NIVISA, or the Tektronix TekVISA (see below for links).

 

For those unfamiliar with SCPI (Standard Commands for Programmable Instruments), it is a standard protocol to access instruments initially based on IEEE-488 connection. Later on as more instruments used serial or ethernet connection, the required GPIB card has became obsolete and made things easier and more straight forward. The instructions available are mostly universal, but each instrument can complement these with any extensions needed to support the features unique on it.  

 Examples of SCPI commands:

*IDN?
ACQUIRE:STATE STOP

I started programming many instruments that way in the past using C/C++ but after switching to python I forgot about any other language in the host machine. Scripting languages offer an easy go process especially for string manipulation etc. For this reason I wrote a python script to capture signals from the oscilloscope and decode them.

In summary to access the SCPI port of a connected instrument you need the following:

1. Python Interpreter (code is written in 2.7 generation) (https://www.python.org/downloads/ )

2. TekVISA/NIVISA (or any other corresponding VISA library) package (https://www.ni.com/visa/ , http://uk.tek.com/oscilloscope/tds7054-software )

3. PyVISA module (https://pyvisa.readthedocs.io/en/stable/  )

4. The Capture/decode python script.

Now under Windows you may use the executable which is a compiled python script (with GUI2Exe). In this case you will need only the VISA installation and the tool itself (no need for python+pyvisa as these are included in the compiled program).

 

How it works

 The tool captures the signals from the oscilloscopes in a CSV file. As I use an old TDS3012B DPO from Tektronix, I only have 2 available channels to capture. This is fine for capturing RS232 or TWI signals, but for SPI you may need to perform multiple runs to capture 2 data lines and CS along with the clock, with you ending up changing probes for each run, but this is better from nothing.

Oscilloscopes with more channels would of course capture more signals and avoid this hick-up. The script is made and tested on this particular oscilloscope, but it should work in most other instruments maybe with slight modifications. If you need to capture more channels then you still need to modify the script to save the extra channels on the CSV and in addition to add to the SPI decode module the extra traces to decode. But this is why we have open source right?

Keep in mind that the script decodes from the CSV file (and not directly from the instrument). Thus if you have old (or test) captures you may instruct it to use these instead of accessing an instrument. This helps for off-line decoding or testing the script.

 After the signals are captured, the script performs a thresholding according to the set logic standard like TTL, CMOS, LVCMOS etc.

 Then it decodes the signal according to the requested protocol and displays the decoded output on screen.

To keep the script simple, we setup the oscilloscope manually to capture the signals fully (with the required time-span, sampling etc) and we leave it running. Triggering should be to Normal mode (not Auto). When we need to capture the waveform we can either run the script which stops acquisitions or we can stop manually and still use the script to acquire and decode the signals.

 Let’s see some examples.

RS-232 Decoding

Running the command:


DSOCapture.py -b RS232 -l TTL -a 115200 -i CaptureRS422_LVCMOS.csv -n 0

We decode a file with data already acquired to a file.

We need to define the baud rate, the voltage level and polarity (positive or reversed).

RS232 Capture
RS232 Command Prompt Capture Example

The decoder displays the resulted characters along with the Start-Stop and Parity bits. In addition a VCD file is exported that we can view with GTKWave. 

rs232_wave
RS232 Waveform Capture Example

 

TWI Decoding

In TWI mode the parameters are more simple as TWI is clocked so we do not need to define sampling period for each character.


DSOCapture.py -v -b I2C -i CaptureI2C.csv
twi_dos_capture
TWI Command Prompt Capture Example

Again the results are shown in the Command Prompt, showing Start-ACK-Stop conditions.
In addition the VCD file provides a graphical view.

twi_wave01
TWI Waveform Capture Example

 

twi_wave02
TWI Waveform Capture Example Detail

 

SPI Decoding

 
In this example we acquire the signal from the instrument (which is the more common way to do it). Every acquisition will store the .csv file with the data, if you need it. However you will need to copy it since the next acquisition will overwrite it.

DSOCapture.py -v -b SPI -t 3 -visa -i TCPIP::192.168.2.226::INSTR -f capture.csv
spi_dos_capture
SPI Command Prompt Capture Example

 

spi_sd_wave
SPI Waveform Capture Example

 

  

Development Methodology

Now the interesting thing is that I used TDD (Test Driven Development) methods to create this script. To help me in this process I used the pytddmon.py script. So when developing you just need a command prompt or shell in the directory holding the script and run pytddmon.py (should be at the same directory). See picture below. 

PyTDDmon Example
PyTDDmon View

 

Python Test Code
Python Test Code

 

You may download pytddmon from here. Alternatively you may use:

pip install pytddmon

 

Next thing is that I created the test cases for each module. Initially I used real data captured on my oscilloscope or used similar values to test the various cases. For example thresholding the incoming stream is more challenging than just compare and output a logic value. As you pass through the transition phase from 0-1 or 1-0 then some values may trigger multiple transitions on each edge. In this case you need to perform hysteresis ( link ) to avoid it (as the electronics do as well).

 After the signals have been interpreted to logic levels (or edges as well, as TWI uses the edges), then the thresholded outputs are passed to the corresponding serial bus analyzer.

 

Limitations

The script does work well but there are limitations mainly due to the hardware (Digital Oscilloscope) which is worth to mention.

RS232

The problem with RS-232 is that it is an asynchronous transmission. To better state the problem, there is no clock transmitted or derived from the data. It is assumed that the receiver has a similar clock within a certain range to capture and decode the transmission.

If you need to capture a long stream of bytes and the scope does not have big memory you will find yourself limited on what can be captured. As the scope’s memory is not sufficient to hold a longer time period with much detail the decoded data may be scrambled (usually after a few bytes). To overcome this you may need to decrease the time span of the data or use an instrument with sufficient memory capacity to hold accurately the signal transitions.

TWI

This kind of bus uses edge signalling to determine certain protocol states (like Start/Stop conditions). If the sampling of the scope is not sufficient you may get bad decoding of the signals. Although not so severe as RS-232 protocol, this will happen only if you stress the instrument.

SPI

SPI bus is a clocked transmission so sampling is not much an issue as in the other protocols. However because you usually need three lines (SCK, MISO, MOSI), you need extra effort to read all lines with a two channel scope. As this script was built around TDS3012B, it captures only two channels. If you have an instrument with more channels available I would recommend to modify the script and capture the extra channels.

 Summary

Improvements to the script can be performed, but as I am busy doing other things, I currently do not plan to do any additions. In any case any ideas are welcome and at some point may integrate them.

I created this script some time ago, initially for TWI, then I added RS232 and finally the easiest part SPI. I completed the whole script in about a two week time (with full functionality) and it was pretty easy. If I had to work without TDD, I would need to capture all the possible variations on my scope and debug a much larger codebase at the time. 

 You may find a compiled Python executable here

You may find the source tree in BitBucket.

 

To RTOS or not to RTOS?

To RTOS or not? (to RTOS)

 

Summary

Often in the embedded world the question of using a Real-Time Operating System (RTOS) or not, is the big question amongst engineers. The answers found on-line are usually biased opinions without metrics or scientific support of the argument. They usually state the advantages or disadvantages over the classic round-robin systems. The truth is that engineers prefer and like evidence instead of heuristics. I will try to answer this, as I did for myself. I believe this small guide will help decide if an RTOS is worth the effort or not. 

Task Schedule
Example of Task Schedule in Pre-Emptive System

 

Many times the same question has been asked by embedded systems engineers, regarding whether they should use an RTOS or not.

 

Requirements of Embedded Systems

The embedded systems are often called Real-Time Systems and the terms are used interchangeably although this may not always be the case. Ie. There are embedded systems that may not be real-time. Think for example of a thermometer. There is an embedded processor, but failing to measure the temperature in time does not have an impact. 

One misconception is that Real-Time means very fast. This is a misleading interpretation. Real-Time means deterministic. For example if an event happens, our system needs to respond within a time limit. Depending on the application this time-limit will vary. In addition if multiple events are processed and all have the requirement of real-time then every event should be processed according to its own deadline.

As you may see we stress the property of time. Embedded systems may have other restrictions as well like memory footprint etc. but because here the resource we need to analyze and RTOSes handle it, is time.

 

Round-Robin

Round-Robin systems may come in two varieties. Fixed execution, or scheduled.

Round Robin AVRILOS
Example of Round Robin System

Fixed execution is the system which is hard coded. The main loop calls a list of functions (tasks) which each one process their own events.


Void Main(void)
{
   SystemInit();
   For(;;)
   {
     SerialPort();
     ProcessData();
     SetIO();
     SysTick();
   }
}

Obviously the tasks should not block on waiting, but they should rather return. This might add some complexity but in general this is probably the easiest method to build the system.

 

Scheduler

In this case each function is executed on demand. There is a top scheduler which observes which task has an event to process. Each task has also its deadline. Then the scheduler executes the best possible sequence of the previous tasks, according to their priority. This adds more complexity for the advantage of improving schedulability. Each time a task finishes the scheduler runs and decides which will the next task be.

This simple approach has a problem. What if, one of the tasks needs much processing time that upon returning to the main loop or scheduler, another task may have lost its deadline? Maximum response time in this case is the worst execution (till CPU release) of the set of all tasks. On average you may get better response time than Round-Robin systems but the average value cannot be used for real-time systems.

 

What is RTOS?

RTOS tries to solve the schedulability problem by pre-empting the tasks. This means that each task thinks that runs alone into the system without interruption. The scheduler will pre-empt each task according to some rules and will return back at the same point to continue when all the higher priority events have been processed. Although it might seem too complex, it is not so difficult to be done. Looking at the code of some RTOSes you will understand the logic behind it. Beware thought that many RTOSes are not code friendly. Ie. The code is not well written in respect to reading. For me this is what I consider one of the parameters of choosing an RTOS. Even if the RTOS is not perfect, if you have the source and it is readable you can fix or improve things. If the code is unreadable, then it is far more difficult to change it.

In fact even Round-Robin systems do pre-emptions! Can you guess it? The interrupts. The interrupts do exactly the same; However these are considered hardware priority (which is true) and limited in capacity or numbers. Thus actually RTOSes add one more layer of software pre-emption.

 

Comparison

So why not everyone use an RTOS? There are many reasons.

Example of RTOS Driver
RTOS COFILOS

 

First there is complexity. Using an RTOS may require additional code (like protection mutexes, semaphores, messages etc), that were not needed before.

Second the question of which RTOS to choose is not simple. Some are free, some need licenses and are expensive; others have a large footprint, or no do not provide source code.

Third are resources. What if your memory requirements need a very low footprint system or time constrains prohibit such systems? Maybe an RTOS cannot fit?

And the list goes on.

 

Solution?

And that’s how the question described in the beginning starts. Seems that there is no specific engineering parameter that would pin-point if we really need an RTOS or not. But let’s go back to the principles of decision. What all systems try to do? Share the CPU time resource. Is RTOS better in schedulability in respect to the other non pre-emptive systems?

Actually there is a very good principle that helps us in general with schedulability. This is called Rate-Monotonic Approach (RMA). This method analyzes a system to check if it is possible to schedule its tasks. The inputs are various parameters like period of events, sporadic events, deadlines, etc that help derive mathematically if the system is schedulable. This approach works with fixed-priority schemes and with either pre-emptive or non pre-emptive systems.

Thus the methodology would be to estimate each tasks worst execution time, gather all the deadlines, fill in the matrices and get a result if the specific system is schedulable. Analyzing the round-robin systems first you get the idea if this will work or you stress the system.

RMA proves that a pre-emptive system is better. Thus if you round-robin systems fails to be schedulable, you should try the RTOS. Of course you have to add the context switching time and any other overheads. If the system is schedulable (with a safe margin), then using an RTOS is the solution with the given hardware. It might be the case that neither solution works. In this case you need probably to upgrade the hardware.

Fixed priority is not the best scheduling method, but it is predictable. Earliest Dead Line (EDL) priorities are better, but RMA cannot define how much better. Thus an EDL system will work if the fixed scheduled system is also schedulable.

 Of course there other parameters to consider in this case like costs, memory footprints etc, but the fact is that if you need an RTOS, the question moves from “To RTOS or not to RTOS?” to “Which RTOS?”  which is a whole new story.

 

Examples and Experience

During my carrier I rarely needed an RTOS. Classic round-robin systems would fit the bill very well. And that’s how AVRILOS came along over the years started from 8051 in the ‘90s and then ported in assembly to AVR in 2000. Then after a couple of years AVRILOS was re-written in C for AVR.

 

A case I should have used an RTOS

Looking back, I could see a project where an RTOS was necessary and I failed to see that at that point. The project was a cash register machine. I wrote the OS, which performed dynamic scheduling with Earliest Dead Line first priority but without pre-emption. The main application which we wrote with my colleague had to be split in sub-states and return to the main loop at the 2mS slot interval allocated for every task. This got us a tedious development. The funny thing was that we had implemented a one task pre-emption in case a task was missing its deadline. This was of course a safeguard and was not really used. But the pre-emption mechanism was there. Things got worse when I had to do the 10.4 digits BCD division which itself was longer than 2mS on our Z80 core. The full system could work skipping all these states/substates coding part from our side, if we had chosen to use an RTOS (or build our own). Although we had memory limitations, the extra code to support the splitting of the main application probably would be covered by the RTOS itself with a cleaner implementation.

 

A case that an RTOS would not fit

In another instance we were building the MAC layer of 802.11abg with smart antennas. Our core was an ARM9 with tightly coupled memory (TCM) running at 80MHz. The external bus was running at half the frequency as the connected logic (FPGA) could not go faster. At that point I was wondering if using an RTOS would be beneficial. The system had to respond in 2uS from an event, which means it had to process the received packet and prepare a response. Utilizing some system pipelining we could extend to 4-6uS. When I tried an RTOS to see the overheads, I realized that the penalty was about 2uS. The time cost was way too much, so I used the classic round robin approach which was matching the system needs pretty well.

So you may see from the above examples that the decision to use or not an RTOS was irrelevant of the system complexity or the execution speed, but rather a matter of schedulability.

 

Conclusion

The question to place an RTOS or not can be greatly answered depending on schedulability. If the system can be scheduled without an RTOS safely then you do not need an RTOS. If not, then RTOS is the way to go. Of course there can be other reasons for the decision, like future expansion, ready stacks to use etc. but these goes beyond the basic principles of decision. You may use the RMA method to provide the criteria for your decision.

 

References

[1] Meeting Deadlines in Hard Real-Time Systems

The Rate Monotonic Approach, by Loic P. Briand and Daniel M. Roy, ISBN 0-8186-7406-7,

IEEE Computer Society

 

[2] AVRILOS

AVRILOS: A simple OS for AVR microcontrollers

https://www.codeproject.com/articles/127095/avrilos-a-simple-os-for-avr-microcontrollers