Debugging an embedded application can be a frustrating task. There aren’t many experiences in life more humbling than spending an entire week trying to get an analog-to-digital converter configured properly. Unlike web and mobile app development, embedded developers generally have a limited set of resources at their disposal. When you’re first getting started, it’s easy to feel lost and hopeless when you can’t get things working the way you want them to.
Over the years, I’ve debugged many problems with embedded systems and helped others debug problems as well. Here’s a list of my top 10 go-to techniques for squashing those pesky bugs!
1. Toggle a pin.
One of the simplest forms of debugging an embedded application is to use a GPIO pin to verify that a certain line of code in your application is being hit – similar to using a breakpoint. You simply initialize the pin low when your app starts and then set it high, or pulse it, when the code in question runs. You can use an oscilloscope or connect an LED to the pin to see when it transitions.
This approach can also be useful if you want to easily see how frequently a certain event is happening. If it’s an infrequent event, you might be able to verify it visually. Otherwise, use your oscilloscope to measure the frequency.
2. Enable serial output.
Toggling a pin is useful for very simple debugging, but once your application gets more complex, it’s often helpful to see more information. The UART peripheral — available on just about every micro these days — can be your best friend for live application logging.
To use this technique, create a helper function that takes a C string and transmits the characters out through the UART. Next, use your function to log key events that happen while your application is running. Finally, get yourself a USB-to-UART cable, connect it to your computer, and open up the port. You’ll be able to see exactly what’s happening and in what order.
Keep in mind that transferring data over UART can be relatively slow. Printing long messages may take a non-insignificant amount of time, impacting the timing of other events.
3. Consult the docs.
A friend of mine used to say, “The first rule of embedded programming is: read the datasheet. The second rule of embedded programming is: read the datasheet. The third rule of embedded programming is: Don’t trust the datasheet!” The point is that the datasheet usually contains the information you need to know to properly configure your microcontroller, but they’ve also been known to contain a bug or two!
Also, keep in mind that the datasheet is not the only document worth consulting. Typically, chip vendors provide piles of application notes, example projects, etc., to help make it easier for people to use their products.
4. Go back to Hello World.
When applications get really complex, sometimes one part of the application may negatively impact another part without you realizing it. For example, one peripheral or clock configuration may have an unexpected effect on another peripheral’s behavior. This can cause you to spend a lot of time trying to debug the wrong code.
One technique to counteract this is to isolate the code you’re trying to get working. Create a brand new empty project, verify something very basic works, then bring in just the code that you’re struggling with. This sanity check can help you ensure that you’re focusing your efforts in the right place.
5. Try the process of elimination.
Similar to the technique above, the process of elimination technique can help you resolve issues where one part of your application unexpectedly impacts another part. When this happens, start commenting out or removing large swaths of functionality until the problem goes away. This will help you determine the interdependent code.
6. Consult the community.
Online technical communities can be a great resource to help you resolve tricky problems. Many chip manufactures now have forums on their websites dedicated specifically to their products. Often these forums are even monitored by engineers who work for those companies and have the expertise to help.
Pro tip: Don’t wait until you’ve exhausted every other technique to post your question to the community. If you do it earlier in the process, you might be able to save yourself a lot of time.
7. Use a logic analyzer.
If these techniques were ordered by the most to least useful, this one would be first. A logic analyzer can be worth its weight in gold when it comes to debugging embedded applications. You could spend hours trying to capture the information on an oscilloscope that you could capture with a logic analyzer in two minutes.
If you’ve never used one, a logic analyzer can record waveforms from multiple signals at the same time for long periods of time. That can be tens of seconds if you want. They can also decode the data that is reordered from a number of standard interfaces.
If you’re struggling with getting the communication between two chips right, go buy a logic analyzer today. A company called Saleae makes a great one that is reasonably priced. You’ll thank me afterward.
8. Consider your environment.
Sometimes the problem you’re experiencing can be caused by outside factors you would have never thought possible.
One time, a colleague and I were trying to track down what was causing our microcontroller to randomly reset. Was it a stack overflow? Was it a bus error? Nope. It was a chair error!
You’ve never heard of a chair error? It’s when someone stands up from their chair, and, for some reason, that creates electromagnetic interference that is picked up by the cable connecting the debugger to your board. The oddest thing was that it was just one specific chair that caused the problem.
Another time, we were working on a wearable electronics project that required extremely low power consumption. After spending days trying to figure out why the board would consume more power at some times than at other times, we discovered the culprit. It was the sun. The board had an LED on it, and when the sun would shine directly on the LED, it acted as a photodiode and changed the power consumption ever so slightly.
The point is, when you’re debugging problems with embedded systems, you really have to think outside the box. Sometimes the problem isn’t in the code!
9. Try another board.
I’m trying to keep this post mainly out of the hardware engineering weeds because that’s a whole separate category of embedded problem-solving. That said, even if the PCB design is fine, you may still have hardware-related problems with your microcontroller. For example, I’ve seen individual pins completely stop working. Inside the chip, tiny little wires connect the external pins to the silicon. If the chip gets hit with static, those wires can fry, leaving you with a dead pin. If you have another physical board handy, try swapping out your hardware for a new board.
10. Leverage code review.
Finally, don’t overlook the value of a good code review. As the author of the code, it can be really easy to overlook a function that you feel overly confident about. Sometimes a fresh set of eyes is all you need.
I used to joke that problems that would normally only occur once in a million times occur much more frequently in the embedded world because someone probably tried to use an 8-bit integer to count up to a million!
Those are my top 10 go-to techniques for debugging embedded applications. What techniques have you found helpful?
Hi Jordan,
One other technique that is often useful (it is implied in your list but is worth specifying explicitly) is to go back to an older version that is working, then use a repeated binary chop technique to reduce the number of differences and keep testing (after each chop) to see when the problem reappears.
Best regards,
Tony
Thanks Tony. Great callout! This technique is super helpful.