You know when you learn a new word and then suddenly your colleague uses it in an email? Then it pops up in a book you’re re-reading (although you swear that word wasn’t in there the first read-through), and you hear the host of your favorite podcast say it in the latest episode. That word is now everywhere. You freak out and start to Google things like “is The Secret real” and “what did Matilda have and do I have it” and “free, quick psychic test.”

What you’re experiencing is the frequency illusion aka the Baader-Meinhof Phenomenon, which is a term you’ll probably start seeing everywhere now. When you learn something new, it’s at the front of your mind, so you start to notice it even if it was right under your nose many times before.

As a firmware engineer at Samsara, I fall victim to the Baader-Meinhof Phenomenon a lot. Samsara is where I first learned about refrigerated trailers (also known as reefers, Fig. 1) and how critical they are to the logistical backbone that empowers our economy.

Fig. 1: Refrigerated trailer also known as a “reefer”

Reefers are used to transport temperature-sensitive goods and can be found working around the clock. Products like fresh produce, dairy, and even some medicines are transported by reefers, making them an integral part of our day-to-day lives. I am now acutely aware of the fact that the world is crawling with reefers. Pre-Samsara, I never noticed reefers. Now I find myself whipping my head around on the freeway, checking the make and model of each reefer I pass. I am also confirming whether that reefer has an Asset Gateway (AG) (Fig. 2) from Samsara installed (Fig. 3).

Fig. 2: Asset Gateway (AG) from Samsara

Fig. 3: Reefer with an AG installed

With Samsara, our customers can remotely track, monitor, and control their reefers to make their operations more efficient, safe, and sustainable. Customers can see where their reefers are in order to operate their business effectively. In addition, they can monitor the temperature of their reefers, avoiding costly product spoilage and rejected deliveries. They can also control their reefers, turning the units on and off remotely to save time and fuel.

In this post we are going to explore how Samsara’s small yet powerful Asset Gateways are able to control something not only 75 times their size but way, way cooler (ha). In my experience, the key to a successful AG-reefer solution is, first, to have robust communication between the AG and the reefer. Then, to be completely accurate and honest with the data. And finally, to equip the AG with the tools to handle any curve-balls thrown its way out in the real world.

Speaking the same language

It is obvious that, in order to communicate effectively, it helps if both parties speak the same language. Samsara AGs “speak” CAN communication, which is extremely common among vehicles, including trailers. If the reefer also “speaks” CAN, a simple cable is used to connect the AG and reefer. The AG is then able to send and receive information to and from the reefer, enabling our reefer monitoring and control features.

For example, let’s walk through how a customer might monitor and control reefer temperature. The AG receives a CAN frame (message) from the reefer that contains current temperature information. The AG captures, parses, and sends that information to the Samsara backend and the temperature value is surfaced on the Samsara dashboard for the customer to see. The customer then uses the Samsara dashboard to request a change in temperature. The request is received by the AG, and the AG sends a CAN frame to the reefer. The CAN frame contains a particular payload that commands the reefer to change its temperature (Fig. 4).

Fig. 4: Reefer, AG, and Samsara backend communication

What if the reefer we are tasked with monitoring and controlling does not support CAN? At Samsara we don’t often say “no” in the face of inconvenience or adversity. What do you do when two parties need to communicate but speak different languages? You get an interpreter!

We use intelligent cables to perform this important task. In one particular case, a reefer only communicated to third parties, like our AG, via RS-232. RS-232 is a protocol for serial transmission of data and is something a CAN device cannot understand.

Introducing an RS-232 interface to our AG would have been a costly hardware and firmware undertaking for this specific use case. Instead, we opted to add intelligence to the cable that connects the AG and the reefer. With a microcontroller unit (MCU) added to the cable, we could perform the translation between RS-232 and CAN (Fig. 5).

Fig. 5: Reefer cable MCU translation

• When a new character appears on the RS-232 interface from the reefer, the cable MCU processes it and transmits it right along to our AG in a CAN frame.

• When a CAN frame shows up from the AG, the cable MCU processes and transmits that data contained in the CAN frame to the reefer in RS-232 format.

In the case where the reefer “speaks” RS-232 and the AG “speaks” CAN, communication is possible thanks to the cable MCU performing the needed translation.

Translating honestly

Samsara firmware engineers are tasked with reporting honest data. In the case described above, we inevitably added complexity to our reefer integration by introducing an intelligent cable. During testing we noticed that, extremely sporadically, AG-reported reefer temperature would momentarily spike to some unreasonable number (for example, on one occasion it reached 7200 degrees F), then return to normal operating range. Our role as firmware engineers was to figure out exactly what happened and to make sure it would not happen again.

Working our way from the AG to the reefer we inspected every place where data was exchanged or processed. We did this by breaking out communication wires and using minicom to look at RS-232 traffic and CANables to look at CAN traffic. We verified that data coming out of the reefer was valid and normal (Fig. 6, phase 1 & 2). Data then being translated from RS-232 from the reefer to CAN was done accurately (Fig. 6, phase 3). CAN frames being sent from the cable MCU to the AG were all spot-on and in order (Fig. 6, phase 4). It was the final stretch — CAN frames showing up on the AG CAN bus — that proved to be the issue (Fig. 6, phase 5).

Fig. 6: Reefer data out of order on the AG!

We found that CAN frames were showing up out of order on our AG CAN bus. When those frames were parsed by the AG firmware application we ended up with a very incorrect temperature value.

To make sure data showed up on the CAN bus in the order that it was sent from the reefer, we tacked on a sequence number to each CAN frame sent by the cable MCU to the AG (Fig. 7).

Fig. 7: CAN frames with sequence numbers

In the example described in Fig. 6, the AG firmware application would recognize frames were out of order and would ensure frames were in the correct order before processing the data and reporting information to the Samsara backend.

When it comes to showing customers their data, you simply cannot fake it or expect the customer to ignore any missteps. Our customers rely on Samsara gateways and dashboards to run their businesses. As firmware engineers at Samsara, it is our responsibility to report data and report it honestly. By adding sequence numbers to CAN frames, we increased our confidence in the veracity of our AG-reported data, and us firmware engineers can sleep a bit more soundly at night.

Communicating in the real world

Once a reefer integration is released to customers, we want to make sure the solution is working successfully, which is what we expect after all of the hard work that goes into developing and extensive testing. We do this by monitoring heartbeat data, tracking all the different reefer makes and models AGs are installed on, and making some key pieces in firmware configurable.

Monitoring heartbeats The AG firmware sends periodic heartbeats to the Samsara backend. This heartbeat captures whether the reefer is up and running and if we’re seeing good data that the firmware can successfully parse. If we run into issues — the reefer can’t turn on, frames of data are unexpectedly too long or too short, etc. — we make sure to report this in the heartbeat. We proactively monitor heartbeat data using dashboards to surface issues that are presenting in the field on real-world units. When an issue is identified, it is helpful to know scope — how many units is this issue affecting? How often is this issue occurring? Heartbeat data from the reefer allows us to identify and characterize issues in the field so that we can address them effectively.

Tracking installations There is a surprising amount of subtle variance between different manufacturing years and models of reefers from a firmware perspective. We report as much data about the reefer as possible from the AG firmware. Whatever we can capture — reefer model, serial number, year built, software version, etc. — we report to the Samsara backend. Being able to track which reefers we can successfully integrate with helps us develop a clear picture of how our solution is scaling.

Configurability where you least expect it Releasing an integration out widely exposes surprising exceptions and edge cases, so we have to stay on our toes. We try to avoid hard coding certain things in the AG firmware. For example, the reefer specification might say that it will take a maximum of 2 seconds for the reefer to respond to a request. Then during field testing, you run into a reefer that takes 3 seconds, and that’s just how long the 2003 Model ABC takes. The lesson here is not “you can’t trust spec sheets.” The lesson is to add configurability where possible so that you can remain flexible as you face the realities of operating in the real world.

This may be the first time you have ever heard about refrigerated trailers. Maybe you will start noticing them on the road now, and you will ask yourself, “Have those always been there?”. Yes, they have. These refrigerated trailers can be found on roads all hours of the day and night. They’re transporting many of the goods we enjoy, like ice cream, and sometimes require, like medicine (maybe ice cream falls into this category, too).

Samsara is proud to offer a solution to customers that enables them to operate their refrigerated trailers in a more efficient, safe, and sustainable way, and I am proud to be a part of it along with the rest of the Assets team!

Interested in working on projects like this? Apply to our open positions! We’re always looking for great people to join us as we learn and grow together, and if you love learning and building things in a highly collaborative environment, we’d love to hear from you! 👋