On the challenges of deploying a distributed IoT sensing platform with limited access and in harsh environments
Imagine you design an IoT product, from the smallest set screw thru the schematics of the printed circuit board and all the way to an impressive software stack, and now you have to install it out in the world, but it’s not a smart-watch or a new Bluetooth speaker, its a sensor installed underwater in the ocean, or in the middle of a live and busy highway. Now let’s say you managed to install it there, but it isn’t responding, or you just want to update its firmware that you misconfigured on installation and now you’ve lost track about where it is. Now imagine it’s not a single sensor, but actually 20,000 of them, laid across and about and playing together in a synched orchestra.
The above, and many (many) more, are daily obstacles when developing a distributed IoT sensory platform deployed in hard-to-reach areas, where testing, deploying and maintaining the sensors is sometimes as equally challenging as designing them.
It’s the span that matters
So what is a distributed sensing platform? Basically, as in any distributed system, its a coalition of nodes/devices working together towards a common goal, only now it’s towards a mutual sensing capability such as thermally mapping an object/area, surveillance, presence, theft detection or basically any task that involves multiple, independent sensor units working together to provide a broad yet high-resolution view over a certain area.
Unlike single unit/sensor systems, distributed sensing platforms often require hundreds of sensors, whereas each of the sensors’ readings are also meaningful with respect to their position, state and specific task. Thus, it’s key to know where each sensor is and its’ mode to actually interpret its readings appropriately.
While the general challenges of IoT platforms also hold here, and amongst them the communication protocols which often prove tricky requiring low-energy yet robust protocols, the power management (energy consumption, charging, relatively low capacitance) and environmental conditions, we will focus below on the equally important and often overlooked attributed and design considerations for the deployment and maintenance of such systems.
So why distribute?
In recent years, distributed platforms are becoming more and more common as technology evolves and communication, computing power and energy advance rapidly allowing deployments of decentralizing systems towards scalable, redundant and highly available platforms. Examples range from storage systems, communication platforms and even currency (blockchain). Thus distributed sensing platforms are inevitable to become more and more common in coming years. Amongst the advantages of such system are:
- Span — the ability to uniformly cover large and varying areas/terrains
- Scalability — by nature, these systems are built to scale and could seamlessly scale as much as needed
- Robustness — no single point of failure, allowing for significantly more robust systems
- Redundancy — even in case of a node momentary failure, other nodes act to “cover” for him and keep the correctness and availability of the system
- Accuracy —Overcoming the induced dependence of accuracy on the distance from the sensor in centralized systems (e.g. a cameras sensitivity drops as you move further away) thus increasing overall accuracy and resolution
- Cost — while deploying hundreds of units intuitively sounds a costly solution, often the “loosening” of requirements per node (as 100% per-node availability is not needed) allows for much cheaper solutions, especially with the decreasing costs of IoT hardware in recent years
- New dimensions — reach insights and data granularity often unobservable otherwise, and on that note do read Avi Tel-or’s article IoT — Paradigm Shift in Traffic Monitoring on the how and why now
These are just some of the benefits of deploying distributed IoT sensing systems, so if at any stage reading this article you get intimidated by the challenges, do refer back here and assure yourself the benefits greatly overweight the challenges and are just obstacles on the way to the future of sensing.
Breaking it down, it comes out to 3 main challenges:
- Physically installing the system
- Initializing it (both individually and as a whole)
- Remote (and on premise) maintenance
1. Installation Flow
A critical and often overlooked flow is the physical installation of the sensors, even more so when installing a few hundreds of them per day.
It starts from the acknowledgment of this step as a design requirement in the mechanical design process, thoroughly thinking out the work flow, simulating it and physically testing it, not only by yourself but also on all sorts of different users. A key understanding here is that you will probably not be the one using/executing this work flow and as such it should be robust, idiot-proof and fault tolerant. If installing your device requires it to be within a millimeter tolerance, make sure you design tooling to help you position it, and in the same breath, make sure you also have a way to re-align it if wrongly placed.
Be agile. Not only in your software development team, but also in the deployment and installation procedures and protocols. Typically, environments hard-to-reach are also full of surprises, and as such, you need to maintain high sustainability to ad-hoc changes and the unpredictable, even more so when you have limited access time and without the ability to look back and fix a previously installed device.
And on a personal note, make sure you always have enough time to finish what you start, you don’t want to find yourself leaving a loosely hanged over-head camera in a populated area or an empty hole in the middle of a highway.
Lastly, don’t forget to try it out. Preferably as much as possible, and iteratively improve both the tooling and methods. It’s impractical and irresponsible to plan and implement such procedures without actually executing them in their true environment and with the actual constraints and limitations.
2. Initializing a distributed sensory system in a harsh, hard to reach environment
Let us, for the sake of the example, imagine we wish to install 1500 temperature and pressure sensors along an oil pipeline laid across some section of ocean. Naturally, we would like to know where each sensor is along the pipe (so we know where the potential issue is), and we would also like to make sure it knows its own location (because it configures its communication params, its sensing needs or plenty of other reasons)
A vanilla solution might be to pre-configure each sensor, mark it appropriately, and just install them in the order configured. But, unfortunately, our diver who is installing the sensors, has to start in the middle section, then jump to the beginning and lastly do the end. And we should also mention once installed reaching it again would be a costly, lengthy process as these divers tend to be expensive.
So, a configuration util/protocol is needed and preferably a simple and effective one, so we can deploy our sensor system arbitrarily, configure it ad-hoc, and potentially have a robust acknowledgement that everything is in order and running and we can send our diver back up (remember he only has this much air).
A key take here, and one written in tears, is constantly acknowledge/verify each deployment step or at the very least don’t leave a device behind without getting a robust signal that it’s functional or remotely reachable. An example for this could be turning on a green light once the device is connected or adding a control app indicating the device is initialized and running. This might sound trivial when installing 3–4 devices but when installing a few hundreds a day you can easily lose track of things and find yourself endlessly searching for a misconfigured device you left behind.
It’s true that if everything rolls as planned you could get off without it, but things tend to not work as such, and keeping a constant and live feedback loop along such deployments is a crucial and worthwhile effort, even if slightly slowing the installation flow and requiring more development.
“It does not do to leave a live dragon out of your calculations, if you live near one.” — J.R.R. Tolkien, The Hobbit
3.1. But what if we mis-configured them?
Nevertheless, apparently, sensors 3,4 and 5 were configured to a wrong channel, we now need a way to remotely access them, re-configure and send them back out. What could presumably seem like an easy task with a distributed internet based system, becomes a challenge when your sensory platform is radio based, without access and partially asynchronous.
Maintenance tools in such system are typically proprietary, much trickier and require a very thorough and robust architecture making sure each of the sensors can always be reached, revived and reconfigured no matter its external and internal state. Examples of such tools and mechanisms could be:
- Monitoring device health metrics — having devices periodically transmit back their health metrics such as battery state, charging, operation mode and any other relevant metric. This should actually be a must-have feature in any such platform.
- Two-way communication — while having the sensors transmit their readings is naturally a must (as it’s their job), having the ability to transmit back to them certain pre-defined parameters configuring anything from their working state to specific thresholds is also a key and useful feature.
- Configuration management utils — both to monitor devices current configuration as well as load and update all or certain sub-sets of sensors for different tasks and operational modes.
- Watch-dogs — a programmatic mechanism for detecting and overcoming software or hardware bugs. You better have a whole pack of those to save you from unfortunately inevitable situations.
- Firmware-update Over the Air — see next clause for more details, but definitely don’t underestimate its importance.
Make sure you design and implement at least some of these, lying to yourself that everything will go as planned will only get you so far.
“Just because you made a good plan, doesn’t mean that’s what’s gonna happen.” — Taylor Swift
3.2. Firmware-update Over The Air (FoTA)
What might seem as a trivial feature in many applications today, is a crucial and tricky feature in distributed radio-based systems where accessibility is an issue but so is bandwidth and synchrony. Nevertheless, maintaining the ability to remotely upgrade and upload new images to the sensors is a must and could not only allow the ability to upgrade the system but also save you when finding (inevitable) bugs in the software.
Having said that, this is a tricky feature both to implement robustly (avoiding deploying firmware that could cause communication loss with a device) but also in terms of service availability and user experience. As firmware updates in their simplest form often require system down-time, this is also a customer-facing feature and should be treated as such. And on that note, implementing a background FoTA process, where the new firmware is uploaded in the background and the sensor boots from the new image upon a command, could be a way to mitigate frequent updates while keeping system availability.
3.3. A Maintenance State of Mind
When all fails remotely, often one would need to physically access the sensor and check/repair it. While in some system maintenance ops can be done on premies, in those hard, harsh environments often the trivial yet very effective method of swapping the unit with a new one and testing/refurbishing the old unit in the shop is the right path to go with.
This requires two critical building blocks:
- A mechanical design allowing for easy swapping of devices. So adhering the sensor to the ceiling with epoxy is probably not a very good approach.
- A software protocol for easy replacement of sensors, initializing the new sensor with the correct parameters.
Having both, you can not only swap a single malfunctioning device, but also upgrade an entire system with minimal disturbance and in a very short time. While it might seem like a nice-to-have luxuries feature, when replacing a sensor requires closing a live and busy highway, cutting down time by half could be a golden feature in the system spec.
A True Story About a Smart Road Platform
At Valerann, we developed a smart road platform with a proprietary and unique data source embedded in the asphalt and installed across and about highways, generating a live and stable traffic map to a single vehicle level regardless of weather, visibility and traffic conditions.
The design of such sensory system incorporates also the deployment and installation of the system, from the physical flow of installing each sensor, initiating it automatically and maintaining it both remotely and if need be also locally.
As part of it, we have developed mechanical peripherals for robust and fast sensor placement, booting applications and protocols and a maintenance stack allowing to remotely monitor, update and tune each sensor in the grid. Doing so, we not only enable installations of over a Km every night, but also constantly upgrade and improve the system as we evolve and enable new state-of-the-art capabilities in road sensing and monitoring, all towards creating the roads of the tomorrow.
