What is the current state of Internet of Things testing?
Is there a current state of IoT testing? The Internet of Things is in its infancy, making testing it just as green, if not greener.
European QA testing expert Paul Gerrard says IoT testing is still pretty undefined:
“Hardly anyone has mentioned it. People don’t understand the risks yet. They don’t know what goes wrong. Most people are thinking of IoT as just an extension of the mobile app, like the Fitbit on your wrist, but that’s kind of the most trivial.”
He says it’s so much more than that — it’s about the dramatic impact the Internet of Things will have on society, where we’re looking down the pipeline at 50 to 100 devices in every home, completely connected or sporadically and unpredictably connected smart cities, interacting with autonomous smart cars and things on or within our bodies.
According to Gerrard, “People haven’t really figured out the social impact of all this. In many respects, these devices can’t do any harm, but the infrastructure is still evolving, the standards aren’t stable yet. We are still in the ‘Betamax versus VHS stage’ where we just don’t know where it’s going to go.”
The risk is that IoT itself is still being released out into the wild in more of a beta version, where just about everything is being created and tested in isolation. But what happens when we bring different brands, manufacturers, protocols, standards, and use cases together?
“The first struggle with Internet of Things testing becomes knowing what to test because it’s no longer just a website, a server, and a mobile app coming together.”
But as the pioneers and the tinkerers of the tech world are giving IoT a go, a lot of the discovery finds that, while it’s undeniably more complicated, IoT brings to head the best practices of software testing that have been touted over the last forty or so years.
Connecting devices to a broader network is nothing new
According to IBM API and IoT expert Andy Thurai, The concept of connecting sensors/devices to a broader network (or even Internet), is not all that new.
There were connected devices/sensors — we called them M2M or machine-to-machine — since the seventies and eighties, particularly in the Industrial Internet areas such as factory assembly lines, utilities, and power industry. They used to be controlled by PLCs in the localized private networks. Hence there was no need to worry much about security or constantly patching/upgrading firmware or replacing it with newer devices.
“It never used to be an issue until now because they were all on a private network, completely controlled, completely isolated, never on the Internet, and not accessible to the bad guys.”
Thurai notes that only a smaller portion of the modern Internet of Things will fall within this controlled environment, as it will be too expensive to maintain private networks. The rest will be out in that unknown we’ll refer to throughout this eBook.
Gerrard makes the comparison that, in the Internet of Things, the Internet is just acting as an implementation of Client-Server. “Suddenly what you were testing is distributed, but now it’s the system of systems that are interacting. It’s now hundreds of thousands of devices 
running thousands of different applications on different networks.”
Most of the Internet of Things is not the same type of controlled environment you’d experience when developing mobile apps. You can be reasonably confident of the specs you’re dealing with when building an app for an iPhone. There are many, many more factors that need to be considered if you want an application to be compatible with a variety of IoT products, said Mike Kruk, Former CEO of Crowsnest, customer support analytics for the Internet of Things.
As exciting of a time it is for rapid innovation, it all makes for a risky investment. “Manufacturers have no control over the entire stack,” said Brian Knopf founder of BRK Security and 20- year veteran of security research and testing.
“If you have to update something, [it] has to be done across whole devices. It’s going to lead to recalls.”
With no clear protocols, no clear standards, and hundreds of devices, across every vertical, there will be millions of devices you won’t have access to for upgrades or patches. And even when you can be almost certain of your own device or software’s security, you can’t be sure what it’ll be integrated with will be anywhere near as secure.
Too Many Cooks in the Kitchen: The challenges of testing in the Internet of Things
“You can’t know what all the connections are and you can’t really test for what all those connections could be,” said Bruce de Grazia, program chair of the cyber security management and policy department at University of Maryland, University College.
Grazia succinctly describes perhaps the most overwhelming part of tackling IoT testing in an up-to seven-layer stack. These stacks aren’t dominated by one provider either but rather they are many companies integrating together, with or without their own knowledge or consent, which sparks the questions:
- Who is in charge of testing which layers?
- Who is responsible for enforcing the protocols and standards that weave the layers together?
As Aditya Gupta puts it, IoT testing is about constantly asking “What network it’s accessing and what data they are getting?” Gupta is a mobile and IoT security researcher and founder of Attify, which helps organizations secure their IoT devices and code.
Diwakar Menon, CEO of Last Mile Consultants says that it all depends on the context of what you are testing. He points to four different possible interactions the device makes in IoT:
- With sensors
- With aggregate gateways
- With the cloud network
- With the application itself
For his consultancy, Menon says it all depends on what they’re looking for.
“If I am testing at the gateway level, I would test it differently than sensors, than apps.”
He calls the “human level of testing,” going through chains, constantly having to be aware of user context, of what the application does, and of what data is exchanged between the users and the sensors.
As a tester — as well as a designer and developer in the IoT space — you must constantly consider these different facets:
- User information: kind of, amount, nature of, who can see it
- Security
- Functionality
- Performance
- What happens if you lose connectivity?
- What happens if you lose power?
- Data aggregation, security, corruption, storage
Stacey Mulcahy, technical evangelist at Microsoft, said,
“In general, you can’t predict your conditions or your environment, so you’ve got to do the best to replicate that, and prepare for downtime and security.”
However, not everyone can afford to simulate many of the unreachable and pricey devices and environments out there. Perhaps even more unpredictable than conditions is the human variable in the Internet of Things, when you have a group of users that are so far from the testing space using tools in unknown ways
“Up to now, we are guessing usage. We always really guess or try to guess in terms of how it’s going to be used, but the bigger challenge is that at the end of the day we can’t really know if we can,” Menon said.
“There are too many facets and one of the challenges of the Internet of Things is that you don’t know where to begin and end,” he continued. “With so many users, you have to be aware of the fact that something is going to be used in other ways.
We are dealing with a whole new beast that will continue to surprise us for a while.
You can uncover some really obscure patterns, said Vlad Trifa, co-founder and VP of research and development, EVRYTHNG IoT platform and of WebofThings.org and the book of the same name said,
“It requires a whole new way to think about that. It’s really hard to test IoT digitally. It’s not like software where I have a piece of code: OK, pass, fail.”
Menon pointed to how a security device could have medical applications or school safety applications. “How on earth would we take that platform and say we’ve tested it for everything?”
But, this is one of the many examples of how the tester’s job in IoT becomes more interesting, with a side of a greater sense of responsibility.
Trifa said, “Because it’s real people and real cities, the impact is much bigger and the complexity of what can happen is massive. It’s pointless to test a smart lock in a lab.
Put that in hundreds of houses where you can uncover information like how it reacts with metal doors, when it rains, when the lock is actually outside the geofence of your house and, if you lose your phone, will you have to sleep on the street?”
Your IoT APIs are next level awesome, your API testing tool should be too
Creating API tests manually based on a written description can be time-consuming and inaccurate. With ReadyAPI, you can use our API Discovery feature to find APIs and generate test structures for them. You can also easily refactor your tests when any of your APIs change.
- Use API descriptions like Swagger, OAS 3.0 to ensure that you've got full test coverage
- Capture API traffic from a website or switch to proxy mode for mobile devices
- Use our refactoring capabilities to update your tests when API definitions change
ReadyAPI provides easy to use 'point and click' capability that lets teams to bring REST and SOAP tests into automation platforms with just a few clicks without writing any scripts.
Our tool includes a configurable command-line interface tool, this enables you to run your tests from any task scheduler or as an integrated part of your build process.
Learn More: