Overview / Usage

World's first modular, artificial intelligent and drive-able smart home monitoring device that can stream live video, audio and sensors data to any smart phone, tablet or computer.

It has an HD Camera, Microphone, Speaker, on-board storage, Wireless charging and an array of environmental sensors so user can monitor home/office from anywhere in the world.

It is the most innovative smart home monitoring solution. A user can control it from anywhere in the world and drive it from room to room. With its intelligent object recognition system on-board(with the help of Movidious), it can distinguish general movements from something that needs user's attention and can send you alerts accordingly.

Methodology / Approach

Traditional security cameras are not intelligent cameras and they have limited visual coverage, but this will be smart / Artificial Intelligent and drive-able camera, powered by Intel Movidious, to recognize and differentiate between different objects and movements, and alert user accordingly, and with drive-ability it can get more visual coverage than any other security camera in the market, as it can drive from room to room.

THINGS WE USED IN FIRST PROTOTYPE:

• Raspberry Pi 3 with Windows 10 IOT https://www.windowscentral.com/how-install-windows-10-iot-raspberry-pi-3
• Sensors and serial motor driver https://www.pololu.com/product/1110
• Two Geared DC Motors (6.0 Volt) https://www.pololu.com/product/3074
• A USB Web cam https://www.amazon.com/Logitech-Widescreen-designed-Calling-Recording/dp/B004FHO5Y6/ref=lp_172511_1_2?s=pc&ie=UTF8&qid=1532393246&sr=8-2
• Intel Movidious Compute Stick
• Download Pre-Trained Model (SqueezeNet which has about 1000 items pre-trained) from https://github.com/onnx/models/tree/master/squeezenet and Extract model.ONNX file
• Download the Updated Sample UWP App Source Code that also includes SquuezeNet Model in Assets folder https://github.com/omarbarlas/WinMLExplorer
• Visual Studio 2017
• Windows SDK – Build 17110 or above https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewSDK (If you have latest Windows 10 Pro it should be already on the system)

STEPS OF BUILDING PROTOTYPE:

1. At first we Setup Windows 10 IOT on Rasberry PI 3, with instructions on https://www.windowscentral.com/how-install-windows-10-iot-raspberry-pi-3
   (Note that Raspberri PI3 B+ is currently not fully supported on Windows 10 IOT)

2. Once Windows 10 IOT installed, connect to Raspberri Pi 3 to Wi-Fi (It is part of Setup process)

3. Next we started writing Windows 10 IOT App in Visual Studio 2017,

a) We downloaded WinML Explorer Sample App to start with, We wrote Custom code in the MainPage.XAML.cs to Load USB Camera stream as VideoFrames and pass it to MainViewModel which has functions to run inference

b) Using the interface's generated wrapper classes, we follow the load, bind, and evaluate pattern to integrate SqueezeNet ML model into the app. More in dept details and some sample code Is at https://docs.microsoft.com/en-us/windows/uwp/machine-learning/integrate-model

c) After completing the above steps we were able to Show Live Video Stream on the Main Page and on right side of the page it was showing Labels of recognized objects, The code is able to run inference using CPU, GPU or VPU (Movidius Compute Stick) as Windows 10 IOT Enables drivers for Movidious in the next release of Windows 10 IOT the code will by default utilize the Connected VPU and that will be increase the Object Recognition speed significantly and CPU becomes available to do other tasks more efficiently.

d) Next we started Adding Virtual Joystick Control and DC Motor Driver Code in the Source Code (That part of code is not in the above sample WinMLExplorer project) but here is some c# code to give you an idea
We are using Serial Motor Driver from Pololu that is capable of taking Serial Commands from Microcontroller over UART Serial Bus and translate those Commands to Drive 2 DC Motors. and Using Inputs from Joystick Control on screen, we sent command packets that included Motor Direction and Speed values to Serial Port and Motors drive accordingly, which drives the whole Robot Forward, Backward, Turn Right or Left.

Here is the C# Code to drive 2 Motors with Serial Port:

dataWrite.WriteBytes(commandByteArray);
//SerialPort is of type SerialDevice that handles writing ByteArray to Serial port on hardware
uint bytesWritten = await SerialPort.OutputStream.WriteAsync(dataWrite.DetachBuffer());

We also connected IMU (Gyro, Accelerometer and Magnetometer) to Raspberry PI using I2C pins so we can calculate Heading Angle, Pitch/Yaw and adjust motor speeds to correct heading angle while driving. And eventually with the inputs from Camera and IMU Sensor we will be able to Drive Autonomously in the indoor environment, we may also utilize VSLAM (Visual Simultaneous Localization and Mapping) from third party SDK’s

We are also working on onboard API which will be capable of streaming Live Video and sensors data to a remote client app, and client app running on phones or tables will be able to send drive commands back to camera robot. To navigate to specific room while streaming video, audio and sensors data.
We are already working on New PCB for our next prototype and 3D printed housing and shells for the robot, the Video of the prototype is included with this article.

As Windows 10 IOT supports Remote Desktop Connection so in the Current prototype we are using RDP client to connect to the App running onboard and able to see live Video feed and drive with Joystick on screen.

Technologies Used

• Intel Movidious,
• Windows 10 IOT
• UWP Apps
• Win ML
• Visual Studio 2017,
• Rasberry PI 3
• USB Camera
• Pololu Serial Motor Driver
• 7.4 Volt Lithium Battery
• 5.0 Volt Voltage Regulator to Step Down Battery Voltage to 5.0 Volt.

Repository

https://github.com/omarbarlas/WinMLExplorer

Other links

• Indiegogo Campaign Page