Most the people who were in the TechEd 2012 India would have witnessed the cool demo by me and Abhijit Jana made on the opening Keynotes of the event. And now we would like to share how this solution was developed so that you can also build your own.
Background
Before we could pitch in, let me give a little background on how we decided to build this on our garage. Developer Conference 2011 on Hyderabad was scheduled on October 2011 and myself and Abhijit were looking forward to show something cool on our own interested areas. I was looking out for something on Windows Azure and Windows Phone while Abhijit was fiddling around Kinect. Ran out of ideas we both met together and thought of integrating everything and that is when we came up with the idea of building our own home security. We started building this application as early as September 2011. And by now, we have already revised the application more than 5 times due to newer SDK release and also due to the addition of newer components.
Overview
So how does the solution work. You should place Kinect devices on the locations you want to secure and keep it connected to a high end PC capable enough to do a video encoding and up streaming. The kinect devices will be run by a WPF application running on the PC which will look for any human intrusion. While there is no human intrusion, it will remain in “Patrol Mode” where it will upload pictures of its view on regular intervals to Windows Azure. These pictures can be monitored at any point of time through the application running on Windows Phone and Windows 8. Whenever there is an human intrusion detected the Kinect changes itself to “Intrusion Mode”; it then alerts Windows Azure and also starts streaming the video to the Smooth Streaming Server hosted on Windows Azure. The Azure notifies all the devices subscribed (Windows Phone / Windows 8 ) using a toast and will allow those application to view the live streaming. From your Windows Phone / Windows 8 app you can also control the Kinect device.
Windows Azure
Now that we are done with the overview of how the solution works, lets talk about how it is built. Starting with Azure, below is how the application hosted on the cloud.
Windows Azure Management Portal
The Windows Azure Management Portal is created using the default template provided from Windows Azure Toolkit for Windows Phone or Windows Azure Toolkit for Windows Phone. This management portal can be customized later on to support the needed services. To begin with, the management portal by default integrates with all the above mentioned blocks except for the Smooth Streaming server; which means the API which would require for all the communication with these components are already defined. Now lets go with the flow.
Custom WCF Service Methods
In order to support the WPF application running on the PC connected to the Kinect devices, we will expose a custom WCF method called “Upload” allowing the it to upload an object of type “Photo”.
The image is uploaded as a byte array along with the type (Patrol / Alert) and time. As soon as the WCF service gets the message, it serializes it to string and puts it into the blobs and also pushes it to queue with the expiration time set for 1 minute. This will help the user to see the latest picture from his home at any point of time.
Shared Access Signature Service
The portal derived out of the template would require the authentication done through the certificate along with a user name and password for the generation of the token. Internally the portal uses ASP.net membership authentication through which it authenticates the user and provides a mutually agreed token to the client. So doesn’t matter whether the Windows Phone, Windows 8 needs the access to the Azure, it will require this mechanism to get authenticated. The token then will be required to access any methods from the service.
Microsoft Push Notification Service
This service allows the solution to send a notification message to any smart device which is capable of subscribing / receiving push notifications. The application which is written in Windows 8 and Windows Phone subscribes itself for the notification as soon as application is run for the first time. All the device needs is an internet connection for subscribing / receiving the notifications. Once the Azure service receives an alert message from the Kinect, it sends a push notification to all the devices subscribed. The subscription details are stored in the Azure Tables by the management portal.
Smooth Streaming Server
Configuring this server in the Azure would be the toughest part but would be a cake walk if you understand how it is done. Azure Live Streaming on codeplex allows you to do this. The Web role will have a “Startup.cmd” file which will contain the command as below to install the Smooth Streaming Server on the Azure Server.
set msiexec=%systemroot%system32msiexec.exe
set appcmd=%systemroot%system32inetsrvappcmd.exe
%msiexec% /i “%~dp0IISMedia64.msi” /qn ADDLOCAL=ALL /Le startup_media.txt
exit /b 0
The setup file IISMedia64.msi has to be added to the project as a content. The service definition file for the azure will have the below tags
<Startup>
<Task commandLine=”startup.cmd” executionContext=”elevated” taskType=”simple”></Task>
</Startup>
This will allow the batch file to run once the web role starts up.
You will also require to add “.isml” files for configuring the uplink / downlink streaming end point. For more information on how to configure the streaming server in Azure, please follow the documentation. The kinect will up stream the compressed video stream to the uplink endpoint and once the video stream is available on the Azure, it will automatically be available on the downlink to have a live streaming.
Note: You can configure any number of instances for a downlink server but only one instance for uplink server.
AppFabric Service Bus
We are using the appfabric service bus as a remote endpoint for a WCF service hosted in the application which is connected to the device. This will allow anybody to control the Kinect device like tilting just by sending a message to the WCF endpoint. This will eliminate the need of a static IP / DNS and the firewall overheads which may incur.
In our application there is a method exposed called “SendMessage” by the WCF hosted in the WPF application running along the Kinect devices. Once Windows Phone / Windows 8 sends a command to this method namely “up/down”, the device tilts as per the command.
Now that we have talked enough about Azure, lets talk a little about Kinect on how it is going to act as a watch dog for the solution.
Kinect for Windows
Kinect works as Watch Dog at your home which is connected with Windows Azure. The application was built on Kinect for windows SDK ( V 1.0) . We have used the Skeleton detection features to detect the human body motion. As we have already mentioned by default Kinect will be in “Patrol mode” , where it will upload the image with every 10 sec of interval and if there is any skeleton detection by Kinect sensor, it will start the streaming. We have used the expression encoder SDK to do the stream from application. Below are the list of components which we have used in WPF Client application.
Below are the very top level view of code block used for detection of skeleton and uploading it to azure.
Find if Skeleton is tracked and then set a global variable which is being used across the application.
if (this.skeletonData.Where(item => item.TrackingState == SkeletonTrackingState.Tracked).ToList().Count > 0
{
Global.IsSkeletonTracked = true;
}
Create a patrol type image is there is no Skeleton otherwise send the alert image.
if (Global.IsSkeletonTracked)
{
photo = newPhoto() { CreatedOn = DateTime.Now, Image = filesbyte, Type = PhotoType.Alert };
}
else
{
photo = newPhoto() { CreatedOn = DateTime.Now, Image = filesbyte, Type = PhotoType.Patrol };
}
The type of Photo is defined with our service contract and we are uploading the images as bellow:
KinectReceiverClient client = newKinectReceiverClient(“BasicHttpBinding_IKinectReceiver”);
ServicePointManager.ServerCertificateValidationCallback = ((obj, certificate, chain, sslPolicyErrors) => true);
client.UploadPhoto(photo);
Windows Phone
As I mentioned earlier, Windows Phone will act as one of the end user device in this solution. To interact with Azure easily and securely, we will be using the Windows Azure Toolkit for Windows Phone client. As mentioned in the Azure, the windows phone app will authenticate itself against the Azure on the start up using a user name and password signed by the certificate. Once the application is authenticated, the app in the first screen will show the latest pictures which is peeked from the queue. Remember that these images are pushed to the cloud by the Kinect in the “Patrol” mode.
When there is an intrusion, the Windows Phone receives a toast notification. There is no need to keep the application running / keep your phone unlocked for you to see the notification as the Windows Phone takes care of all the services needed for listening to the notification. Upon tapping the notification, you will be navigated to the page where you will see the live stream through the Multimedia Player for Windows Phone streaming through the downlink url.
Windows 8
The metro app developed on Windows 8 runs pretty much the same way as Windows Phone except that it uses Windows Azure Toolkit for Windows 8. Since metro apps also supports the notification service, it’s the same functionality you will require to implement.
Application Functioning on Patrol Mode
Application Functioning on Intrusion Mode
Application Functioning on Accessing Kinect Remotely
We haven’t shared the code here while we have shared everything that you will require to build this application. We would encourage you to try building the application with all the knowledge we have shared and be wild with your imagination. If you require any help on building the same, please do drop a note to us and we will be more than happy to help you.
Resources
- Windows Azure
- Windows Phone
- Windows 8
- Kinect For Windows
- Windows Azure Toolkit for Windows Phone
- Windows Azure Toolkit for Windows 8
- Windows Azure Live Smooth Streaming
Showcased
- Multiple events inside Microsoft.
- Developer Conference Hyderabad (Oct 2011)
- TechEd India 2012 (Jan 2012)