VideoStream for Microsoft® PixelSense™

Search and watch your favorite YouTube® videos

The first time I saw Bing for Microsoft® PixelSense™ I knew it was one of those application that mark the era of the multitouch multiuser NUI development. It is no surprise if Microsoft has a patent pending on the result control.

The way users can search is one of the nuiest interaction I ever saw.

From when I had the opportunity to use the Bing app, in early 2012, I was so impressed from that user experience that I decided to reproduce it.

When I see and I fell that a user interaction is well done I like to study and research on it and I try to create my own exploring all the possible implementations.

This is a technical challenge: look at the Amnesia Connect from Razorfish, how do you reproduce this awesome effect? Well, it is not simple but my curiosity is strong, I need to know.

At the end I came out with my solution on this that I published on CodePlex sometime ago. Well, that is not a production code it's only the idea on how to achieve the Amnesia Connect effect but with no budget on that my curiosity is satisfied, if a customer asks for a similar user experience we are ready for the development of a real world application.

Same happens with Bing for Microsoft PixelSense, but this time I noticed that the Bing app miss something: the ability to search for videos so I decided to work on reproducing the experince but searching for YouTube videos.

Result is the sample that you can download on MSDN Code Gallery, you can also watch a video of it on Software Lab's YouTube Channel

 

There are some point of interesting the sample:

  • Reproduced user experience of the search result thrown away from the search box

  • Async search for YouTube videos

  • Async search using Bing API on Windows Azure to search for images

  • Async file download

  • Thread safe observable collection

Reproduced user experience of the search result thrown away from the search box

The challenge of reproducing this experience is to create the ScatterView near, or better below, the search box and then throw it away from the box. Obviously, every new search result have to be thrown in a different place.

Well, this is pretty simple, the idea is to create the search result below the search box by using the center property of the search bow and then applying a point animation to animate the generated result and simulating the throw action.

The trick to throw the generated every time to a different position is to use a static random number generator. The random number generation starts from a seed value. If the same seed is used repeatedly, the same series of numbers is generated because of this using the same instance of the random generator will drastically increase the probability to get a different number every time.

And here is the implementation:

 

// result is added to ScatterView

SearchResultControl ctrl = new SearchResultControl();

ctrl.DataContext = list;

ScatterViewItem svi = new ScatterViewItem();

svi.Style = (Style)LibrayStiles["LibraryContainerInScatterViewItemStyle"];

svi.Content = ctrl;

svi.Center = sviSearch.ActualCenter;

scatterContainer.Items.Add(svi);

 

// svi created near seach control and thrown away

Storyboard stb = new Storyboard();

PointAnimation moveCenter = new PointAnimation();

// _randomizer is static the istance is created once in the static constructor

Point endPoint = new Point(_randomizer.Next(0, 1500), _randomizer.Next(0, 1000));

moveCenter.From = svi.Center;

moveCenter.To = endPoint;

moveCenter.Duration = new Duration(TimeSpan.FromSeconds(1.0));

moveCenter.FillBehavior = FillBehavior.Stop;

stb.Children.Add(moveCenter);

Storyboard.SetTarget(moveCenter, svi);

Storyboard.SetTargetProperty(moveCenter, new PropertyPath(ScatterViewItem.CenterProperty));

stb.Begin(this);

svi.Center = endPoint;

 

As you can see on the video or by using the app every time a search result is thrown the destination is completely random.

 

Async search for YouTube videos

As I already wrote, the five design principles of Microsoft design for Windows Store app are fully applicable to the design and development for PixelSense: 1. Pride in craftsmanship. 2. Be fast and fluid. 3. Authentically digital. 4. Do more with less. 5. Win as one.

By now, the problem to solve it to respect the second principle when the user types it's reach, the search result is created and the application have to:

  • Query YouTube with the search string of the user
  • Download and parse results
  • For each result download a thumbnail image of the video

Unfortunately Surface SDK 2.0 is targeting the .Net Framework 4.0 so virtually we can't access all the asynchronous programming facilities introduced with the .Net Framework 4.5 and we have to write all the code by hand but with a bit of effort we will have our asynchronous query to YouTube.

We will need a Video object that will represent a single result from YouTube containing all the properties to point to the correct stream source, thumbnail image an so on.

 

public class Video : ObservableObject
{

    #region private props
    string _linkUrl;
    string _embedUrl;
    string _thumbNailUrl;
    #endregion

    #region public properties
    public string LinkUrl
    {
        get { return _linkUrl; }

        internal set
        {
            _linkUrl = value;
            RaisePropertyChanged("LinkUrl");
        }
    }

    public string EmbedUrl
    {    
        get { return _embedUrl; }
        internal set
        {
            _embedUrl = value;
            RaisePropertyChanged("EmbedUrl");
        }
    }
 
    public string ThumbNailUrl
    {
        get { return _thumbNailUrl; }
        internal set
        {
            _thumbNailUrl = value;
            RaisePropertyChanged("ThumbNailUrl");
        }
    }
    #endregion

    #region ctor
    public Video()
    { }

    public Video(string linkUrl, string embedUrl, string thumbNailUrl)
    {
        LinkUrl = linkUrl;
        EmbedUrl = embedUrl;
        ThumbNailUrl = thumbNailUrl;
    }
    #endregion

    public override string ToString()
    {
        return string.Format("LinkUrl: {0}; EmbedUrl: {1}; ThumbNailUrl: {2}", _linkUrl, _embedUrl, _thumbNailUrl);
    }
}

 

The base class ObservableObject incapsulates all the logic and implementation for INotifyPropertyChanged interface.

Then we need a custom event arg that we will use when we will raise the download event

public class VideosDownloadCompletedEventArgs : EventArgs
{
    // Fields
    private Guid _id;
    private ThreadSafeObservableCollection<Video> _downloadedVideos;

    public VideosDownloadCompletedEventArgs()
    {
        _id = Guid.NewGuid();
    }

    public VideosDownloadCompletedEventArgs(Guid id)
    {
        _id = id;
    }

    public virtual Guid Id
    {
        get
        {
            return _id;
        }

        internal set
        {
            _id = value;
        }
    }

    public virtual ThreadSafeObservableCollection<Video> DownloadedVideos
    {
        get
        {
            return _downloadedVideos;
        }

        internal set
        {
            _downloadedVideos = value;
        }

    }
}

 

 

Now we are ready to define the event handler:

public delegate void VideosDownloadCompletedEventHandler(object sender, VideosDownloadCompletedEventArgs e);

And now we can implement the logic to interact with YouToube in an helper class:

public class VideoDispenser : INotifyVideosDownloadCompleted
{
    private static readonly string SEARCH = "http://gdata.youtube.com/feeds/api/videos?q={0}&orderby=relevance&start-index=1&max-results=40&v=2&alt=rss";

    public void LoadVideosAsync(ThreadSafeObservableCollection<Video> collectionToFill, string keyWord, int results, int stratAt)
    {
        // TODO: Verify CancellationToken http://stackoverflow.com/questions/3712939/cancellation-token-in-task-static readonlyructor-why

        Task.Factory.StartNew(() => LoadVideosAsyncInternal(collectionToFill, Guid.NewGuid(), keyWord, results, stratAt));
    }

    public void LoadVideosAsync(Guid downloadEventId, string keyWord, int results, int stratAt)
    {
        ThreadSafeObservableCollection<Video> collectionToFill = new ThreadSafeObservableCollection<Video>();

        Task.Factory.StartNew(() => LoadVideosAsyncInternal(collectionToFill, downloadEventId, keyWord, results, stratAt));

    }

    private void LoadVideosAsyncInternal(ThreadSafeObservableCollection<Video> collectionToFill, Guid downloadEventId, string keyWord, int results, int stratAt)
    {
        try
        {
            var xraw = XElement.Load(string.Format(SEARCH, keyWord));
            var xroot = XElement.Parse(xraw.ToString());
            var links = from item in xroot.Element("channel").Descendants("item")
                            select new Video(item.Element("link").Value, GetEmbedUrlFromLink(item.Element("link").Value), GetThumbNailUrlFromLink(item));

            foreach (var i in links)
            {
                collectionToFill.Add(i);
            }

            RaiseDownloadCompleted(collectionToFill, downloadEventId);
        }
        catch (Exception e)
        {
            Trace.WriteLine(e.Message, "ERROR");
        }
    }

    #region Download Event
    public event VideosDownloadCompletedEventHandler DownloadCompleted;

    protected virtual void OnDownloadCompleted(VideosDownloadCompletedEventArgs e)
    {
        var handler = DownloadCompleted;
        if (handler != null)
        {
            handler(this, e);
        }
    }

    protected void RaiseDownloadCompleted(ThreadSafeObservableCollection<Video> downloadedVideos, Guid downloadEventId)
    {
        OnDownloadCompleted(new VideosDownloadCompletedEventArgs() { Id = downloadEventId, DownloadedVideos = downloadedVideos }); 
    }
    #endregion

    #region Private Methods
    /// <summary>
    /// Simple helper methods that tunrs a link string into a embed string
    /// for a YouTube item.
    /// turns
    /// http://www.youtube.com/watch?v=hV6B7bGZ0_E
    /// into
    /// http://www.youtube.com/embed/hV6B7bGZ0_E
    /// </summary>
    private static string GetEmbedUrlFromLink(string link)
    {
        try
        {
            string embedUrl = link.Substring(0, link.IndexOf("&")).Replace("watch?v=", "embed/");
            return embedUrl;
        }
        catch
        {
            return link;
        }
    }

    private static string GetThumbNailUrlFromLink(XElement element)
    {
        XElement group = null;
        XElement thumbnail = null;
        string thumbnailUrl = @"../Images/logo.png"; 

        foreach (XElement desc in element.Descendants())
        {
            if (desc.Name.LocalName == "group")
            {
                group = desc;
                break;
            }
        }

        if (group != null)
        {
            foreach (XElement desc in group.Descendants())
            {
                if (desc.Name.LocalName == "thumbnail")
                {
                    thumbnailUrl = desc.Attribute("url").Value.ToString();
                    break;
                }
            }
        }
        return thumbnailUrl;
    }
    #endregion
}

 

Now we are ready to query video on YouTube and show it in result control, when a user touches the search button we will automatically create the result control and we will launch asynchronous query on the video provider.

private void Search_Event(object sender, SearchEventArgs e)
{
    // search for videos on YouTube
    string a = e.KeyWord;
    ThreadSafeObservableCollection<Video> list = new ThreadSafeObservableCollection<Video>();
    videoDispenser.LoadVideosAsync(list, e.KeyWord, 10, 1);

    // result is added to ScatterView
    // svi created near seach control and thrown away
    // implementation showed previously
}

 

The send principle is satisficed: the UI will remain responsive during all the search process, when video are ready will be shown in the search result ready to be watched and streamed.

Something really similar will happen for the Async search using Bing API on Windows Azure to search for images and for the Async file download you can browse the complete code on MSDN Code Gallery where you will find the complete full working application

Thread safe observable collection found the snippet below online and used because it works really well and it's a simplified version of the one we use in some enterprise application.

This collection is really important when you work with async and thread because it encapsulate all what you need to synchronize object created and modified in different thread with the UI thread this will prevent any cross thread exception.

Basically it extends the Observable collection:

public class ThreadSafeObservableCollection<T> : ObservableCollection<T>
{
    // Override the event so this class can access it
    public override event System.Collections.Specialized.NotifyCollectionChangedEventHandler CollectionChanged;

    protected override void OnCollectionChanged(System.Collections.Specialized.NotifyCollectionChangedEventArgs e)
    {
        // Be nice - use BlockReentrancy like MSDN said
        using (BlockReentrancy())
        {
            System.Collections.Specialized.NotifyCollectionChangedEventHandler eventHandler = CollectionChanged;

            if (eventHandler == null)
                return;

            Delegate[] delegates = eventHandler.GetInvocationList();

            // Walk thru invocation list
            foreach (System.Collections.Specialized.NotifyCollectionChangedEventHandler handler in delegates)
            {
                DispatcherObject dispatcherObject = handler.Target as DispatcherObject;
                // If the subscriber is a DispatcherObject and different thread
                if (dispatcherObject != null && dispatcherObject.CheckAccess() == false)
                {
                    // Invoke handler in the target dispatcher's thread
                    dispatcherObject.Dispatcher.Invoke(DispatcherPriority.DataBind, handler, this, e);
                }
                else // Execute handler as is
                    handler(this, e);
            }
        }
    }
}

And here we go, another great user experience is ready for the real life implementation for our customer. The goal of the sample is reached: create a full working application that will reproduce the basic behavior of the budgeted Bing app for Microsoft PixelSense.

In this way we will have an exact idea of technologies to use and how effort will take a real life implementation of the app.

 

Next time I will discuss about content exchange between the Samsung SUR40 and handled devices such as the newMicrosoft Surface, Windows Phone and why not iPhone and Android.

With the same approach, I will give you an idea of the technologies to adopt so you will have a clear idea of all the effort and skills needed to create a great user experience.

 

Ultimi Post

Discalmer

Articles and content of this blog aren't and shouldn't be interpreted as professional advice or opinions. Author writes on a personal basis and as part of their own research, experimentation and knowledge. Opinions expressed aren't in any way attributable to professional affiliations / institutional thereof or to opinions that the author express as part of their roles / positions that may be quite different from that reported here.