Microsoft: Get Your Shit Together on Touch Development

Testing Samples from Microsoft Surface Toolkit...

Image by John Bristowe via Flickr

I have been playing with multi-touch development for a while, both on Windows 7 with my 2740p and on the Microsoft Surface table (version 1, not the new one).

I have consistently had challenges using WPF4 multi-touch events on the 2740p, or using the Microsoft Surface Toolkit for Windows Touch Beta, and now with the newly released Surface 2 SDK.

With any of these tools, I have challenges getting the software to recognize touch events, and even more trouble getting the software to recognize 2 simultaneous touch points (the 2740p supports 2 touch points). A second touch point always cancels the first touch point.

What is funny (to me, anyway) is that the samples included with the machine in the Microsoft Touch Pack for Windows 7 work just fine on the 2740p. This indicates that it is something in the managed drivers used with WPF that is not working.

I am fairly certain that this is a driver issue. I had it working at one point after a lot of hacking and installing drivers different from the default updates. I guess an update somewhere (Windows Update, or HP Tools) has overwritten the driver I had setup to make it work.

Can I fix the drivers to make this work again? Absolutely! But that is not the point here.

This should just work!

How am I as a software developer, or as an ISV, supposed to recommend this platform to my customers, or build applications for it, when even getting it to run on different machines requires significant hacking of drivers?

If Microsoft wants to stop being seen as a joke in the multi-touch and/or tablet market, they really better find a way to get their shit together on this.


Displaying an XPS Document on the Microsoft Surface

As a part of an application I am prototyping, I ran into the need recently to display a document inside a ScatterViewItem on the the Microsoft Surface. Since there is (intentionally) no built-in way to display HTML content on the Surface, and displaying a Word document did not seem feasible, I settled on using an XML document, and using the WPF DocumentViewer control.

This seems pretty easy, right? Just put a DocumentViewer inside your ScatterViewItem, load the document into the DocumentViewer, and you’re done. Couldn’t be easier.

Well, displaying the XML document was indeed that easy. Unfortunately, as I expected, the DocumentViewer would not respond to touch at all. I posted a question to the MSDN Surface forums about this, and did not receive any response for several weeks.   The response I finally did receive was that I would have to develop a User Control, and handle the touch events myself.

Not being one to take advice, I decided to try another approach – I decided to see if I could hack the ControlTemplate for the DocumentViewer in such a way as to have it support touch (keep in mind that I am a relative noob when it comes to all of this WPF/Styles/ControlTemplate stuff).

So I created a simple Surface project in VS2008, and modified the SurfaceWindow1.xaml file to have a ScatterView control, containing a single DocumentViewer control, as shown below:

<s:SurfaceWindow x:Class="SurfaceDocumentViewer.SurfaceWindow1"
    Title="SurfaceDocumentViewer" Loaded="SurfaceWindow_Loaded">
    <ImageBrush x:Key="WindowBackground" Stretch="None" Opacity="0.6" ImageSource="pack://application:,,,/Resources/WindowBackground.jpg"/>
    <s:ScatterView Background="{StaticResource WindowBackground}" >
        <DocumentViewer Margin=”15” Height="Auto" Name="docViewer" Width="Auto" />

Note that I added a margin around the DocumentViewer so that there would be something to grab on to in order to resize the ScatterViewItem. 

I then opened the project in Expression Blend so that I could get a copy of the “default” style/template for the DocumentViewer control. Opening the project in Blend, right click on the DocumentViewer, and select Edit Template | Edit a Copy…, as shown


Name the style (I used SurfaceDocumentViewerStyle), and define a location for it (I left it in the SurfaceWindow1.xaml file), as shown below.


Click OK, then save your changes and close Expression Blend.

Returning to Visual Studio, you will get a message that the project has been changed outside of Visual Studio. Click Reload to load the changes into Visual Studio. Opening SurfaceWindow1.xaml in design view, you should now see a <Style> element under <s:SurfaceWindow.Resources>, and within that, a ControlTemplate for the DocumentViewer. There are several parts to the ControlTemplate:

  • A ContentControl for the toolbar, that loads from an external assembly.
  • A ScrollViewer named PART_ContentHost that is the container for the actual document display
  • A DockPanel that provides background for PART_ContentHost
  • Another ContentControl names PART_FindToolBarHost where the search box is hosted

The only part important to me was the ScrollViewer. I also wanted to get rig of the toolbar and the search box in order to keep things as clean as possible. So I deleted the other parts.

Here then is the key step to making the DocumentViewer touch aware: I replaced the ScrollViewer with s:SurfaceScrollViewer. My new ControlTemplate now looks as shown below:

<Style x:Key="SurfaceDocumentViewerStyle" BasedOn="{x:Null}" TargetType="{x:Type DocumentViewer}">
    <Setter Property="Foreground" Value="{DynamicResource {x:Static SystemColors.WindowTextBrushKey}}"/>
    <Setter Property="Background" Value="{DynamicResource {x:Static SystemColors.ControlBrushKey}}"/>
    <Setter Property="FocusVisualStyle" Value="{x:Null}"/>
    <Setter Property="ContextMenu" Value="{DynamicResource {ComponentResourceKey ResourceId=PUIDocumentViewerContextMenu, TypeInTargetAssembly={x:Type System_Windows_Documents:PresentationUIStyleResources}}}"/>
    <Setter Property="Template">
            <ControlTemplate TargetType="{x:Type DocumentViewer}">
                <Border Focusable="False" BorderBrush="{TemplateBinding BorderBrush}" BorderThickness="{TemplateBinding BorderThickness}" CornerRadius="5" >
                    <Grid Background="{TemplateBinding Background}" KeyboardNavigation.TabNavigation="Local">
                            <ColumnDefinition Width="*"/>
                            <RowDefinition Height="Auto"/>
                            <RowDefinition Height="*"/>
                            <RowDefinition Height="Auto"/>
                        <s:SurfaceScrollViewer HorizontalScrollBarVisibility="Hidden" VerticalScrollBarVisibility="Auto" x:Name="PART_ContentHost" IsTabStop="true" TabIndex="1" Focusable="{TemplateBinding Focusable}" Grid.Column="0" Grid.Row="1" CanContentScroll="true" />

Now, to test this, add an XPS document to your project (mine is name test.xps). Add an assembly reference to ReachFramework (XPS package support), and add using statements to SurfaceWindow1.xaml.cs as shown:

using System.IO;
using System.Windows.Xps.Packaging;

Add the following code to the end of the SurfaceWindow1 constructor to load and display the XPS document:

XpsDocument doc = new XpsDocument("test.xps", FileAccess.Read);
docViewer.Document = doc.GetFixedDocumentSequence();

Finally, to make the XPS document resize when you resize the ScatterViewItem, Add an event handler to your DocumentViewer to handle the SizeChanged event, as shown below:

private void docViewer_SizeChanged(object sender, SizeChangedEventArgs e)

If everything went according to plan, you should now be able to run your code and you should get a ScatterViewItem displaying your XPS file which is resizable, and which supports touch to navigate around the document.

(I think this should also work with the Surface Toolkit for Windows Touch, but I haven’t tried it yet)

Some challenges with MS Surface Development

So I have been playing with the MS Surface for a couple of weeks, and have a pretty good handle on the basics of the development model. As I said previsouly, the nice thing (for me, anyway) is that it is pretty standard .NET stuff. You can do pretty much anything you need to using Windows Presentation Foundation (WPF). That being said, it is not without its challenges, and I would like to share some of what I have seen so far. 

1) The SDK only installs on 32-bit Windows Vista. This is a challenge for me, since my T4G laptop is running XP, and all of my other computers are running 64-bit Windows 7. The big value of the SDK is that it contains a “Surface Simulator” which allows you to experiment with Surface development without actually having a Surface. I tried setting up a 32-bit Vista VM to use for the SDK, but the simulator does not work in the VM. Now the good news, after a couple of weeks of messing around, I managed to hack the .msi file for the SDK, which then allowed me to install on 64-bit Win7. All seems to work great now.  

2) WPF experience is hard to come by. I can program in WPF, and understand how it works, but when it comes to the fancy styling and more creative aspects of what you can do with XAML, I am definitely no expert. Apparently, neither is anyone else I know!

3) Changing the way you think about the user interface. This is the biggy. The UI model for the Surface is different than anything else with which I have worked. yes, it is a multi-touch platform, which is cool, but hardly unique. If all you want to do is develope multi-touch apps, you can do it much more cheaply on a multi-touch PC (both WPF and Silverlight now support multi-touch development on Windows 7). The unique aspects of the Surface are that it is social, immersive, 360-degree, and supports interaction with physical objects. In order to make full use of the Surface platform, you have to think about all of these things. You also have to break old habits regarding how the user interacts with the platform. We are used to menus, text boxes, check boxes, drop downs and all the usual UI components we have lived with for so long in desktop applications. Or the content and navigation models we are used to on the web. The Surface requires us to forget all of that, and think of interaction in a new way. In this sense, it is more like iPhone development. However, even iPhone development gives you a fairly strict environment which defines how your app ahould look. The Surface on the other hand, is wide open. You can create almost any interaction model you can imagine, supporting multiple user working either independantly or collaboratively, working from any or all sides of the screen, with or without physical objects. This requires a whole new way of thinking, at least for me.

4) Ideas. This is another big challenge. I have lots of ideas for applications for the Surface. Some of them I am pretty sure are good. Some of those are even useful. Some of my other ideas are probably downright stupid. I would like to hear your ideas. I have always believed that, the more people you have coming up with ideas, and the more ideas you come up with, the better your chances of finding great ideas. So shoot me email with any or all ideas you might have – and don’t worry, they cannot be any more silly than some of mine!

Finally, I have added a little video showing just how far you can go with the Surface UI. Hopefully in the next couple of days, I will have a video of some of what I am working on to show.

DaVinci (Microsoft Surface Physics Illustrator) from Razorfish – Emerging Experiences on Vimeo.

First Thoughts on Microsoft Surface Development

A brand new Microsoft Surface development unit arrived this week in the Moncton T4G office. As I start to develop some prototypes, I will be doing some related posts, but I wanted to start by talking about the platform a little, and the development environment.

For anyone who has no idea what the surface is, it is a multi-user, multi-touch platform released by Microsoft a couple of years ago. Have a look at this video to see what it can do.

Other the last few weeks, before the unit arrived, I have learned quite a bit about the Surface. The first interesting thing I learned was the the surface is not a touch screen in the sense that your iPhone or multi-touch laptop are. The surface of the Surface is just glass – it is not a capacitative or pressure sensitive material at all. All of the touch behaviours and interactions are based instead on a computer vision system. Inside the box there is a fairly standard PC running Windows Vista, with an DLP projector pushing the image up to the table top. There are also 5 cameras inside the box which perform the actual "vision". These feed into a custom DSP board which analyses the camera feeds into something a little more manageable for the PC. The fact that it is a vision-based system leads to some interesting capabilities, as well as some idiosyncrasies.

When the Surface is running in user mode, the Windows Vista UI is completely suppressed. There are no menus, no windows, and no UAC dialogs – nothing that would indicate it is even running Windows. There is also an Administrator mode which shows a standard Vista UI for administrative functions or for development.   

As far as development goes, the good news is that it is all pretty standard stuff. There are two approaches to programming for the Surface. The first is to use the Microsoft XNA Studio platform, the other is to use Windows Presentation Foundation (WPF). Using XNA gives you a little bit more power, as well as access to more of the "lower level" information like raw images from the video feed. Using WPF is a higher-level programming model, and comes with a set of controls specific to the Surface UI model. The nice thing is that all you know about .NET and WPF programming applies to the surface. And from a larger architectural perspective, Surface can tie into any infrastructure accessible to any other .NET-based model. It is just a different .NET UI layer.

The bigger challenge in developing for the Surface is changing the way we think about the UI, and selecting the right solutions. First and foremost, Surface applications are not just a port of a standard Windows UI. Stop thinking about Windows, Icons, Menus and Pointers (WIMP). The surface calls for a completely different models, one that I am just learning. One of the interesting statement I have read describing the Surface model is "the content is the application."
The Surface is more than just a multi-touch platform. Sure, you could implement a multi-touch solution on the Surface exactly the same as a Windows 7 multi-touch solution, but that is only using a subset of the Surface capabilities. The key characteristics of Surface interaction are:

  • multi-user, multi-touch (up to 52 simultaneous touch points)

  • social interaction – multiple simultaneous users, collaborating or working independently

  • 360 degree user interface – users on all sides of Surface at the same time, with UI oriented to support all of them

  • Natural and immersive – like the physical world, only better

  • Support for physical objects integrated into the experience (tokens, cards, game pieces, merchandise)

When it comes to selecting a solution to deploy on the Surface, the two most important keywords are "social" and "immersive". Social, because the best Surface applications are those in which the computer is not replacing human interaction, it is enhancing it. Immersive, because you want the user(s) to forget that they are using a computer, and to only be thinking about what they want to accomplish. The how should be transparent.

Over the coming days and weeks, I will post more about the Surface and what we are doing with it. Hopefully next week I will be able to post a short video. If you have any thoughts or suggestions, I would love to hear them.

The Wonder Of Apple’s Tablet –

The Wonder Of Apple’s Tablet –

Well, well, well….yet another “hype” article for the rumoured (though probably real in some form) Apple Tablet. I must admit, that I am of two minds on the the Apple Tablet (what ever it is will be called). On the one hand, I am very interested in seeing what Apple does with the idea. Will it be a real tablet, or will it just be a big iPhone? Will it run the iPhone OS or a real operating system?

I am mostly concerned simply because it comes from Apple. I personally find Apple to be one of the most troubling companies on the planet. Their closed systems and closed attitude towards the rest of the computing world bother me. Even worse are Apple fans. I dread to see the Apple Tablet merely on the grounds that 6 months later all of Apple fandom will be declaring loudly “how brilliant Steve Jobs is – he invented the Tablet!”.

Back to the article in the Washington Post. The author rightfully asks the question “Why would anyone want a tablet computer?” I personally love them. I have been using them for years (remember this for next Christmas kids – APPLE DID/WILL NOT INVENT THE TABLET PC). I have written several other posts about why I like them, and where I would like them to go in the future. Right now I have two Tablets – one is a slate model which I love. The other is the convertible Tablet given out to attendees at Microsoft PDC . This one has a great multi-touch interface running Windows 7. Its only weakness is pour handwriting support due to interference between touch capabilities and handwriting. In the house we also have two HP Touchsmart convertible tablets. These both support multi-touch and handwriting extremely well, and are well priced at just under $1000 (in Canada).

(Note here that MS already has a multitouch interface that supports gestures, handwriting, and runs a real OS, so is useful beyond just being another gadget.)

Now for the stupidest statement in the Washington Post article (possibly the stupidest tech statement made this year):

“The truth is that most of us don’t understand the allure of a tablet computer because they’ve all sucked up until now.”

Ok, the author just revealed himself to either be a moron, woefully uninformed, or just completely lacking in objectivity (perhaps stemming from the Crunchpad association). There are a number of very good tablets out there (and have been for a number of years). Any of the tablets from Motion Computing are great, though they are not consumer oriented (I have been using an LE1600 personally for 4+ years). The HP tablets have been consistently good. I have also heard great things about Toshiba, Fujitsu, and Dell tablets. The one complaint I have about all of them (except maybe the HP Touchsmart) is that the prices are way too high, but that is improving.

I will say I really want more out of a Tablet, as I said in a previous post. But that does not mean that all of the existing devices suck. Such a broad generalization, is well, just stupid.

Here is another statement from the article:

We’ll be living in a future with Minority Report, Star Trek, and Avatar interactive technology

it is interesting to note that the user interface in Minority Report was actually inspired by another non-Apple device – the Microsoft Surface.

The last quote I will take from this article is

Part of it is that Apple has a sterling record with consumer-oriented products.

Well, seems to me that Apple has failed a few more times than the author mentions. Seems the Mac Book Air didn’t do so well. Going back much further, anyone remember Steve Job’s Newton? Going back even further, Apple could be the dominant desktop OS right now if not for Job’s immeasurable ego back in the 80s (has that changed at all?).

My big concern here is how much of the consumer community reads and believes unsubstantiated drivel like this, and so dismisses anything non-Apple without even looking at it.   

A big part of the blame for this has to go to Microsoft, as well, and their atrocious marketing department. Tablet PCs have been around since 2002, and yet I still get stopped everywhere I travel by folks asking what my tablet is. How is that for getting the word out on one of your coolest technologies? It does not help that the press does not like to write about anything Microsoft because it is not “cool” to support MS.

So please folks, remember this – multi-touch, gesture-based computing is real and available today, and it is not from Apple. In addition, it runs an OS that lets you use everything you have been used to using, and does not lock you in to buying everything you ever want through Apple. And, you can even replace your own battery, unlike most Apple devices 🙂

PS – More hype for the “Apple saves the tablet” community is here. Also there is an older article Why Have Tablets Flopped? Here Are Five Reasons referenced. Of the five reasons quoted, only one is valid – price. Note also that the only pictures they use are of the Newton – the only real failure of the bunch. It is really sad that all of the media writing about tablets seems to have drunk the Apple Koolaid.