Design started from Sixth Sense Device: from the guy who started this movement Pranav Minstry, if this doesn’t get you fired up. I don’t know what will.
http://sixthsense.github.com/sixthsense/

1. Removing Sixth Sense Application while not removing the “spirit”.  There is  much work to do.  I would rather write these things as  libraries so that other applications outside of the SS  can use it.    “Separate Augmented Reality Libraries”. By combining each of these libraries would cover everything demonstrated and a heck of a lot more. These are complex when you see them on screen. However, if you sit down and look at what you need for each step, then it is easy.
      a. Speech (Includes text to speech, speech to text, sounds.)
      b. OCR Libraries (Using EGNU: http://www.emgu.com/wiki/index.php/Main_Page)
      c. BarCode, ISBN and QR code Readers.
      d. WebQueriers – Bing or Google about one of the returns from the OCR, QR, barcode or ISBN captured.
      e. Video Flip or Unflip to support mirror or no mirror.
2. Removing need for color fingers, due to real life testing I have found color markers to be undependable. Using Kinect and depth motion I have found the devices to be too bulky. Replacing with www.leapmotion.com . (Kinect 2.0 (2015?) or for Kinect phone could change my mind).
3. Creating reusable interchangeable libraries for all developers to use rather than one program that must be open in windows. This is where myself and most of the sixth sense team differ. I have passion for Pranav’s IDEA.  However, I am about “what fits the puzzle best”.  I honestly believe he would have the same outlook.  Interacting with real life is the goal. 4. Removing any need for any specific hardware. Since Leap is the new “Mouse” and promises to work on apple and Linux devices, then NOTHING ties this device to any OS, to any HARDWARE, furthermore no need for Direct X. Finally, using the API’s I will be building you should be able to create the functionality we all saw in his videos on any device in any program.