Monday, September 1, 2014

Kinect on Raspberry Pi

Recently I decided to take another crack at getting the Microsoft Kinect to work with the Raspberry Pi. Last time I went at it, it was about a year ago. After a grueling failure to get the depth information, that is, what actually makes the Kinect see in 3D, it seemed that the online consensus was that the USB drivers on the Pi were simply not up to snuff when it came to getting all of that information out of the Kinect.

This time, however, there was a new library available called 'librekinect' that supposedly allowed yo to get that depth stream right as a camera device on the pi. This meant it could essentially be used exactly like a USB webcam and, more importantly, be quickly integrated into any Python OpenCV code. After allowing the pi almost a whole day to compile the code and through various different build errors I was able to have the final 'make load' command execute. The result was a sparkling grayscale depth feed out of the Pi. To allow the human eye to better distinguish the different levels I can also feed the image through a OpenCV color map. Check out the VNC remote view of the feed (note that those views are not from the same time):


This will hopefully make it easier for our robotics team to implement the Kinect on small linux platforms in the future or simply some other project I decide to do!


22 comments:

  1. Hi Griffin
    Im working on a project and i would like to make the kinect work on the RPI. So I would be pleased if you can give me more information or links on How to get this work. Im a little new to the RPI but i would be very thankfull if you can help me. Thanks

    ReplyDelete
    Replies
    1. Here is the library I used. Assuming you have the OS all installed (Im using Raspbian), try to follow what the library creator has here.

      Try to work though it, if you have any more questions I'll try my best.

      Delete
    2. Hi, I follow the library creator but im getting problems with the make load process to install the driver. This is what i get when trying to make load.

      root@raspberrypi:/home/pi/librekinect# make load
      make unload_drivers
      make[1]: Entering directory `/home/pi/librekinect'
      sudo sh unload_drivers.sh
      Error: Module gspca_main is not currently loaded
      Error: Module gspca_kinect is not currently loaded
      Error: Module gspca is not currently loaded
      Error: Module kinect is not currently loaded
      make[1]: Leaving directory `/home/pi/librekinect'
      sudo modprobe videodev
      sudo insmod gspca.ko
      sudo insmod kinect.ko
      sudo chown -f -R root:root /dev/video*
      make: *** [load] Error 1
      root@raspberrypi:/home/pi/librekinect#

      I follow the process to compile the kernel that in my case is 3.12.29
      Do you know what can I do to fix it ?? Thanks

      Delete
    3. Before running the make command, try running 'lsusb'. Are the Kinect devices listed there?

      Delete
    4. Yes. It list the 3 devices the camera, motor and audio

      Delete
    5. hi Griffin
      I think I manage to get this to work so i will ask you for another favor. Can you give me some good references to integrated with a Python OpenCV code or if you have one of your own? cause im a little new to this, but i really want to develop something with this. Thanks

      Delete
    6. The python code is simple. The Kinect will act just like a webcam would once its running correctly. You can just do this:

      import cv2

      cap = cv2.VideoCapture(0) 'The zero indicates its the first (and only) camera input

      Then handle the VideoCapture accordingly. The Docs online for OpenCV will help you out a lot. My code was messy, so if you have any specific OpenCV questions I'll help out.

      Delete
    7. Hi Daniel, how did you manage to solve it? I'm getting the same error, i'm just wondering if it's because i'm not using a powered usb hub.

      Thanks

      Delete
    8. You're definitely going to need a powered USB hub. I had all sorts of issues before I used one.

      Delete
  2. hi,
    how can i get skeleton joint?

    ReplyDelete
    Replies
    1. I haven't seen anything about getting the skeleton data off of the Kinect using the Pi, so I don't think its possible yet. Right now we're just getting image streams.

      Delete
  3. Replies
    1. I don't have the set up to give you an exact number, but I recall it being quite usable, like at least 4 fps. Not great but more than enough for most vision applications.

      Delete
  4. Hi, Griffin,

    First of all, my congratulations with your achievement so far. It does demonstrate that you have a lot of skills!

    I do own a raspberry pi 2 and I have installed Ubuntu Mate (15.04) on this one.

    It turned out that with this the pi more or less works like a normal desktop, so I imagine that it would be great for scanning and experimenting with this.

    Because I do not have the experience to really set this up and install/hack files and libraries, I was wondering what your own ideas about this are and I am highly interested in something like a tutorial? Sebastiaan - Amsterdam (gamefabriek [at] gmail [dot] com)

    ReplyDelete
    Replies
    1. Hi Sebastiaan-

      First of all, thanks for the nice words! School's not in full swing anymore, so I'm just getting to this comment :P

      The process I was using was very finicky and hard to set up (definitely a hack like you said), most of which had to do with the not-excellent job to usb drivers did on the original Model B. However, the new Raspberry Pi 2 comes with a bunch of improvements, and it seems that now the pi can readily handle a depth stream from the Kinect. Check out the tutorial I followed here: http://www.mariolukas.de/2015/04/proof-of-concept-3d-scanner-with-kinect-and-raspberry-pi2/

      The first time through I got a build error that didn't make any sense to me. As with most things, I just tried all of the building commands another time through and lo and behold it worked. Its much easier than doing this technique.

      Delete
  5. Hi, I have got librekinect working and also had problems loading the kernel driver until I used an external USB powered hub. I am wanting to get the driver loaded permanently on the raspberry pi so when it starts up librekinect is default driver. How would I do this?

    Thanks

    ReplyDelete
  6. I am more interested in the audio portion of it all. I want to be able to say voice commands and have the kinect be used as a microphone. If I could also get the camera to work as a security cam within Kodi, that too would be ideal. Can anyone help me?

    ReplyDelete
  7. I am more interested in the audio portion of it all. I want to be able to say voice commands and have the kinect be used as a microphone. If I could also get the camera to work as a security cam within Kodi, that too would be ideal. Can anyone help me?

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Have you find a way to use the Kinect mic? I would like to do the same. Thank you.

      Delete
  8. Hi Griff.. Any change you can get kinect working as remote for super mario? I saw a video but can find the code

    ReplyDelete
  9. Hey guys, im having a little trouble setting up my kinect with the raspberry, I don't know what to do my kernel is 4.4.11-v7+

    ReplyDelete