How to use a Kinect for Motion Capture

 

Motion capture is a method used in the film industry to copy the motions of real-world actors into a 3D environment. These movements can then be skinned onto CGI characters resulting in very natural-looking animation that is not achievable with regular keyframing. Some movies that used this technique are The Polar Express and The Adventures of Tintin: The secret of the Unicorn.

But you don’t need any extra super-expensive finicky tools to do this! You can use an Xbox Kinect! This webpage will show you how to make your own very natural looking animations with very inexpensive gear.

Video Tutorial:

Seb8tian graciously created this video tutorial from this webpage. Go subscribe! Make sure to enable Closed Captions (CC) on the video.

 

Things you will need:

  1. A 64-bit computer running Windows 7 or higher with USB 2.0 or higher
  2. At least 6 GB free of hard disk space
  3. a CPU and GPU that meet system requirements for Maya
  4. A 3-button mouse (Not required, but you can’t pan without one)
  5. An Xbox Kinect 360 sensor. (The instructions for an Xbox One are different and I don’t know how to do it. You can try with these steps but success is not guaranteed.)
  6. Kinect 360 USB adapter (if your Kinect does not have a standard USB connector)
    1. An adapter you could use: https://www.amazon.com/Microsoft-Xbox-360-Kinect-Sensor-Adapter/dp/B00JVI6BVY
  7. An open space free of obstructions (10ft*10ft recommended, 5ft*5ft minimum)
  8. Once you have gathered all of these parts you are ready to proceed!

The software we will install:

A complete list of all the software we will install just as a heads-up. No surprises! (Don’t install any of this yet)

  1. Autodesk Maya 2016 or higher (Can be acquired for free legally by making a student account on Autodesk.com, more info about this later)
  2. Autodesk MotionBuilder 2016 or higher (make sure to use the same version as Maya!)
  3. Kinect SDK 1.8 (or 2.0 for Xbox One)

Instruction Guide

    Installation Instructions

  1. Download and install the Kinect SDK 1.8 (Do not plug in the Kinect first!)
  • Run the installer to install the SDK
  • If you haven’t already, create an Autodesk student account
  • Install the student version of Autodesk MotionBuilder
  • Repeat the process for Maya (make sure you set the same version!)
    1. www.autodesk.com/education/free-software/maya
  • Plug  your Kinect sensor into any power USB 2.0 or higher port on your computer
    1. Plug the power cable into a wall socket if your Kinect requires a power outlet
    2. Wait while your computer automatically installs any required drivers for your computer. Do not unplug the Kinect during this process!
  • Congratz! you have made it through all the installation steps! Now on to the Operation instructions

        Operation Instructions – Setup

    If you need help finding things here is a link to an Imgur Album with screenshots. http://imgur.com/a/hLZVI

    1. Plug in your Kinect and place it in a spot so it is about 4 feet off the ground and level horizontally. Do not force the sensor arm backward or forwards! You will damage the motor!
    2. Launch MotionBuilder. On the controls popup, in the first checkbox, select [Maya] (This makes it easier later on because the 3D controls will be the same for both programs)
    3. Go to the [Asset Browser], expand the [Templates] folder, and click [Devices]
    4. Drag [Kinect 1.0] onto the 3D grid
    5. Under [Model Binding] click [Create]. A yellow joint skeleton should appear in the view
      1. Note: for MotionBuilder 2017 or higher, you will need to go to [Characters] -> [MocapDeviceCharacter] and change [Character Solver] to [MB Character Solver]
    6. Click the Online button and wait until it turns green. If a dialog for voice control pops up, press [No]. The Kinect should auto-level its sensor if necesary.
    7. Under [Setup] click [Setup Recording]. A man should appear in the viewport.
    8. [Optional] Delete all of the mesh for the man. You do not need it unless you don’t have any rigs to retarget the motion to. But keep the white skeleton because it is essential.
    9. Click [Calibration], and then [Ok]
    10. Stand about 6 feet away from the Kinect and directly in front of it. You will notice the yellow skeleton on the screen will begin to match your movements.
    11. Put your arms out in a T-Pose (look at the white skeleton for reference)
    12. When the Kinect has calibrated successfully, you will hear a sound effect and the white skeleton will start moving with the yellow one.

       Capturing Motion

      1. Complete Setup mentioned above.
      2. In the timeline, press the circular Record button (If you have another person to operate computer, do step 3 first and have the other person do steps 1 & 2)
      3. Press the triangular Play Forwards button
      4. Stand about 6 feet in front of the Kinect and perform (make sure the player is running while you perform)
      5. Press either Play Forwards or Stop (square) to end the recording
      6. Wait until progress bars stop appearing.
      7. Move the time scrubber to the beginning of the timeline and play back your recording. Notice that only the white skeleton is moving — this is what we want.
      8. [optional] Turn the camera around to explore all areas of the recording. Congratulations! You have captured real-world movements into 3D space!

        Trimming out bad recordings

    If the skeleton glitches a lot at the beginning and end of the clip, we trim them out now.

    1. Go to the Story tab.
    2. Drag the time slider to where the skeleton stops shaking.
    3. Click on the edge of the clip and drag it over to the time slider. It should snap to the location of the time slider.
    4. Drag the time slider to just before the animation becomes shaky again at the end of the clip.
    5. Click on the edge of the clip and drag it over to the time slider. It should snap to the location of the time slider.
    6. Click in the middle of the clip and drag it to the left so it lines up with frame 1.
    7. Now play back the animation. The trimmed out portions will be gone.

        Sending to Maya

    Maya is an advanced 3D animation program that was used to make movies such as Frozen, Wreck-it-Ralph, Cloudy with a Chance of Meatballs, and much more. The only reason why we installed MotionBuilder is that Maya has no motion capture tools. Instead, the artists use keyframing to animate the characters for a more cartoony look.

    1. Deselect everything in MotionBuilder. Simply drag in the 3D view to make a dashed-line rectangle. Make sure that rectangle is not covering anything in the scene and release the mouse button.
    2. Go to File -> Send to Maya -> Send as New Scene
    3. Click [OK]. This is why we deselected everything. That way MotionBuilder won’t leave out anything important when sending the scene.
    4. Wait for Maya to launch. When the popup shows up, press [Don’t save]
    5. Use the scroll wheel to zoom out. Keep zooming out until you see the Mocap Skeleton.
    6. You can now close MotionBuilder. At this point, it doesn’t matter if you save it or not. You can if you want to.
    7. In Maya, open up the Outliner. (In Windows -> Outliner)
    8. [Optional] You can delete MocapDeviceReference because it doesn’t do anything.
    9. Click on master. This control allows you to scale the skeleton and move it around, without changing animation curves. I prefer to scale it down for better visibility. This is optional but highly recommended.
      1. Press ‘R’ on your keyboard to bring up the scale controls. (W for translate, E for rotate, R for scale, Q for no controls)
      2. Middle-Mouse drag to the left over the 3D view to scale the skeleton down so that its feet fit inside 2 grid squares
      3. While hovering over the 3D view, press ‘F’ on your keyboard to re-center the camera
    10. You can now close the Outliner.
    11. Right-click the timeline, go to Playback speed -> Real-time to play back at 24 frames per second.
    12. Play back your animation. Congratulations! You have successfully captured real-world motions into a computer!

    To clone these movements onto a CGI character, you can follow this guide.

    If you want any of your existing rigs to copy the motions of the mocap skeleton, use HIK Retargeting for HIK equipped rigs, or simply use Parent and Orient constraints from the mocap skeleton to controllers on your custom rig.

    Thank you for reading this. If you want tutorials on other things, just let me know!

    And if you know how to do this with an Xbox One Kinect, please tell me so I can add it to this page.

     

     

    2
    Leave a Reply

    avatar
    1 Comment threads
    1 Thread replies
    0 Followers
     
    Most reacted comment
    Hottest comment thread
    1 Comment authors
    Johntham Recent comment authors
    newest oldest most popular
    Johntham
    Guest
    Johntham

    Hi, I having an error in the Motion builder on the model binding, when click the Online red button, it show “couldn’t Start Devices”. My Kinect devices is Xbox 360. Win 8.1, kinect SDK 1.8.
    Thanks you

    Johntham
    Guest
    Johntham

    It work now. solved by reinstall the kinect driver.