What is Motion Capture?

(1)

Capturing full body motion

Antoine Kaufmann

antoinek@student.ethz.ch

April 9, 2013 Distributed Systems Seminar 1

(2)

What is Motion Capture?

(3)

What is Motion Capture?

Motion capture is the process of recording the movement of objects or people... In filmmaking and video game

development, it refers to recording actions of human actors, and using that information to animate digital character models in 2D or 3D computer animation. [Wikipedia: Motion Capture]

Sources:http://lukemccann.wordpress.com/motion-capture/

(4)

What is it used for?

(5)

Applications: Filmmaking

Sources:http://www.fxguide.com/featured/the-hobbit-weta/ http://www.ugo.com/therush/avatar-moments-that-give-us-pause-6-gallery http://www.animationmagazine.net/events/ted-ruffles-feathers-at-the-oscars/

(6)

Applications: Game development

Sources:http://www.shipwrckd.com/2012/09/capturedinto-ubisofts-motion-capture-studio-launch- party/ http://gamerant.com/bioshock-infinite-elizabeth-trailer/

(7)

Applications: Gaming

Sources:http://123kinect.com/kinect-sports-season-achivements/25170/ http://pikigeek.com/2012/03/06/peter-molyneux-says-we-need-more-kinect-games-or-we-will-die/

(8)

Applications: Virtual Reality

Sources:http://www.wired.com/dangerroom/2012/01/army-virtual-reality/ http://www.doolwind.com/blog/where-is-virtual-reality/

(9)

Applications: Biomechanics

Sources:http://orthopedics.childrenscolorado.org/our-programs/center-for-gait-and-movement- analysis http://www.motionanalysis.com/html/movement/movement.html

(10)

How can it be done?

(11)

System Types: Mechanical

Potentiometers and exoskeleton

Accurate

Post-processing straight forward

No global root-motion

Restricts range of motion

Sources:http://www.metamotion.com/gypsy/gypsy-motion-capture-system-workflow.htm

(12)

System Types: Inertial

Multiple inertial measurement units (IMUs) to track motion

Gyroscopes or accelerometers

Only relative position

Problems withdrift

Sources:Source:http://www.xsens.com/en/general/mvn

(13)

Problem: Drift

Occurs if only relative measurements are available

Due to inaccuracies of sensors and calculations

Sources:PracticalMotionCaptureinEverydaySurroundings,Vlasicetal.2007

(14)

System Types: Optical

Usually cameras and markers

Markers are

passive (reflective) or

active (controlled LEDs)

Can achieve high level of detail

Sources:http://beforevfx.tumblr.com/image/44047276135

(15)

System Types: Optical

Sources:http://www.cgadvertising.com/pages/posts/vicon-technologies-give-usc-students-hands- on-motion-capture-experience128.php Problems:

Occlusion

Marker-swapping

Requires good contrast and lighting

(16)

System Types

Mechanical

Inertial

Optical

Image based

Magnetic

Acoustic

Radio / Electromagnetic

Motion Tracking: No SIlver Bullet but a Respectable Arsenal by Greg Welch and Eric Foxlin

(17)

What’s wrong with existing systems?

(18)

Problems with existing systems

Heavy instrumentation of user and/or environment

Require line of sight

Limited range

Inaccurate

High latency

Expensive

(19)

Prakash

Prakash: Lignting Aware Motion Capture using Photosensing Markers and Multiplexed Illuminators

Ramesh Raskar (Mitsubishi Electric Research Labs) et al.

(20)

Prakash

Address the following problems:

Expensive high-speed cameras required

Limited number of markers

Marker swapping

Special clothing and lighting required

Sources:http://www.vicon.com/products/cameras.html http://parasite.usc.edu/?p=403

(21)

Prakash

Main Idea: Use most basic optical devices

LEDs as transmitters that are fixed in the scene

Photosensors as tags to be tracked

Cheap components

Sources:http://de.wikipedia.org/wiki/Datei:Uvled_highres_macro.jpg https://de.wikipedia.org/wiki/Datei:Photodiode-closeup.jpg

(22)

Prakash: Receiver Tags

Photosensors, a micro controller and a transmitter

Multiple photosensors used for different measurements

Sources:Prakashpaper

(23)

Prakash: Projectors

Projectors built from multiple LEDs

Labelling space with binary mask

(24)

Prakash: Location

Sources:Prakashpaperandpresentation

Basically binary search

Accuracy doubled by every LED

3 projectors needed for 3D location

(25)

Prakash: Orientation

Multiple fixed beacons

Analog photosensor without lens

Cosine fall-off for estimation of angles

(26)

Prakash: Illumination

Measure RGB illumination

Use one photosensor per color

Sources:Prakashpresentation

(27)

Prakash: Advantages

IDs for markers: no swapping

Not sensitive to lighting conditions

Imperceptible tags in regular clothing

Orientation and illumination information

Faster than regular cameras

(28)

Prakash: Drawbacks

No solution for occlusion

Wires on tags

Not suitable if motion is too fast

Simultaneity assumption: Tags don’t move while lit by projector

(29)

Prakash: My thoughts

Elegant idea using simple means

Nice fit for filmmaking

Basically a distributed system

How does it scale in practice?

(30)

Body-Mounted Cameras

Motion Capture from Body-Mounted Cameras Takaaki Shiratori (Disney Research) et al.

(31)

Body-Mounted Cameras

Heavy instrumentation of environment

System confined to studios

Sources:http://www.creativeplanetnetwork.com/the_wire/2008/07/29/vicon-house-of-moves-builds- new-motion-capture-sound-stage-expands-staff-with-full-service-animation-team/

(32)

Body-Mounted Cameras

Attach multiple outward looking cameras to the subject

No instrumentation of the environment

Use structure-from-motion to recover movements from camera footage

Sources:Body-MountedCameraPaper

(33)

Body-Mounted Cameras: Approach

(34)

Body-Mounted Cameras: Approach

(35)

Body-Mounted Cameras: Approach

(36)

Body-Mounted Cameras: Global Optimization

Why global optimization?

Keep motion smooth and minimize reprojection error

(37)

Body-Mounted Cameras: Advantages

Works outside: no limited range

Motion of skeleton and global root motion

3D structure of scene as byproduct

(38)

Body-Mounted Cameras: Drawbacks

Heavy instrumentation of user

Very long processing time

Problems with motion-blur

Motion in the scene problematic

(39)

Body-Mounted Cameras: My thoughts

Useful as soon as cameras are significantly smaller

Main application: biomechanics research and sports

(40)

Humantenna

Humantenna: Using the Body as an Antenna for Real-Time Whole-Body Interaction

Gabe Cohn (Microsoft Research) et al.

(41)

Humantenna

Heavy instrumentation of environment and/or user

High latency

Portability

(42)

Humantenna

Body as an antenna for receiving EM noise

Minimal instrumentation of the user

Goals:

Gesture recognition

Location classification

Sources:http://en.wikipedia.org/wiki/File:Antistatic_wrist_strap.jpg http://mizzoumagarchives.missouri.edu/2011-Summer/features/the-struggle-for-signal/index.php

(43)

Humantenna

Sources of electromagnetic noise:

Sources:HumantennaPaper

(44)

Humantenna: Gesture Recognition

Predefined whole body gestures

Offline approach (initially)

Manual hints for start/end of gesture

Machine learning for classifying gestures

(45)

Humantenna: Gesture Recognition

Three steps for recognizing a gesture:

1. Segmentation 2. Feature extraction 3. Classification

(46)

Humantenna: Gesture Recognition

Step 1: Segmentation

1. Down-sample (to 244S/s) and low-pass filter→DC waveform 2. Divide waveform into≈100mswindows

3. Check every window if it is active

4. Everything between first and last active window is the gesture

(47)

Humantenna: Gesture Recognition

Step 2: Feature extraction 1. Divide gesture into 5+2 windows

(48)

Humantenna: Gesture Recognition

Step 2: Feature extraction

2. Compute features for each window

Mean of the DC waveform

Apply high-pass and compute root-mean-square

Compute FFT, frequencies 0 - 500Hz.

(49)

Humantenna: Gesture Recognition

Step 3: Classification

1. Use support vector machine to classify gesture

Sources:http://docs.opencv.org/doc/tutorials/ml/introduction_to_svm/introduction_to_svm.html

(50)

Humantenna: Gesture Recognition

Experimental results

Performed experiments in multiple homes and rooms

Different participant used for every home

Trained classifier on 36 examples, tested on 4

µ σ Min Max

92% 3% 86% 98%

Table: Accuracy across homes and participants

A training set of only 4 gestures still results in 84% accuracy

(51)

Humantenna: Gesture Recognition

(52)

Humantenna: Location Classification

Just use the first 0.5 seconds instead of segmentation

Use same DC and RMS features as for gesture recognition

High frequency peaks provide a lot of information about the location

Use frequencies up to 125kHz

Sources:http://activerain.com/blogsview/1022299/viewpoint-midtown-buy-one-get-one-free-what- a-deal-

(53)

Humantenna: Location Classification

Experimental results

Three participants and two homes, two participants per home

8 locations per home, each participant performed gestures in 5

System achieves an accuracy of 99.6% (σ=0.4%)

Classifier works across users with an accuracy of 96%

(54)

Humantenna: Advantages

No instrumentation of environment

Minimal instrumentation of the user

Gesture recognition and location classification in real time

(55)

Humantenna: Drawbacks

By design coarse-grained

Training is necessary

Only works inside the home

(56)

Humantenna: My thoughts

Again a simple and elegant solution

Weak evaluation in the paper

Might be very interesting in connection with a smart phone

(57)

Conclusions: Basics

Prakash

Optical, inside-looking-out

Instrumentation of user and environment Body-Mounted Cameras

Optical, inside-looking-out

Instrumentation of user Humantenna

Electromagnetic, inside-looking-out

Minimal instrumentation of user

(58)

Conclusions: Accuracy

Prakash

As accurate as desired

Fast movements possible Body-Mounted Cameras

Limited by camera performance

Difficulty with fast movements Humantenna

Coarse-grained: only classification of predefined gestures

(59)

Conclusions: Latency and Cost

Latency Costs

Prakash Very low: 1ms Cheap

Body-Mounted Cameras Very high: days Expensive

Humantenna Low: 0.5s Cheap

(60)

Conclusions

Motion Tracking:

No Silver Bullet, but a Respectable Arsenal

(61)

What is Motion Capture?

Capturing full body motion

What is Motion Capture?

What is Motion Capture?

What is it used for?

Applications: Filmmaking

Applications: Game development

Applications: Gaming

Applications: Virtual Reality

Applications: Biomechanics

How can it be done?

System Types: Mechanical

System Types: Inertial

Problem: Drift

System Types: Optical

System Types: Optical

System Types

What’s wrong with existing systems?

Problems with existing systems

Prakash

Prakash

Prakash

Main Idea: Use most basic optical devices

Prakash: Receiver Tags

Prakash: Projectors

Prakash: Location

Prakash: Orientation

Prakash: Illumination

Prakash: Advantages

Prakash: Drawbacks

Prakash: My thoughts

Body-Mounted Cameras

Body-Mounted Cameras

Body-Mounted Cameras

Body-Mounted Cameras: Approach

Body-Mounted Cameras: Approach

Body-Mounted Cameras: Approach

Body-Mounted Cameras: Global Optimization

Body-Mounted Cameras: Advantages

Body-Mounted Cameras: Drawbacks

Body-Mounted Cameras: My thoughts

Humantenna

Humantenna

Humantenna

Humantenna

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Gesture Recognition

Humantenna: Location Classification

Humantenna: Location Classification

Humantenna: Advantages

Humantenna: Drawbacks

Humantenna: My thoughts

Conclusions: Basics

Conclusions: Accuracy

Conclusions: Latency and Cost

Conclusions

Motion Tracking:

No Silver Bullet, but a Respectable Arsenal

Questions?