A studio covered with chroma-key material with suitable lighting rig
A camera filming one or more actors [real action, foreground action]
Virtual Scenery [The Background : 2D/3D, Real-time computer generated, pre-rendered, normal video]

Requirements

Match of the camera perspective of the real action with virtual components
Proper handling of occlusions
Match of lighting among the real and virtual components
Proper handling of shadows
Proper handling of reflections

Requirements for an ideal system

Unconstrained movement of the camera over an area up to 800 sq meters.
Work with a wide variety of camera mountings [manual and robotic pedestals, cranes or hand held cameras]
Positional accuracy of the order of +- 1 mm
Angular accuracy +- 0.01 degrees
Minimum delay
No constraints on the scene content
No constraints on the studio environment

High-End Interaction Between Real and Virtual Content

Animated virtual components
Occlusions of scene components by other components
Shadows
Reflections
Moving the viewpoint away from that of the real camera
Addition of virtual lighting

Full and realistic looking optical interaction requires a 3D description of
the real and virtual scene - 3D Models for Humans

Camera Tracking System

Parameters

camera panning
tilting
zooming
rotation about the optic axis
translation

Available Systems

Mechanical Sensors

Practical but have limitations
Operational constraints
Can not be used on existing film stock
Calibration Difficulties

Image Analysis Based Systems

Estimation of 3 parameters: pan tilt and zoom
Use of key signal and blue background
Requires at least two tones of blue both allowing a key signal to be generated

The ELSET Camera Tracker

Motion Estimation Algorithm

Based on differential estimation of translation and scale change
Computation of Spatial and Temporal luminance gradients at many points in successive images
A set of over-determined linear simultaneous equations [one per image point]
Least squares solution

gt = gx xt + gy yt + (m – 1). (gx x + gy y)

gx , gy , gt horizontal, vertical and temporal luminance gradients
xt , yt horizontal and vertical translation
m scale change
x, y pixel coordinates

The System

Prefiltering due to the assumption of smoothly varying luminance [inherent in the method of differentials]
Subsampling to reduce computational requirements
Use of a reference image to prevent the accumulation of error
Reference image is changed when the degree of overlap between the reference image and the current image is small

Limitations

No good for hand held cameras
No camera movement in the studio
Part of the blue texture should always be visible
Limitations with wide-angle lenses
can not handle geometrical distortion caused by camera optics