Residue Encoding:

For each residue map, we also compute a foreground mask bit to identify pixels which belong to the actor in each view. The mask is obtained by thresholding the input depth values. The residue values are represented using a sign-magnitude format to facilitate bit-plane encoding of residues. We explore two compression schemes for the residues.

Since residues are correlated spatially and temporally and are otherwise like videos for each view, we can encode them using a standard video coding scheme like MPEG. This gives $m$ residue movies and has the random access properties and bit-rate control as the underlying video coding scheme. The video scheme may incur high reconstruction costs.

The other method uses bit-plane encoding of the residues, using as many bits as client demands. The scheme (Figure 3), inspired by video coding, is as follows.

1. Exploit temporal correlation by computing residue differences as $D_i = R_i - R_{i-1}$, where $R_i$ is the residue for frame $i$.
2. Encode the residue map as blocks that contain one $\bf R$ frame of residue values and several $\bf D$ frames of residue differences. A block is a random-access unit and its length is determined by the requirements for random access.
3. Encode R frames with $K$ most-significant bits of the residues and the D frames with $k$ most-significant bits of the residue difference.
4. Encode mask bits using the JBIG algorithm.

Different quality points can be obtained by varying $K$ and $k$, which can be varied online in a real-time client-server setup. We also use an incremental representation to exploit any additional available bandwidth. We send the next (i.e., $K+1$th) bit-plane of the residue as an $\bf I$ frame when this happens. The value received is added to the current R frame, thus improving the quality of all frames till the end of the block. As shown in Figure 5, an extra bit at I frame provides the increment in the bit representation for the following frames, thus there on increasing the quality of the movie.

$$R_i = R_0 + D_1 + D_2 + .... + D_{i-1}$$ (1)

$$R_i^{'} = R_0 + D_1 +\cdots+ D_j^{'} + D_{j+1}^{'}\cdots+ D_{i-1}^{'}$$ (2)

Equation 1 shows $R_0$ encoded with $K$ bits and subsequent $D$ frames encoded with $k$ bits. When 1 extra bit is added to the $j^{th}$ $D$ frame as shown in equation 2, $D \rightarrow D^{'}$, and the subsequent $D$ frames also get better representation. Thus, the quality of $R_i^{'}$ gets more than $R_i$.

Figure 5: The structure of a block with R frames, D frames and optional I frames.