This is part of my journey of learning NeRF.

## 2.2. Hybrid representations

### Tradeoffs of choosing a proper representation

You may choose one proper representation depending on your own application

### 1. Grid

Input is too huge. Then you need too huge neural network. So, this grid interpolation acts like a "position encoding", which encodes the low dimensional features into high dims.

NeRFusion CVPR22: online!

### 2. point cloud

Cons:

1. To access local points, you need to specifically design the data structure. Otherwise, it is O(n)!
2. Choose different kernels to retrieve nearby points' features. Oftentimes you assume it is local kernel.

### 3. Mesh

Unstructed grids. Compared with point clouds, meshes have connectivity info.

### 4. Multiplanar Images

Something like project a 3D grid into an axis to get levels of planes.

Pros:

1. Compact
2. Very efficient because the hardware and software designs are accelerated to these 2D operations, like bi-linear operations.

Cons:

1. Resolution bias on plane axis: coz it is discrete betweens planes.

This is not very wise in my opinion. It is just a temporary tradeoff given nowadays' technologies. Coz everything will be 3D in the future.

Generate 2D images from different camera views (perhaps). Key point is the tri-plane representation of 3D features.

### 5. Multiresolution grids

Pros:

1. Stable coz you indeed need both low and high resolution info

### 6. Hash grids

$[x,y,z]\text{ coordinates}\rightarrow \text{Hash function()} \rightarrow \text{Fixed size codebook}$ Pros:

1. No matter how big is the original data, you can use a fixed size codebook as the input feature.
2. Can be online!

Cons:

1. May still need large codebooks
2. Features not spatially local. I don't think the hash grid is a good idea if this drawback exists. But isn't there a simple way to generate features with local info remaining?

### 7. Codebook grids

Instead of storing features of points in grids, store a (index to a) code in a codebook. The size of the codebook is fixed, so the overall size can be controlled as much smaller.

cons:

1. To make the indexing operation differentiable, the computing complexity rises here.
2. Using hash is to get rid of the complex data structure, but the indices bring it back.

### 8. Bounding Volume Hierarchies

Commonly used method in computer graphics

### 9. Others (voxel)

• For dynamic nerfs, is there any better hybrid representation? Sure.
• Is there any explicit bias of these hybird representations that we can discover and then design regularization? Sure.