# NeRF Hybrid representations

This is part of my journey of learning NeRF.

## 2.2. Hybrid representations

### Tradeoffs of choosing a proper representation

You may choose one proper representation depending on your own application

### 1. Grid

Input is too huge. Then you need too huge neural network. So, this grid interpolation acts like a "position encoding", which encodes the low dimensional features into high dims.

NeRFusion CVPR22: online!

### 2. point cloud

Cons:

- To access local points, you need to specifically design the data structure. Otherwise, it is O(n)!
- Choose different kernels to retrieve nearby points' features. Oftentimes you assume it is local kernel.

### 3. Mesh

Unstructed grids. Compared with point clouds, meshes have connectivity info.

### 4. Multiplanar Images

Something like project a 3D grid into an axis to get levels of planes.

Pros:

- Compact
- Very efficient because the hardware and software designs are accelerated to these 2D operations, like bi-linear operations.

Cons:

- Resolution bias on plane axis: coz it is discrete betweens planes.

This is not very wise in my opinion. It is just a temporary tradeoff given nowadays' technologies. Coz everything will be 3D in the future.

Generate 2D images from different camera views (perhaps). Key point is the tri-plane representation of 3D features.

### 5. Multiresolution grids

Pros:

- Stable coz you indeed need both low and high resolution info

### 6. Hash grids

\[ [x,y,z]\text{ coordinates}\rightarrow \text{Hash function()} \rightarrow \text{Fixed size codebook} \] Pros:

- No matter how big is the original data, you can use a fixed size codebook as the input feature.
- Can be online!

Cons:

- May still need large codebooks
- Features not spatially local. I don't think the hash grid is a good idea if this drawback exists. But isn't there a simple way to generate features with local info remaining?

### 7. Codebook grids

Instead of storing features of points in grids, store a (index to a) code in a codebook. The size of the codebook is fixed, so the overall size can be controlled as much smaller.

cons:

- To make the indexing operation differentiable, the computing complexity rises here.
- Using hash is to get rid of the complex data structure, but the indices bring it back.

### 8. Bounding Volume Hierarchies

Commonly used method in computer graphics

### 9. Others (voxel)

- For dynamic nerfs, is there any better hybrid representation? Sure.
- Is there any explicit bias of these hybird representations that we can discover and then design regularization? Sure.