Introduction to NeRF

This is part of my journey of learning NeRF.

1. Introduction to NeRF

What is NeRF

Reference: Original NeRF paper; an online ariticle

在已知视角下对场景进行一系列的捕获 (包括拍摄到的图像,以及每张图像对应的内外参),合成新视角下的图像。

NeRF 想做这样一件事,不需要中间三维重建的过程,仅根据位姿内参和图像,直接合成新视角下的图像。为此 NeRF 引入了辐射场的概念,这在图形学中是非常重要的概念,在此我们给出渲染方程的定义:

那么辐射和颜色是什么关系呢?简单讲就是,光就是电磁辐射,或者说是振荡的电磁场,光又有波长和频率,\(波长\times 频率=光速\),光的颜色是由频率决定的,大多数光是不可见的,人眼可见的光谱称为可见光谱,对应的频率就是我们认为的颜色:

SDF - Signed Distance Function

SDF是一种计算图形学中定义距离的函数。SDF定义了空间中的点到隐式曲面的距离,该点在曲面内外决定了其SDF的正负性。

相较于其他像点云(point cloud)、体素(voxel)、面云(mesh)那样的经典3D模型表示方法,SDF有固定的数学方程,更关注物体的表面信息,具有可控的计算成本。

Features of NeRF

  • Representation can be discrete or continuous. but the discrete representation will be a big one if you have more dimensions, e.g., 3 dim.
    • Actually the Plenoxels try to use 3D grids to store the fields. Fast, however, too much memory.
  • Neural Field has advantages:
    1. Compactness 紧致:
    2. Regularization: nn itself as inductive bias makes it easy to learn
    3. Domain Agonostic: cheap to add a dimension
  • also problems
    • Editability / Manipulability
    • Computational Complexity
    • Spectral Bias

Problem Formulation

  • Input: multiview images
  • Output: 3D Geometry and appearance
  • Objective:

\[ \arg \min_x\|y-F(x)\|+\lambda P(x) \]

y is multiview images, F is forward mapping, x is the desired 3D reconstruction.

F can be differentiable, then you can supervise this.

  • nn本身就是某种constraints,你就不需要加太多handicraft constraints