Point Clouds & 3D Vision
A point cloud is not an image
A point cloud is a set of 3D points, each typically given as (x, y, z) coordinates and sometimes additional attributes like color or reflectance intensity. Point clouds are the native output of sensors like LiDAR (which measures distance by timing reflected laser pulses) and depth cameras. Unlike an image, a point cloud has no grid structure — points are not arranged in neat rows and columns, there may be a different number of them in every scan, and critically, the order in which points are listed is arbitrary and carries no meaning. This breaks the core assumption CNNs rely on: a fixed, regular grid where neighboring array positions correspond to neighboring locations in space.
Two naive approaches, and why they fall short
One tempting approach is to convert a point cloud into a 3D voxel grid — essentially a 3D version of pixels — and run 3D convolutions over it, directly reusing CNN machinery. This works but is expensive: most of 3D space is empty, so a voxel grid wastes enormous compute and memory on empty cells, and fine resolution quickly becomes impractical. Another approach projects the 3D points onto 2D images from multiple viewpoints and runs ordinary 2D CNNs on each view. This avoids the empty-space problem but loses 3D structure and can struggle with occlusion between views. Both approaches motivated a more direct alternative: process the raw points themselves.
PointNet: the permutation invariance trick
PointNet was a landmark architecture that processes a point cloud's raw (x, y, z) points directly, with no voxelization or projection. Its central challenge is that a neural network's output should not change if the same points are simply listed in a different order — the network must be permutation invariant. PointNet solves this elegantly: it applies the same small neural network independently to every single point (producing a per-point feature vector), then combines all those per-point feature vectors with a symmetric function — max pooling across all points — to produce one feature vector for the whole point cloud. Because max pooling (and sum, or average) doesn't care what order its inputs arrive in, the whole pipeline is naturally invariant to point ordering.
Note
What point clouds are used for
LiDAR-based point clouds are central to autonomous vehicle perception, where they provide direct, accurate distance measurements independent of lighting conditions (unlike cameras, LiDAR works at night and isn't fooled by shadows or glare). Common downstream tasks include 3D object detection (drawing 3D boxes around cars, pedestrians, cyclists), semantic segmentation (labeling every point by what surface it belongs to — road, building, vegetation), and simultaneous localization and mapping, where point clouds accumulated over time build a 3D map of an environment.
Approaches to learning from 3D point data
| Approach | How it handles irregular points | Main weakness |
|---|---|---|
| Voxelization + 3D CNN | Buckets points into a regular 3D grid | Wastes compute on mostly-empty space |
| Multi-view 2D projection | Projects points onto several 2D images | Loses 3D structure; occlusion between views |
| PointNet (direct, per-point MLP + pooling) | Processes raw points with permutation-invariant pooling | Weaker at capturing fine local geometric structure than later variants (e.g. PointNet++) |
Note
What's next
Point cloud processing is one piece of the broader problem of building and tracking a 3D understanding of the world over time. Continue to SLAM basics to see how robots and AR devices combine sensor data — including point clouds — to simultaneously map an environment and track their own position within it.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.