What I’ve learned after 100 days of 3D Scans with the iPhone 12 Pro LiDAR

9 min readApr 11, 2021

Updated on Monday June 7 2021:

I’m now working at Unity3d.com as a UX Design Lead, and if you followed today’s WWDC21, you’ve probably heard about the new Object Capture API introduced with RealityKit 2. After focusing on LiDAR captures, Apple is now enhancing the photogrammetry scanning experience on iPhone.

Unity was mentioned during the event as one of the many developers that are already using Object Capture to unlock entirely new ways of creating 3D content. Members of my team had a great experience partnering with Apple to bring support to the Unity MARS Companion App. Learn more in this blog.

Apple introduced new Developer Tools and technologies to create even better apps. Xcode Cloud will help developers build apps even faster, and new App Store features will enable developers to reach more users. You can learn more on the subject in the official press release.

But first… What is LiDAR?

LiDAR stands for Light Detection And Ranging and it measures how long it takes light to reach an object and reflect back.
— Francesca Sweet, Product Line Manager, iPhone

Apple has adopted this technology for the iPhone 12 Pro (and 2020 iPad Pro) and with the machine learning and depth frameworks of iOS14, it builds a precise depth map of the scene, enabling object & room scanning, precise placement of Augmented Reality (AR) objects, people occlusion, and photo & video effects. LiDAR also improves accuracy to focus, reduces capture time, and enables stunning night mode portrait shots.

iPhone 12 Pro: LiDAR for AR & Photography

While it is often used to scan buildings and other objects, the scanner on the iPhone 12 Pro and iPad Pro isn’t yet accurate enough to scan objects precisely, or work with a 3D printer. It is, however, good enough for 3D modelling purposes since all the proportions are respected. It can also scan the distance to surrounding objects up to 5 metres away and works both indoors & outdoors.

Storyline inspired by the Hero’s Journey

1/7. Call To Adventure

It all began with alban denoyel — CEO of Sketchfab.com, the largest platform for immersive & interactive 3D content. When he created the #1ScanADay challenge on Twitter & Sketchfab, a few other folks joined the movement, including some that live in my city (Montreal, Canada). Quickly the impact on the community was felt worldwide.

Whether they’re related to Art (#Inktober), Design (#36DaysOfType), or even Development (#100DaysOfSwiftUI), I always enjoyed participating in challenges to increase my knowledge & skills on subjects that I’m passionate about. This is why I didn’t think twice before getting involved.

I love journaling and trying new routines, so I naturally fell in love with the idea of using a LiDAR scanner to capture my world every day. One 3D scan at a time, ultimately creating a volumetric journal of spatial memories.

2/7. Crossing the Threshold

I already owned a 2019 iPad Pro — which didn’t include a LiDAR scanner — however, nothing would have stopped me from joining the movement. Since I had already done months of research and considered myself involved, I had no other choice but to purchase the iPhone 12 Pro. Although I was committed to learning more, I would have never imagined that my interest would last more than 100 consecutive days.

It took me approximately three weeks to receive my new iPhone. As soon as it arrived, I installed and experimented with every available iOS 3D scanning app. Although I was not a big proponent of iPhone accessories, the first few months of the challenge took place in the Winter. We went through a few snowstorms and holding a piece of metal at minus twenty degrees (celcius) was neither fun for me nor the phone, so I ended up buying a protective case. I also needed a way to reach complex angles in order to be ready to scan even the hardest locations, so I bought my first selfie stick.

If you’re interested, you should also take a look at the Pro Tripod Mount with MagSafe from Moment, which received a lot of good feedback from the community (I own many other accessories from them and I must say that I appreciate their products quality).

3/7. The Road of Trials

Before taking a final decision on which apps are the best, I often do a competitive analysis of all the available apps. This allows to compare their features, to identify patterns, and to consider their pros and cons.

Here is an exhaustive list of all the 3D scanning apps I’ve used during the past 100 days (some have even been released less than six months ago). The majority of them offer a direct “Export-to-Sketchfab” feature. I’ve organized them in three different groups:

Mesh Texture: Orange
Points Cloud: Red
Volumetric Videos: Blue

The ones I’ve used the most and that were available since the very beginning of the challenge are Scaniverse & Polycam. Their main difference is that Polycam is better for large flat surfaces such as indoor walls or outside murals for example, while Scaniverse gets better results for objects & textures in general. Both have great cropping features, as well as unique visuals to showcase the scans being processed.

Here is a comparative table made by members of the community, translated from a Japanese tweet originally:

Source: https://twitter.com/iwamah1/status/1380532437403365381

The only apps that aren’t listed from my personal selection are:

Canvas, which exports to many popular formats such as SketchUp, and even works with iPhones without LiDAR (but on the 12 Pro, the results are even better).
ScandyPro, which offers a complete set of editing options to decimate, smooth, crop and export.
Record3D which is the most used app for LiDAR volumetric video capture (and with the arrival of Volu, we will have a new app to play with and apply dynamic shaders to our volumetric captures. I’m actually meeting with the creators next Sunday, Apr 18th at 1PM EST on the XR Family Club on Clubhouse).

Rememory, which is still in beta and allows to playback 4D captured video in AR, resulting in very interesting scenarios, such as this one:

4/7. The Abyss (Death & Rebirth)

I won’t lie, there were some days where I felt uninspired. Because of the current COVID-19 situation, the opportunities to go outside & scan the world diminished. However, staying home forced me to practice in a controlled environment which increased the quality of my 3D scans. It also allowed me to spend some time exploring Trnio, a photogrammetry based app. Here is a model captured with it:

5/7. Metamorphosis

Then something happened, on Day 69 I posted a tribute to B.I.G’s remembrance day by scanning the murals in my barbershop’s restrooms (the coolest in town), and quickly it gained momentum on Twitter. Later that week I was honored to learn that Sketchfab had selected my 3D scan for their staffpick. It motivated me to continue uploading my daily scans, and to seek out more awesome locations in my city. The month of February was focused on Street Art (for the #LiDARScanChallenge by Sketchfab) and Montreal is filled with them, so a lot of the pieces from my collection feature beautiful graffiti and murals from Montreal.

Sketchfab Staffpick

6/7. The Ultimate Boon

I was able to import my barbershop LiDAR scan into Blender3D and create a brand-new piece of art with it:

Link to video.

The next step was to make it interactive!

A few days before doing so, I received an early access to the Looking Glass Portrait, and connected it to the proper Unity and Blender plugins to create some futuristic prototypes based on my LiDAR Scans.

Here is the result:

Link to video.

7/7. Returning the Boon

Participating in this challenge is more about the community building than the scans. It’s about waking up every morning excited to see what other people living in different cities are doing and about discovering new ways of doing the same thing better while sharing tips and finding solutions with others.

In that same vein, I wanted to give back to the community with this article in order to inspire everyone reading it to join the challenge: If you start today, you too will eventually reach 100 days of 3D scans!

Wrapping up

I will share some tips & resources that can help you along your journey:

You can create a free account on Sketchfab today, and upload and host all your models for free under the public license.
Chances are all future iPhone models this year will have a LiDAR sensor, so it’s better to get started with the technology now — both hardware & software — while #LearningOutLoud with the community.
AR Glasses benefit greatly from LiDAR to enable powerful computer vision and to transform every single user into a Digital Twin creator. Understanding that ecosystem early is a tactical advantage. Look into the Hololens 2 recent news, and the rumors about the Apple headset.
There are many people I could mention here in this article, but the list would be too long and I don’t want to omit anyone. Instead, you can explore the #1ScanADay challenge on Twitter and Sketchfab to find all participants and learn from different perspectives.
For more practical tips and tangible examples, I highly recommend you read the three articles that inspired this one, authored by alban denoyel (link), Patricia Esteban (link) and Kyle Pace (link).

Are you starting to feel ready to create your first 3D Scan?

Think about what you want to capture. Is it a space? An object? Try not to scan objects that are too small or too big. A shoebox is a good reference for the minimum size.
Don’t rush it. Scanning is clearly faster than photogrammetry, but going too fast might result in poor texture, holes in the mesh, or blurry output. The speed matters, so keep it constant.
Don’t scan the same area twice. Instead, try to move your camera around the scene in a continuous manner and make sure you capture all angles.
Take advantage of daylight. Because of the very nature of LiDAR (it’s in the name), lighting affects the end result a lot.
Avoid objects that reflect light such as glass, windows, mirrors, water, most metals, etc.
Prioritize round shapes and soft edges as well as light colors (avoid dark surfaces).
Spend time cropping your scans. Most of these apps have good native mobile tools. Use MeshMixer or MeshLab to go even further.
Edit the 3D settings of your scans uploaded on Sketchfab and create annotations! (I like to add post-processing filters such as sharpness, depth of field, screen space reflection, SSAO, vignette, and a little bit of chromatic aberrations and bloom).