Breaking News

10 great applications for 3D ToF camera sensors on drones

Flash lidars (also known as ToF camera sensors) on drones or ground systems have many powerful uses, including object scanning, measuring distances, indoor navigation, obstacle avoidance, gesture recognition, tracking objects, measuring volumes, reactive altimeters , 3D photography, augmented reality games, etc.

What is a ToF camera sensor?

Time-of-Flight (ToF) is a high-precision distance mapping and 3D imaging technology that measures the distance between a sensor and an object based on the time difference between the signal’s emission and its return to the sensor after being reflected by the object – Time-of-Flight 3D Depth Sensor A very short pulse of infrared light is fired, and each pixel of the camera sensor measures the return time.

A ranging imaging ToF camera is a very advanced lidar system that replaces a standard point-by-point laser beam with a pulse of light or flash to achieve full spatial perception. The camera senses the time it takes for light to return from any surrounding object, combines it with video data, and creates a real-time 3D image. It can also be used to track facial or hand movements, map rooms, remove backgrounds from images, and even overlay 3D objects in images.

Advantages of ToF Sensor Technology

As an emerging technology, ToF or 3DFlashLIDAR offers many advantages over traditional point (single pixel) scanner cameras and stereo cameras, including:

Simple

Compared to stereo vision or triangulation systems, the entire system is very compact: the lighting is only placed next to the lens, while other systems require a certain minimum baseline.

In contrast to laser scanning systems, no mechanical moving parts are required.

One of the big advantages of time-of-flight cameras is the ability to composite a 3D image of a scene from a single shot. Other 3D vision systems require more images and motion.

efficient

Extracting distance information from the output signal of the ToF sensor is a straightforward process. As a result, the task uses only a small amount of processing power, whereas for stereo vision, implementing complex correlation algorithms requires more energy and more processing power.

Object detection is also a simple process after extracting distance data, as the algorithm is not disturbed by object patterns.

speed

A time-of-flight 3D camera can measure distances across the entire scene in a single pass. Since the camera can achieve 160 frames per second, it is ideal for use in real-time applications.

Price

Compared to other 3D depth range scanning technologies such as structured light camera/projector systems or laser rangefinders, ToF technology is very cheap.

ToF is one of the few technologies known as range imaging. Sensor devices used to generate range images are sometimes referred to as range cameras. Other ranging techniques are stereo triangulation, light sheet triangulation, structured light, interferometry, and coded apertures.

ToF camera sensor technology has a huge advantage over other technologies because of its ability to measure distances within a complete scene in a single shot. And ToFFlash lidar does not require a scanner, which means that the entire scene is captured by a single light pulse (flash) rather than point-by-point by a rotating laser beam. ToF cameras use a dedicated image sensor to capture the entire scene in three dimensions, so no moving parts are required.

10 great applications for 3DToF camera sensors on drones

①、UAV ToF sensor for indoor navigation

For drones and even robots, successfully and safely navigating indoors presents a number of technical challenges. In robotic mapping, simultaneous localization and mapping (SLAM) is a computational problem encountered when constructing or updating a map of an unknown environment and simultaneously tracking the agent’s position within it.

In simple terms, SLAM is concerned with the mobile robot using the map to navigate the environment while building a map of the unknown environment.

The drone ParrotARDrone, which uses a time-of-flight depth-sensing camera, enables indoor flight without GPS through its camera. The basic mechanism it uses is to take two pictures on the ground beneath the vehicle. The second photo is then compared to the first and the offset is calculated. This offset indicates how much the vehicle has moved between the two photos.

This technology, combined with a downward-pointing time-of-flight 3D depth sensor, provides a very accurate and low-cost mechanism for indoor horizontal and vertical spatial positioning.

②、UAV ToF sensor for 3D shape scanning

Drones are currently used to create accurate 3D models of buildings and monuments. In 2015, a drone successfully created a 3D image of the Christ the Redeemer statue in Brazil. Before this time, there was no accurate model of this statue anywhere. The Christ the Redeemer statue was surveyed using 3D photogrammetry, which meant drones had to fly around the statue, taking thousands of pictures and then stitching them together using 3D mapping software.

Other 3D shape scanning techniques use specialized and complex sensors, such as structured camera/projector systems or laser rangefinders. Even though these produce high-quality data, they are expensive and often require expert knowledge to operate.

Harvard University has proposed an extremely cost-effective solution for 3D shape scanning using ToF cameras. Now, the solution is ground based. However, this solution will also work when mounted on a drone.

With more innovation and investment in this area, we are likely to see ToF cameras become a cost-effective solution for scanning large objects and monuments.

③、UAV ToF sensor for agriculture

In agriculture, drones equipped with multispectral sensors could allow farmers to manage crops and soils more efficiently. These multispectral imaging agricultural drones use remote sensing technology in the green, red, red-edge, and near-infrared bands to capture visible and invisible images of crops and vegetation.

Data from multispectral sensors is very useful for preventing diseases and pests. They also greatly help farmers calculate the correct amount of water, fertilizer and spraying.

ToF cameras can be part of an agricultural multispectral solution in a number of ways.

ToF cameras excel at quickly measuring volumes. They can capture a complete scene in one shot. ToF3D sensors are ideal for measuring crop density and providing mulch and manure accumulation data.

To get accurate data on crops, the more stable the drone flies, the more accurate the data, images and film will be. A smoother flight can improve the data collection process and also help extend flight autonomy.

ToF sensors such as the TeraRangerOne can act as a very accurate reactive altimeter. Compared to a laser system with a very narrow field of view, the TeraRangerToF camera can provide a smoother and more stable relative altitude when the drone flies alternately over the ground and over the vines.

④、UAV ToF sensor is used to measure volume

FlashlidarToF cameras are being used to measure volumes, such as the packing space of boxes in factory or warehouse environments.

Time-of-flight sensor technology is ideal for this application for three main reasons. It is more cost-effective than other 3D scanning technologies such as fringe pattern projection and 3D laser displacement sensors; like 3D laser displacement sensors, it does not require an axis of motion to scan boxes; it operates at speeds up to 40 per second Frame count, providing real-time volume measurement as the box fills with packaging material.

Outdoors, drones or cranes with ToF cameras are able to measure the volume of inventories or trucks full of raw materials. A drone fitted with a ToF camera can fly over all or part of a storage area and quickly calculate the amount of material deposited or the total load on a truck.

WhitakerContracTIng uses drones with ToF cameras to measure their inventory, saving 22% annually. They save a lot of money while measuring inventory 2x more often. It now takes 75% less time to measure inventory than before.

⑤、UAV ToF sensor for collision avoidance system

In 2016, we first started seeing small consumer and commercial drones appearing on the market that could detect obstacles. Depending on the drone, its sensing technology and software programming, the drone can take actions to stop and hover when it detects an object, or it can navigate around it.

One of the many functions of ToF cameras is for obstacle avoidance and collision avoidance.

While researching some of the latest drones on the market, I could find 3 drones that use ToF for obstacle avoidance. One of the drones only uses ToF cameras to detect objects above the drone. Another drone uses ToF cameras in combination with other sensors to avoid obstacles.

Since this is fairly new technology, with more developments, I believe we will see more drones using ToF flash lidar camera sensors for obstacle detection in the future.

⑥, gesture recognition ToF on drones

Generally speaking, “3D Time of Flight” usage can be divided into “gesture” and “non-gesture” categories. Gesture apps emphasize human interaction and speed. Non-gesture applications emphasize measurement accuracy.

Gesture applications convert human movements (face, hands, fingers, or whole body) into symbolic instructions to command game consoles, smart TVs, or portable computing devices.

For example, channel surfing can be done with a wave of your hand, and you can scroll through a presentation with a flick of a finger. These applications typically require fast response times, low to mid-range, centimeter-level accuracy and power consumption. Gesture interaction using time-flight sensor in Microsoft Kinect2.

There are many drones on the market that have gesture modes, which are very useful for taking drone selfies. DJI MavicPro and Phantom4Pro use gesture mode.

DJI Mavic Air uses advanced gestures and face recognition to fly and shoot. DJI didn’t give too many details about the technology behind their gesture mode. Their many autonomous flight modes and collision avoidance systems combine visual and ultrasonic sensors with sophisticated algorithms.

In general, we will see more drones entering the market, gesture modes and new autonomous flight modes, and ToF will play its part.

⑦、UAV ToF sensor is used to measure distance

Time-of-flight sensors can quickly measure the distance between a ToF sensor and an object based on the time difference between the signal’s emission and its return to the sensor after being reflected by the object.

Applications include ground proximity detection for drones, ceiling proximity detection for indoor drones, and more.

⑧、UAV ToF sensor for warehouse tracking

ToF cameras can be used for object scanning and volume counting, assisting us with drones and solutions for warehousing.

Such as the use of ToF flash lidar cameras on Hardis Group Eyesee drones.

Eyesee drones are equipped with onboard cameras and indoor geolocation technology, allowing them to move using pre-set flight plans and capture data about pallets stored in warehouses.

The drone then correlates the captured image with its location in the warehouse and automatically translates its 3D location into a logistics address (storage location).

⑨、UAV camera using ToF3D depth camera

3D became very popular in TV and movies. Professional photographers, filmmakers and consumers are looking for cameras that can generate 3D content, whether video or still photos. This 3D technology is based on ToF sensors.

Unlike traditional cameras, ToF cameras provide not only an image of light intensity, but also a range map containing distance measurements for each pixel, obtained by measuring the time it takes for light to reach an object and return to the camera (time of flight). principle).

Cameras in drones have come a long way over the past few years. Last year, a few drones with 4k cameras appeared on the market. There are also 2 drones with zoom cameras. They are the WalkeraVoyager and the ZenmuseZ3 camera for the DJI Inspire1 drone.

So someday in the not too distant future, we can expect to see 3D photography cameras on most drones.

⑩. Innovative applications of ToF sensors in drones and automobiles

The automotive industry is developing a variety of systems, including some excellent safety systems using flash lidar time-of-flight sensors.

This is also why I believe the drone industry will follow and start investing more time and resources in time-of-flight technology.

Below are some of the innovations in the use of ToF technology in the automotive industry.

Driver status monitoring, such as head position detection and drowsiness recognition.

Passenger classification is used to autonomously set predefined passenger preferences, optimized head-up Display visualization and optimally adjusted airbag deployment force.

Touchless gesture control for infotainment, navigation and HVAC systems, etc.

Surround view for sophisticated parking assist and obstacle detection

For example, the MLX75123Tof camera sensor is a fully integrated companion chip from Melexis. It is fully suitable for automotive and non-automotive applications including but not limited to gesture recognition, driver monitoring, skeleton tracking, person or obstacle detection and traffic monitoring.

The Links:   AA104VC06 PD10M44OH7B06