Overview

Developing a reliable perception system can be very challenging. A visual perception system must be reliable under a variety of conditions, especially when it is used in a fully automated system that controls a vehicle. This example uses a lane detection algorithm to illustrate the process of using the 3D simulation environment to strengthen the design of the algorithm. The main focus of the example is the effective use of the 3D simulation tools rather than the algorithm itself. Therefore, this example reuses the perception algorithms from theVisual Perception Using Monocular Cameraexample.

The Visual Perception Using Monocular Camera example uses recorded video data to develop a visual perception system that contains lane marker detection and classification, vehicle detection, and distance estimation. Use of the recorded video is a great start, but it is inadequate for exploring many other cases that can be more easily synthesized in a virtual environment. More complex scenarios can include complex lane change maneuvers, occlusion of lane markers due to other vehicles, and so on. Most importantly, closed-loop simulation involves both perception and control of the vehicle, both of which require either a virtual environment or a real vehicle. Additionally, testing up front with a real vehicle can be expensive, thus making the use of a 3D simulation environment very attractive.

This example takes the following steps to familiarize you with an approach to designing a visual perception algorithm:

Introduction to the 3D Simulation Environment

Automated Driving Toolbox™ integrates a 3D simulation environment in Simulink. The 3D simulation environment uses the Unreal Engine by Epic Games. Simulink blocks related to the 3D simulation environment provide the ability to:

The Simulink blocks for 3D simulation can be accessed by openingdrivingsim3dlibrary.

To aid in the design of visual perception algorithms in this example, you use a block that defines a scene, a block that controls a virtual vehicle, and a block that defines a virtual camera. The example focuses on detecting lane markers using a monocular camera system.

Create a Simple Straight Road Scene in 3D Simulation

Start by defining a simple scenario involving a straight highway road on which to exercise the lane marker detection algorithm.

open_system('straightRoadSim3D');

TheSimulation 3D Scene Configurationblock lets you choose one of the predefined scenes, in this caseStraight Road. When the model is invoked, it launches the Unreal Engine®. TheSimulation 3D Vehicle with Ground Followingblock creates a virtual vehicle within the gaming engine and lets Simulink take control of its position by supplyingXandYin meters, andYawin degrees.X,Y, andYaware specified with respect to a world coordinate system, with an origin in the middle of the scene. In this case, since the road is straight, an offset of 0.75 meters in theY方向和一系列的增加Xvalues move the vehicle forward. Later sections of this example show how to define more complex maneuvers without resorting toX,Y, andYawsettings based on trial and error.

The model also contains aSimulation 3D Camerablock, which extracts video frames from a virtual camera attached at the rearview mirror within the virtual vehicle. The camera parameters let you simulate typical parameters of a camera that can be described by a pinhole camera model, including focal length, camera optical center, radial distortion, and output image size. When the model is invoked, the resulting scene is shown from a perspective of a camera that automatically follows the vehicle.

sim卡('straightRoadSim3D');

Design and Debugging of Visual Perception Module

视觉感知通常是复杂的,无论是involves classic computer vision or deep learning. Developing such a system often requires rapid iterations with incremental refinements. Although Simulink is a powerful environment for system-level engineering and closed-loop simulations, perception-based algorithms are typically developed in textual programming languages like MATLAB or C++. Additionally, the startup time for a model that needs to establish communication between Simulink and the Unreal Engine® is significant. For these reasons, it is convenient to record the image data generated by the virtual camera into a video and develop the perception algorithm in MATLAB. The following model records the camera into an MP4 file on disk.

open_system('straightRoadVideoRecording');

The video is recorded using theTo Multimedia Fileblock. The resultingstraightRoad.mp4file can now be used to develop the perception module, without incurring the startup-time penalty of the 3D simulation environment.

To design the lane marker detector, you use a module from theVisual Perception Using Monocular Cameraexample. However, if you simply transplant the existinghelperMonoSensor.mroutine from that example, even the simplest straight road scene does not produce good results. Immediately, you can see how powerful the virtual environment can be. You can choose any trajectory or environment for your vehicle, thus letting you explore many what-if scenarios prior to placing the perception module on an actual vehicle.

To aid in the design of the algorithm, use the providedHelperLaneDetectorWrapper.msystem object. This system object works in MATLAB and when placed inside theMATLAB System Block(Simulink)in Simulink. The following script,helperStraightRoadMLTest, invokes the wrapper from the MATLAB command prompt. This approach permits quick iterations of the design without continuous invocation of the 3D simulation environment.

helperStraightRoadMLTest

算法开始工作后,你可以place it back into a model as shown below. You can attempt to change the car's trajectory, as demonstrated in theSelect Waypoints for Unreal Engine Simulationexample. That way, you can look for ways to move the car such that the algorithm fails. The entire process is meant to be iterative.

open_system('straightRoadMonoCamera');

Navigate Through a More Complex Scene to Improve the Perception Algorithm

While developing your algorithm, you can increase the level of scene complexity to continue adapting your system to conditions resembling reality. In this section, switch the scene toVirtual Mcity, which provides stretches of the road with curved lanes, no lane markers, or merging lane markers.

Before you begin, you need to define a trajectory through a suitable stretch of the virtual Mcity, which is a representation of actual testing grounds that belong to the University of Michigan. To see the details of how to obtain a series ofX,Y, andYawvalues suitable for moving a car through a complex environment, refer to theSelect Waypoints for Unreal Engine Simulationexample. The key steps are summarized below for your convenience.

% Extract scene image location based on scene's namesceneName ='VirtualMCity'; [sceneImage, sceneRef] = helperGetSceneImage(sceneName);

% Interactively select waypoints through McityhelperSelectSceneWaypoints(sceneImage, sceneRef)

% Convert the sparse waypoints into a denser trajectory that a car can% follownumPoses = size(refPoses, 1); refDirections = ones(numPoses,1);% Forward-only motionnumSmoothPoses = 20 * numPoses;% Increase this to increase the number of returned poses[newRefPoses,~,cumLengths] = smoothPathSpline(refPoses, refDirections, numSmoothPoses);

% Create a constant velocity profile by generating a time vector% proportional to the cumulative path lengthsimStopTime = 10; timeVector = normalize(cumLengths,'range', [0, simStopTime]);

refPosesX = [timeVector, newRefPoses(:,1)]; refPosesY = [timeVector, newRefPoses(:,2)]; refPosesYaw = [timeVector, newRefPoses(:,3)];

Load the preconfigured vehicle poses created using the method shown above.

poses = load('mcityPoses');

With the predefined trajectory, you can now virtually drive the vehicle through a longer stretch of a complex virtual environment.

open_system('mcityMonoCamera'); sim('mcityMonoCamera'); clearposes;

Many times, the results are less than desirable. For example, notice where the barriers are confused with lane markers and when the region of interest selected for analysis is too narrow to pick up the left lane.

However, the detector performs well in other areas of the scene.

The main point is that the virtual environment lets you stress-test your design and helps you realize what kind of conditions you may encounter on real roads. Running your algorithm in a virtual environment also saves you time. If your design does not run successfully in the virtual environment, then there is no point of running it in a real vehicle on a road, which is far more time-consuming and expensive.

Closed-Loop Testing

One of the most powerful features of a 3D simulation environment is that it can facilitate closed-loop testing of a complex system. Lane keep assist, for example, involves both perception and control of the vehicle. Once a perception system is perfected on very complex scenes and performs well, it can then be used to drive a control system that actually steers the car. In this case, rather than manually set up a trajectory, the vehicle uses the perception system to drive itself. It is beyond the scope of this example to show the entire process. However, the steps described here should provide you with ideas on how to design and debug your perception system so it can later be used in a more complex closed-loop simulation.

bdcloseall;