CN110198439A - Method and apparatus for testing the image recognition performance of ADAS camera automatically - Google Patents

Abstract

Embodiment of the invention discloses the method and apparatus of the image recognition performance for testing ADAS camera automatically.In method, simulated environment is generated, simulated environment includes multiple target objects, and each target object in multiple target objects has target object information.Simulated environment is presented.Receive the identification object information of the multiple identification objects identified by ADAS camera from simulated environment.The identification object information of multiple identification objects is compared with the target object information of multiple target objects respectively.

Description

Method and device for automatically testing image recognition performance of ADAS cameras

technical field

The invention relates to the field of vehicle technology, in particular to a method and device for automatically testing the image recognition performance of an ADAS camera.

Background technique

The widespread popularity of cars in today's society has made people's lives more convenient. However, with the rapid increase in the number of cars and the frequent occurrence of various road safety accidents, people have begun to pay more and more attention to the safety of car driving. At present, Advanced Driver Assistance System (ADAS, Advanced Driver Assistance System) has been installed in some vehicles, which can also be called active safety system. Identify, detect and track static and dynamic objects. In addition, ADAS also combines the map data of the navigator to carry out systematic calculation and analysis, so as to let the driver perceive possible dangers in advance, and effectively increase the comfort and safety of the car.

The functions of ADAS include, for example, lane departure warning based on the forward-looking camera, forward collision avoidance warning, pedestrian and obstacle detection, traffic sign recognition, etc. In order to ensure the excellent performance of the ADAS system, a large number of tests and evaluations are required in the product development process, and relevant international test standards have been produced, such as ISO26262 and SAE related standards.

Contents of the invention

Embodiments of the present invention provide a method and device for automatically testing the image recognition performance of an ADAS camera. The method can evaluate the image recognition effect of the ADAS camera by generating a simulation environment for simulating the driving of the vehicle equipped with the ADAS camera and an automated test process.

According to one aspect of the present invention, a method for automatically testing the image recognition performance of an ADAS camera is provided. In the method, a simulation environment is generated, the simulation environment includes a plurality of target objects, each target object of the plurality of target objects has target object information. Present the simulation environment. Recognized object information of a plurality of recognized objects recognized by the ADAS camera from the simulated environment is received. And comparing the identification object information of the plurality of identification objects with the target object information of the plurality of target objects respectively.

According to another aspect of the present invention, a device for automatically testing the image recognition performance of an ADAS camera is provided. The apparatus includes a controller including a processor and memory. The processor can generate a simulation environment, the simulation environment includes a plurality of target objects, and the memory can store target object information of each of the plurality of target objects. The display may present a simulated environment. The ADAS camera can recognize a plurality of recognition objects from the simulation environment, and send recognition object information of the plurality of recognition objects to a processor of the controller. The processor of the controller may also receive recognition object information of a plurality of recognition objects recognized by the ADAS camera, and compare the recognition object information of the plurality of recognition objects with target object information of a plurality of target objects, respectively.

According to the embodiments of the present invention, the extensiveness, diversity and balance of test data can be ensured, the test results can be made more objective, fair and comprehensive, and labor costs can be saved at the same time.

Description of drawings

In order to illustrate the technical solutions of the present invention more clearly, the accompanying drawings of the embodiments will be briefly described below. It should be known that the accompanying drawings described below are only some embodiments of the present invention, rather than limiting the present invention, wherein:

Fig. 1 is the flow chart of the method for automatically testing the image recognition performance of ADAS camera according to an embodiment of the present invention;

Fig. 2 is an exemplary schematic diagram of a simulation environment according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an apparatus for automatically testing the image recognition performance of an ADAS camera according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the described embodiments, all other embodiments obtained by those skilled in the art without creative efforts also fall within the scope of the present invention.

Since many functions of ADAS are realized based on the image recognition of its camera, it is necessary to test the image recognition performance of the ADAS camera (hereinafter referred to as ADAS camera). In order to ensure the fairness of the test results, the test samples should conform to the principle of normal distribution as much as possible, that is, they need to cover a variety of weather and lighting conditions and different image background environments, and meet the requirements of breadth, diversity, and balance. If 90% of the data in the test data set is collected under normal weather conditions, and only a small amount of data is collected under severe weather conditions, then such test results will not conform to the real objective actual performance even if the correct rate reaches 99.999%. At present, the test is mainly implemented based on two methods: real vehicle test and sample test set test.

In the real vehicle test, an ADAS camera is installed on the test vehicle, and a special person drives the vehicle on the road for a road test. After collecting the video containing the ADAS image recognition processing results (including the object mark frame), manually watch the video to identify whether the object in the recognized image is consistent with the object in the actual picture. Further, the correct recognition rate and false detection rate of the image recognition are manually counted. However, since this method requires manual driving to collect data and manual identification of whether the image is recognized successfully, it is very time-consuming and labor-intensive. In addition, because there are many factors affecting the actual road and the situation is extremely complicated, it is impossible for the real vehicle test to cover all scenarios. If the weather in a certain city where the vehicle is collected has always been good, the collected test samples cannot meet the requirements for the breadth and diversity of the samples. However, if you want to meet the requirements of breadth, diversity, and balance, you need to collect data in areas with rainy and snowy weather or certain types of weather conditions. The cost of collecting data in this way is too high. In addition, the actual road test is carried out before the ADAS algorithm is mature, and the algorithm cannot be improved quickly, which greatly increases the development cycle.

On the other hand, in the sample test set test, it is necessary to manually collect and prepare some original image data files containing detected objects as the sample test set. After that, run the image recognition algorithm on the computer to detect the entire data set, and calculate some global KPI indicators, such as correct recognition rate, false detection rate, etc. However, this method also has the above problems. The sample test set is obtained through data collection, and it is difficult to guarantee the requirements of extensiveness, diversity, and balance.

In addition, in the above test process, if an object is found not to be detected, it is necessary to manually add the picture of the object to the training data set for retraining and repairing the problem, which consumes a lot of time and manpower.

FIG. 1 shows a flowchart of a method for automatically testing the image recognition performance of an ADAS camera according to an embodiment of the present invention.

In step S110 , a simulation environment for simulating driving of a vehicle equipped with an ADAS camera on a road is generated. For example, simulation environments with different road conditions (such as cities, mountain roads, etc.) can be generated by virtual simulation software, and the weather parameters (cloudy, snowstorm, rainstorm, etc.) , too dark, moderate, etc.) to enhance the diversity and generalization of test samples. In addition, the virtual simulation software can also control the driving of the vehicle, such as controlling the accelerator, steering wheel, brakes, etc., to simulate the automatic driving of the vehicle in the simulation environment.

The simulation environment can include multiple target objects, such as vehicles, pedestrians, and traffic signs. Specifically, traffic vehicles include vehicles, cars, trucks, bicycles, trailers, tricycles, motorcycles, mopeds, and even baby carriages, etc., and traffic signs include traffic lights, road signs, lane indicator lines, etc. In addition, each target object has target object information, such as the type, coordinate position, size, attribute information, etc. of the target object in the image of the simulation environment. Attribute information includes, for example, words on road signs, license plate numbers of vehicles, nameplates of buildings, etc.

Fig. 2 shows an exemplary schematic diagram of a simulation environment. As shown in Figure 2, the target objects in the simulation environment include several vehicles, road signs, buildings and so on.

In step S120, a display is used to present the generated simulation environment.

In step S130, the recognition object information of the plurality of recognition objects recognized by the ADAS camera from the simulation environment is received. The recognition object may be a recognized target object, such as a vehicle, a person, a traffic sign, and the like. Specifically, the ADAS camera and the display are fixed through a mechanical bracket, and the pixel coordinate system of the ADAS recognition image is matched with the coordinate system of the software that provides the simulation environment. The ADAS camera recognizes the image displayed on the monitor and processes the recognition object information of the recognized recognition object at the same frame rate as the displayed image.

As shown in Figure 2, the ADAS camera can identify a car in the middle and lower part of the image, road signs in the middle and right of the upper image, several other vehicles in the middle of the image, and the lane in the lower part of the image Marking lines, buildings on the right side of the image, etc. In this way, the recognition object information of the above recognition objects is received, such as the object type, position coordinates, size, attribute information, etc. of each recognition object in the image.

In step S140, the identification object information of the plurality of identification objects is compared with the target object information of the plurality of target objects respectively.

In the embodiment of the present invention, if the identification information of each recognition object recognized by the ADAS camera is consistent with the recorded target object information of each target object, the image recognition of the ADAS camera is successful. If the target object information of a target object is inconsistent with the corresponding recognized object information, the ADAS camera will identify the target object incorrectly, store it in the data source of the negative sample, and will identify the target object of the wrong target object The information is stored for retraining. In addition, an image of the simulation environment at the time of recognition error may be stored. On the other hand, when at least one target object is not recognized, the unrecognized target object and its target object information are stored, so as to be automatically used as a data source for retraining.

Furthermore, it is possible to count the identified identification objects. When the number of recognized objects reaches a threshold number, the testing process ends. According to the number of successfully recognized recognition objects and the total number of target objects, the success rate of image recognition of the ADAS camera is calculated, so as to test the image recognition performance of the ADAS camera.

In the embodiment of the present invention, after a test process is finished, related test results can be saved and the simulation environment can be regenerated. Specifically, the weather and light conditions can be changed, the planned driving route can be changed, the target object can be replaced, etc., so as to perform a new round of testing again. Therefore, after multiple rounds of testing, the requirements for the breadth, diversity, and balance of test data are ensured.

The method according to the embodiment of the present invention can customize conditions such as test site and weather by adopting a high-quality virtual simulation environment and an automated standardized test process. As a result, the image recognition results can be fairly and comprehensively evaluated, thereby reducing labor costs, ensuring the extensiveness, diversity, and balance requirements of the test data, and making the test results more objective, fair and comprehensive. When the algorithm software is found to have missed or false detections during the test, relevant information can also be automatically collected as a data source for retraining.

FIG. 3 is a schematic diagram of an apparatus 300 for automatically testing the image recognition performance of an ADAS camera according to an embodiment of the present invention. Apparatus 300 includes a controller 310, a display 320, an ADAS camera 330, and several accessories and power supplies (not shown).

The controller 310 is, for example, a PC host computer, and the configuration of the host computer needs to support high-quality simulation software to run smoothly. The controller 310 may include a processor 312 and a memory 314 . The processor 312 can generate a simulation environment through virtual simulation software, thereby simulating the external environment when the vehicle equipped with the ADAS camera is driving on the road, and the simulation environment can include multiple target objects. The parameters of the simulation environment include target object information of surrounding target objects, vehicle operating parameters, weather information, illumination information, and the like. The target object information includes the type, position coordinates, size, etc. of the target object. In addition, the processor 312 can simulate and control the automatic driving of the vehicle, such as accelerator, steering wheel, and brake lights, for example, it can automatically avoid vehicle collisions in the simulated environment, so that the screen information close to the real environment can be obtained without real vehicle driving. The memory 314 may store target object information of each of the plurality of target objects, and other parameters in the simulation environment, and the like.

The display 320 is connected in information communication with the controller 310 . The display 320 receives the simulated environment generated by the controller 310 and presents it in video form. The display 320 is, for example, a high-definition quality large-screen display.

The ADAS camera 320 is fixed by a mechanical bracket, so that the position between it and the display 320 remains unchanged. The ADAS camera 330 may recognize a plurality of recognition objects from the simulated environment presented by the display 320 . The ADAS camera 320 is pre-calibrated so that the pixel coordinate system of the recognized image matches the coordinate system of the image provided by the controller 310 .

On the other hand, the ADAS camera 320 is also communicatively connected with the controller 310 , for example, through a CANOE (CAN Open Environment) device of Vector, Germany, so as to establish a CAN Bus (CAN bus) data communication. Thus, the ADAS camera 320 may send identification object information of a plurality of identification objects to the processor 312 of the controller 310 .

The processor 312 of the controller 310 may receive recognition object information of a plurality of recognition objects recognized by the ADAS camera 330 and compare the recognition object information of the plurality of recognition objects with target object information of a plurality of target objects, respectively.

In the embodiment of the present invention, when the identification information of each identification object recognized by the ADAS camera is respectively consistent with the target object information of each target object in the simulation environment generated by the virtual simulation software, the processor 312 can determine The image recognition of the ADAS camera was successful. When the target object information of a certain target object is inconsistent with the recognized recognition object information, the processor 312 may determine that the ADAS camera has misrecognized the target object, and stores the target object information of the wrongly recognized target object in the memory 314 , and store it in the data source of negative samples. In addition, when at least one target object is not recognized, the processor 312 may also store the target object information of the unrecognized target object, so as to automatically serve as a data source for retraining.

On the other hand, the processor 312 may count the identified identification objects. When the number of recognized objects reaches a threshold number, the testing process ends. The processor 312 calculates the success rate of the image recognition of the ADAS camera according to the number of successfully recognized recognition objects and the total number of target objects, so as to test the image recognition performance of the ADAS camera.

In an embodiment of the present invention, after a test process is finished, the memory 314 can save the relevant test results, and the processor 312 can regenerate the simulation environment. Specifically, the weather and light conditions can be changed, the planned driving route can be changed, and the target object model can be replaced. Then, perform a new round of testing again. Therefore, after multiple rounds of testing, the requirements for the breadth, diversity, and balance of test data are ensured.

The device according to the embodiment of the present invention can customize conditions such as test site and weather by adopting a high-quality virtual simulation environment and an automated standardized test process. As a result, the image recognition results can be fairly and comprehensively evaluated, thereby reducing labor costs, ensuring the extensiveness, diversity, and balance requirements of the test data, and making the test results more objective, fair and comprehensive. When the algorithm software is found to have missed or false detections during the test, relevant information can also be automatically collected as a data source for retraining.

Since the target object parameters of the target object are known in the simulation environment, the processor clearly knows the truth-value attribute of the recognized object, so automatic judgment can be realized. The simulation environment can support a variety of scene switching, and can arbitrarily change the weather (rainfall, snowfall, lightning, etc.), light (dusk, night, etc.), road conditions and other environmental variables, and can also change different target objects (traffic vehicles, pedestrians, traffic logo, etc.). The test data is more extensive, diverse, and balanced, and the final test results are closer to real performance. Therefore, simulating a scene is faster and cheaper than recording a video of a real car driving on a real road.

Several embodiments of the present invention have been described in detail above, but the protection scope of the present invention is not limited thereto. Apparently, those skilled in the art can make various modifications, substitutions or variations to the embodiments of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention is defined by the appended claims.

Claims (14)

1. A method for automatically testing the image recognition performance of an ADAS camera, comprising: generating a simulation environment, the simulation environment including a plurality of target objects, each target object in the plurality of target objects having target object information; presenting the simulated environment; receiving identification object information of a plurality of identification objects identified by the ADAS camera from the simulated environment; and The identification object information of the plurality of identification objects is compared with the target object information of the plurality of target objects respectively. 2. The method of claim 1, wherein, The target object information includes an object type and position coordinates of the target object; The identification object information includes the object type and position coordinates of the identification object. 3. The method of claim 1, further comprising: When at least one of the plurality of target objects is misrecognized, target object information of the misidentified target object is stored. 4. The method of claim 1, further comprising: When at least one of the plurality of target objects is not identified, storing target object information of the unidentified target object. 5. The method of claim 1, wherein, The simulation environment is generated based on weather information, illumination information, road information and target objects. 6. The method of claim 1, wherein, The target object includes at least one of a vehicle, a person, and a traffic sign. 7. The method of claim 1, further comprising counting the plurality of identified objects; When the number of the plurality of identified objects reaches a threshold number, the testing process ends. 8. A device for automatically testing the image recognition performance of an ADAS camera, comprising: a controller comprising a processor and a memory, wherein the processor is configured to generate a simulation environment including a plurality of target objects, and the memory is configured to store each of the plurality of target objects target audience information; a display configured to present the simulated environment; an ADAS camera configured to recognize a plurality of recognition objects from the simulation environment, and send recognition object information of the plurality of recognition objects to a processor of the controller; The processor of the controller is further configured to receive identification object information of a plurality of identification objects identified by the ADAS camera, and compare the identification object information of the plurality of identification objects with the target objects of the plurality of target objects, respectively. Object information is compared. 9. The apparatus of claim 8, wherein, The target object information includes an object type and position coordinates of the target object; The identification object information includes the object type and position coordinates of the identification object. 10. The apparatus according to claim 8, the processor of the controller is further configured to: when at least one of the plurality of target objects is misidentified, store the target object information of the misidentified target object . 11. The apparatus of claim 8, the processor of the controller further configured to: When at least one of the plurality of target objects is not identified, storing target object information of the unidentified target object. 12. The apparatus of claim 8, wherein, The simulation environment is generated based on weather information, illumination information, road information and target objects. 13. The apparatus of claim 8, wherein, The target object includes at least one of a vehicle, a person, and a traffic sign. 14. The apparatus of claim 8, the processor of the controller further configured to: counting the plurality of identified objects; When the number of the plurality of identified objects reaches a threshold number, the testing process ends.