KR102864685B1 - Method of slam based on salient object in image and robot and cloud server implementing thereof - Google Patents

Abstract

The present invention relates to a method for performing SLAM based on a characteristic object in an image, and to a robot and a cloud server implementing the same. According to one embodiment of the present invention, a robot for performing SLAM based on a characteristic object in an image includes a camera sensor for acquiring an image required for performing SLAM, a map storage unit for storing information required for the robot to perform SLAM, and a control unit for verifying and extracting a characteristic object from an image and storing location information at which the image was acquired, the characteristic object, and coordinate information of the characteristic object in the image in the map storage unit.

Description

Method of performing SLAM based on salient objects in an image and a robot and cloud server implementing the same {Method of SLAM based on salient objects in an image and a robot and cloud server implementing the same}

The present invention relates to a method for performing slam based on a characteristic object in an image and a robot and cloud server implementing the same.

In spaces with high levels of human and material interaction, such as large supermarkets, department stores, airports, and golf courses, robots can be deployed and moved indoors or in outdoor spaces connected to indoor spaces to provide information or convenience to people. Furthermore, robots can move outdoors in the form of vehicles or by driving on sidewalks. These robots can perform functions such as delivery, security, cleaning, and guidance.

Meanwhile, in order for these robots to drive autonomously without human control or to drive semi-autonomously based on temporary human control, they need to store or receive from the outside the information necessary to confirm the robot's current location, avoid obstacles placed around it, and move to the destination.

To this end, the robot can store map information, which is information about space, and receive map information in real time from a server or other nearby robots.

However, in order for the robot to accurately determine its current location during this process, it needs to compare the information obtained from its current location with map information. To achieve this, the map information must contain precise spatial information. Furthermore, it must contain information that is easy to search.

Accordingly, this specification proposes a method for generating a map suitable for a robot to perform simultaneous localization and mapping (SLAM).

In order to solve the aforementioned problem, this specification allows a robot or a cloud server to store image information and characteristic objects necessary for a robot to generate a map and perform location estimation.

Additionally, in this specification, we aim to extract characteristic objects suitable for increasing the accuracy and speed of location estimation within an image.

Additionally, the present specification stores the location of a characteristic object within an image so that the robot can estimate the location more accurately.

The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

A robot performing SLAM based on a characteristic object in an image according to one embodiment of the invention includes a camera sensor for acquiring an image required for performing SLAM (simultaneous localization and mapping), a map storage unit for storing information required for the robot to perform SLAM, and a control unit for verifying and extracting a characteristic object from the image and storing location information where the image was acquired, the characteristic object, and coordinate information of the characteristic object in the image in the map storage unit.

A robot performing SLAM based on a characteristic object in an image according to one embodiment of the invention detects one or more objects in the image, verifies whether the detected object corresponds to a characteristic object, and stores the object as a characteristic object in a map storage unit based on the verification result.

A robot performing SLAM based on a characteristic object in an image according to one embodiment of the invention verifies an object commonly included in a first image acquired at a first time point and a second image acquired at a second time point subsequent to the first time point as a characteristic object.

A robot performing SLAM based on a characteristic object in an image according to one embodiment of the invention outputs an image, a detected object, and a bounding box placed around the object from an interface unit and receives an input for selecting the same.

A cloud server according to one embodiment of the invention receives an image required for performing simultaneous localization and mapping (SLAM) from a robot, verifies and extracts a characteristic object from the image, and stores the location information where the image was acquired, the characteristic object, and the coordinate information of the characteristic object within the image in a map storage unit.

A cloud server according to one embodiment of the invention detects one or more objects in an image, verifies whether the detected objects correspond to characteristic objects, and stores the objects as characteristic objects in a map storage unit based on the verification result.

A method for performing SLAM based on a characteristic object in an image according to one embodiment of the invention includes a step of allowing a camera sensor of a robot to acquire an image necessary for performing SLAM (simultaneous localization and mapping) while a moving part of the robot moves the robot, a step of allowing a control part of the robot to verify and extract a characteristic object from the image, and a step of allowing the control part to store location information where the image was acquired, the characteristic object, and coordinate information of the characteristic object in the image in a map storage unit.

When applying embodiments of the present invention, the robot or cloud server can store image information and characteristic objects necessary for the robot to generate a map and perform location estimation.

In addition, when applying embodiments of the present invention, a robot or cloud server can extract a characteristic object suitable for increasing the accuracy and speed of location estimation from an image, and can also verify whether it is a characteristic object.

In addition, when applying embodiments of the present invention, the location of a characteristic object within an image is stored together so that the robot can estimate the location more accurately.

The effects of the present invention are not limited to the effects described above, and those skilled in the art can easily derive various effects of the present invention from the composition of the present invention.

Figure 1 shows the configuration of a robot that generates a map.
Figure 2 shows a detailed configuration of a control module according to one embodiment of the present invention.
Figure 3 shows the configuration of a cloud server according to one embodiment of the present invention.
Figure 4 shows a process for selecting a characteristic object according to one embodiment of the present invention.
Figure 5 shows a characteristic object verification process according to one embodiment of the present invention.
Figure 6 shows the arrangement of bounding boxes around a characteristic object according to one embodiment of the present invention.
Figure 7 shows a process in which a robot determines a characteristic object according to one embodiment of the present invention.
FIG. 8 shows an example of comparing an image acquired by a robot during a movement process according to one embodiment of the present invention with an image stored in a map storage unit.
Figure 9 shows a localization process for generating pose information from a map storage unit using a characteristic object according to one embodiment of the present invention.
Figure 10 shows a process in which a robot detects movement of an object in continuously captured images according to one embodiment of the present invention and determines a characteristic object.
Figure 11 shows a process in which an interface unit of a robot according to one embodiment of the present invention outputs a characteristic object and receives an input for selecting the same.
Figure 12 shows the result of modifying a bounding box according to one embodiment of the present invention.
Figure 13 shows another configuration of a cloud server according to one embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present invention. The present invention may be implemented in various different forms and is not limited to the embodiments described herein.

In order to clearly explain the present invention, parts that are not related to the description have been omitted, and the same or similar components are designated by the same reference numerals throughout the specification. In addition, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, the same components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when explaining the present invention, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present invention, the detailed description may be omitted.

When describing components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms. When it is described that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that other components may also be "interposed" between each component, or that each component may be "connected," "coupled," or "connected" through another component.

In addition, in implementing the present invention, components may be described in detail for convenience of explanation, but these components may be implemented in one device or module, or one component may be implemented by being divided into multiple devices or modules.

Hereinafter, the term "robot" in this specification includes devices that have various shapes, have specific purposes (delivery, cleaning, security, monitoring, guidance, etc.), or provide functions based on the characteristics of the space in which the robot moves. Therefore, the term "robot" in this specification refers to a device that possesses a means of transportation capable of moving using predetermined information and sensors and provides predetermined functions.

In this specification, a robot can move while holding a map. The map represents images of streets that are confirmed to not move in space, as well as information about fixed objects such as fixed walls, buildings, and stairs. Additionally, information about periodically placed moving obstacles, i.e., dynamic objects, can also be stored on the map.

In this specification, a robot may be used to generate a map. However, the robot that generates the map and the robot that moves using the map do not necessarily have to have the same configuration. The robot that generates the map includes components from a first group suitable for map generation, and the robot that moves using the map includes components from a second group suitable for this purpose. Furthermore, the components from the first group and the components from the second group do not necessarily have to be identical.

In this specification, the robot communicates with a cloud server to send and receive information, and may also send and receive map information during this process. Alternatively, the cloud server may transmit map information within a certain range, including the area where the robot is located, to the robot, and the robot may perform SLAM based on this information.

To perform SLAM, robots can use lidar sensors to determine the location of objects or camera sensors to capture images. In particular, when navigating in wide spaces, such as outdoor spaces, robots can store image information about the space on a map. The image information stored on the map is not simply an image, but includes information about specific features within the image.

Therefore, to increase the speed or accuracy of location estimation, the robot can automatically select characteristic features during the map generation process and automatically register them on the map. Alternatively, the robot can provide the user with candidate information on characteristic features, allowing the user to select from among these candidates and then register the selected characteristic features on the map.

In an embodiment of the present invention, a robot is equipped with a camera sensor that acquires images necessary for performing simultaneous localization and mapping (SLAM). Furthermore, we will examine the process of storing information about salient objects within the image while creating a map using the camera sensor. A salient object is information necessary for identifying a space, such as a sign or a marker. A salient object can be a feature within the image.

Additionally, the robot can extract characteristic objects suitable for SLAM from the accumulated images. Furthermore, the robot can use artificial intelligence to extract characteristic objects within the images that can easily distinguish specific images from the accumulated images.

Figure 1 illustrates the configuration of a map-generating robot. The robot moves through space, captures images using a camera sensor, and extracts characteristic objects from the captured images, storing them on a map. The configuration shown in Figure 1 is a map-generating robot.

The configuration of Fig. 1 also applies to map-generating robots, but they may be referred to by a different name: a map-generating device, or simply a mapper device. Map-generating robots are not limited to robots with a specific appearance, but include all types of devices that move through space, capture images of their surroundings, store them, and extract characteristic objects from them.

Hereinafter, the characteristic object described herein will be centered on an object identified within an image. However, the present invention is not limited thereto. Within the lidar sensing data generated by a 3D or 2D lidar sensor, a robot can designate a specific area as a characteristic object.

Meanwhile, the robot (100) may further include a control module (150) within it. The control module (150) controls the robot (100) like a type of computer or processor. Accordingly, the control module (150) is placed within the robot (100) and performs functions similar to the main processor, and may be responsible for interaction with the user. In addition, the control module (150) is responsible for communication with the cloud server.

A control module (150) is mounted inside the robot (100) to control the robot by detecting its movements and surrounding objects. The robot control module (150) can be implemented as a software module or a chip that implements the same as hardware.

As illustrated in Fig. 1, the robot (100) includes a control module (150), a storage unit (110), a functional unit (120), a battery (180), and a moving unit (190). The robot of Fig. 1 can be classified into industrial, medical, household, and military use types depending on the purpose or field of use.

The robot (100) optionally includes a storage compartment (110). The storage compartment (110) is a space where objects are stored or loaded by the user. In addition, the robot may follow the user and move while carrying the storage compartment (110).

The functional unit (120) is a component that performs a specific function assigned to the robot. In a robot with a cleaning function, the functional unit (120) includes a mop, a vacuum cleaner, etc. for cleaning. In a robot for delivery, the functional unit (120) includes a storage space, a transport unit for moving stored cargo, etc. In a robot for security, the functional unit (120) includes a safety inspection device (air quality inspection, explosive inspection, etc.).

The battery (180) provides the electrical energy required for the robot (100) to operate. The moving part (190) provides the robot's movement function.

The robot is equipped with a moving part (190) including an actuator or motor, and can perform various physical actions, such as moving the robot joints. In addition, the moving part of the mobile robot includes wheels, brakes, propellers, etc., and can drive on the ground or fly in the air through the moving part.

Additionally, robots can perform autonomous driving. Autonomous driving refers to the technology of driving on one's own, and autonomous robots drive without or with minimal user intervention.

For example, autonomous driving can include technologies that maintain a distance from other obstacles in the driving space, technologies that automatically adjust speed such as adaptive cruise control, technologies that automatically drive along a set route, and technologies that automatically set a route and drive when a destination is set.

The robot may be equipped with an internal combustion engine or an electric motor for autonomous driving, which become sub-components of the moving part (190).

Figure 2 shows a detailed configuration of a control module according to one embodiment of the present invention.

The components of the control module (150) are logically components of the control module regardless of their physical location or coupling method.

The control module (150) controls the robot so that the robot can perform both the function of generating a map and the function of estimating the location of the robot using the map.

Alternatively, the control module (150) controls the robot to provide only the function of generating a map.

Alternatively, the control module (150) controls the robot so that it can only provide the function of estimating the robot's location using a map. That is, the control module (150) can control the robot so that the robot generates a map, estimates the location using a map, or performs both of the aforementioned functions.

Robots can transmit information acquired by sensors during their movement to a cloud server. Let's examine the different types of sensors.

First, the LiDAR sensor (220) can sense surrounding objects in two or three dimensions. A two-dimensional LiDAR sensor can sense the location of objects within a 360-degree range or smaller around the robot. LiDAR information sensed at a specific location serves as an example of sensor data.

Alternatively, sensor data acquired by a lidar sensor (220) may be referred to as a lidar frame. That is, the lidar sensor (220) senses the distance between an object placed outside the robot and the robot to generate a lidar frame.

The camera sensor (230) is, for example, a standard camera. To address field-of-view limitations, two or more camera sensors (230) can be used. Images captured from specific locations constitute image information. In other words, the image information generated by the camera sensor (230) by capturing an object placed outside the robot constitutes an example of sensor data.

Alternatively, sensor data acquired by the camera sensor (230) may be referred to as a visual frame. That is, the camera sensor (230) captures the exterior of the robot to create a visual frame.

The robot (100) applying the present invention below performs SLAM (simultaneous localization and mapping) using one or both of a lidar sensor (220) and a camera sensor (230).

In the SLAM process, the robot (100) can create a map or estimate a location by using the lidar frame and the visual frame separately or by combining them.

The interface unit (290) receives information from the user. It receives various types of information, such as touch input and voice input, from the user and outputs the results. In addition, the interface unit (290) can output a map stored by the robot (100) or output the robot's movement process by overlapping it with the map.

Additionally, the interface unit (290) can provide certain information to the user.

The control unit (250) generates a map, as described below, and estimates the robot's location based on this map as the robot moves. Alternatively, it may communicate with a cloud server to transmit and receive information, and based on this, create a map or estimate the robot's location.

The communication unit (280) enables the robot (100) to communicate with other robots or an external cloud server to transmit and receive information.

The map storage unit (210) stores a map of the space in which the robot moves. Specifically, this map may be a map created by the robot itself. Alternatively, this map may be a map synchronized with a cloud server and stored by the robot. The map storage unit (210) may be optionally maintained by the robot. For example, the robot (100) may utilize a map stored on a cloud server without storing a map.

The wheel encoder (260) collects information such as rotation and direction of the wheels that constitute the robot's moving part, generates wheel odometry information, and provides it to the control unit (250). The control unit (250) can calculate the movement distance and movement direction based on the information provided by the wheel encoder (260).

The artificial intelligence department (255) will be described later.

Meanwhile, during the map creation process, a cloud server may perform some map creation functions. For example, the robot (100) may transmit data (visual frames, lidar frames) acquired by the robot's camera sensor (230) or lidar sensor (220) to the cloud server, and the cloud server may accumulate and store the data before generating a map.

That is, the map storage unit (210) stores information necessary for the robot to perform a slam. In addition, the robot's control unit (250) verifies and extracts characteristic objects from the images captured by the camera sensor (230) and stores the location information where the image was acquired, the characteristic objects identified within the images, and the coordinate information of the characteristic objects within the images in the map storage unit (210).

Figure 3 illustrates the configuration of a cloud server according to one embodiment of the present invention. The map storage unit (310) stores high-quality generated maps. The maps in the map storage unit (310) can be partially or fully transmitted to the robot (100). The communication unit (380) communicates with the robot (100). During this process, multiple robots (100) can communicate with the communication unit (380).

And the artificial intelligence unit (355) can extract features (Feature Extraction) or match the extracted features with the map, etc. during the process of creating or updating the map. The server control unit (350) controls the aforementioned components and creates various information necessary for the robot (100) to perform SLAM and provides it to the robot (100).

The cloud server (300) constitutes a cloud system. The cloud system comprises multiple robots and one or more cloud servers (300). Because the cloud server is capable of high-capacity/high-performance computational processing, it can quickly create high-quality maps using SLAM.

That is, the cloud server (300) can perform SLAM using as much information as possible by utilizing high computing power. As a result, the quality of the map created by the cloud server (300) can be greatly improved. When the communication unit (380) receives the sensor data transmitted by the robot (100), the server control unit (350) performs feature extraction, map creation, and map update based on the data, and when the robot is kidnapped, it can recover the robot's location (kidnap recovery) or provide information necessary for location recovery.

The cloud server (300) may also include a map storage unit (310). In addition, the cloud server (300) may include an artificial intelligence unit (355).

The communication unit (380) receives images required to perform simultaneous localization and mapping (SLAM) from the robot. The map storage unit (310) stores information required for the robot to perform SLAM.

The server control unit (350) verifies and extracts characteristic objects from the received image and stores the location information where the image was acquired, the characteristic objects, and the coordinate information of the characteristic objects within the image in the map storage unit (310).

The information stored by the cloud server (300) is then transmitted to the robots so that the robots can perform location estimation.

Below, we will look at the detailed process of creating a map using characteristic objects.

The control unit (250) of the robot (100) or the server control unit (350) of the cloud server (300) identifies a characteristic object or a candidate for a characteristic object with a high probability of becoming a characteristic object within the acquired sensor data. In this embodiment, the sensor data is a visual frame, which is an image, or a lidar frame, which is lidar sensor data. The image (visual frame) acquired by the camera sensor (230) will be examined in detail in FIG. 4.

Fig. 4 illustrates a process for selecting a characteristic object according to one embodiment of the present invention. The process of Fig. 4 is explained based on the content performed by the control unit (250) of the robot (100). For the convenience of explanation, the description focuses on the control unit (250) of the robot (100). However, the tasks performed or functions provided by the control unit (250) of the robot (100) in the process of Fig. 4 or the content described below can also be performed or provided in the same manner by the server control unit (350) of the cloud server (300).

The control unit (250) of the robot (100) detects an object from the acquired first sensor data (S11). In one embodiment, the camera sensor (230) may acquire an image while the moving unit (190) of the robot (100) moves the robot, and the first sensor data is image data in one embodiment.

For example, the control unit (250) detects objects that can be features within an image acquired by a camera sensor. To this end, the control unit (250) may reference individual images of pre-stored objects.

The control unit (250) can additionally verify whether the detected object is a characteristic object (S12). Then, the control unit (250) sets the detected object as a characteristic object and stores it in the map storage unit (210) (S13).

At this time, the control unit (250) may additionally store coordinate information for a bounding box that distinguishes the periphery of a candidate for a characteristic object from other parts within the image. In addition, the control unit (250) may determine whether to include the object as a characteristic object.

Figure 4 presents a process for extracting characteristic objects from images forming a map to increase search speed and accuracy during the map generation process. To this end, the control unit (250) detects one or more objects within the image (S11) and verifies whether the detected objects correspond to characteristic objects (S12). Then, the control unit (250) stores the objects as characteristic objects in the map storage unit (210) based on the verification results (S13).

By storing characteristic objects in the map storage unit (210), when performing subsequent location estimation, rather than entering the entire image captured during the robot's movement as a search query, a specific object within the image can be entered as a search query. This reduces the time required for image searches and improves search accuracy. As a result, the speed and accuracy of location estimation are increased.

Figure 5 illustrates a characteristic object verification process according to one embodiment of the present invention. The process of Figure 5 is explained based on the content performed by the control unit (250) of the robot (100). The server control unit (350) of the cloud server (300) can also perform the process of Figure 5 in the same manner.

The control unit (250) distinguishes between fixed and dynamic objects within sensor data, such as images. It then selects objects by distinguishing between near and far objects (S16). During the selection process, a score may be assigned to each object. Furthermore, the control unit (250) selects objects with a high probability of being characteristic objects as characteristic objects. Alternatively, to enhance accuracy, the control unit (250) may selectively perform steps S17 to S19.

S16 presents an embodiment in which the control unit (250) determines whether each object extracted from an image is a dynamic object or a fixed object. Furthermore, the control unit (250) determines a fixed object as a characteristic object only if the object is a dynamic object. This means that the image and characteristic object stored in the map are compared with the image captured during the next robot movement and the characteristic object extracted from the image.

Accordingly, people, cars, etc. that only temporarily stay within the space are dynamic objects, so the control unit (250) does not need to store them as characteristic objects.

Additionally, S16 determines whether each object extracted from the image is positioned close to the robot or far away from it through the control unit (250). Furthermore, the control unit (250) calculates the fitness for the characteristic objects of the objects based on the order in which they are positioned closest to the robot among one or more objects.

This is also intended to improve the accuracy of images and characteristic objects stored on the map. Objects placed farther away have lower accuracy within the image. Therefore, storing objects placed closer together as characteristic objects is advantageous for subsequent location estimation. Therefore, in S16, fixed and nearby objects are selected as characteristic objects.

The control unit (250) calculates the similarity between the selected objects and previous characteristic objects (S17). In addition, the control unit (250) calculates the similarity with objects stored in the characteristic object universal database (S18). Then, the control unit (250) completes the verification of the characteristic object based on the calculated results (S19).

Here, S17 and S18 can be performed independently of S16. That is, the control unit (250) can calculate a similarity by comparing each of the objects detected within the image with objects stored in the database or previously selected characteristic objects. Furthermore, the control unit (250) can calculate a suitability of the objects to the characteristic object based on the similarity. Depending on the suitability, the objects can be stored as characteristic objects.

Here, step S17 refers to the process of comparing the characteristic objects previously selected by the control unit (250) during the process of generating the same map. For example, the control unit (250) acquires images from different first spaces, second spaces, and third spaces during the process of generating the map. Then, the first characteristic object and the second characteristic object are set for the images of the first space and the second space, respectively, and stored in the map storage unit (210).

And the control unit (250) selects a third characteristic object from the image acquired in the third space. And compares the similarity with the first characteristic object and the second characteristic object.

If the similarity with the first characteristic object is high, it can be judged to be an important characteristic object.

On the other hand, if the similarity is low, it can be determined that it is a characteristic object that can only be confirmed in that space.

Accordingly, when the similarity is high, the control unit (250) can secure an additional characteristic object (fourth characteristic object) that can distinguish the difference between the first space and the third space within the image of the third space and store the third characteristic object and the fourth characteristic object in the map storage unit (210).

Additionally, if the similarity is low, the control unit (250) can store the third characteristic object in the map storage unit (210) as a characteristic object that can distinguish the third space from the first space or another space.

Meanwhile, the control unit (250) can compare objects stored in a database (a universal database of characteristic objects) that stores characteristic objects with a third characteristic object (S18). The universal database of characteristic objects is a database that stores images of objects that can be characteristic in a space where a map is created or a similar type of space.

For example, if the space you're creating a map for is a city, a feature-specific object database could store images of specific signage needed to perform SLAM within the city. For example, a feature-specific object database could store images of signs for numerous coffee shops, department stores, public phone booths, or convenience store signs.

The control unit (250) then calculates the similarity between the stored image and the characteristic object within the currently captured image. If the similarity is high, it is determined that the characteristic object within the image has been verified.

When verification of a characteristic object is completed based on the produced result, the verification result is reflected and stored as a characteristic object of the corresponding image in the map storage unit (210) (S19).

Meanwhile, in a process similar to that shown in Figure 5, an administrator can select one of the candidate characteristic objects. For example, various methods can be used to verify whether an object detected in an image corresponds to a characteristic object.

In one embodiment, the control unit (250) controls the interface unit (290) so that the interface unit (290) outputs an object detected on an image and a bounding box around the object. When selection information regarding a specific object or a specific bounding box is input to the interface unit (290) from an administrator, the control unit (250) can set the selected object or the object included in the selected bounding box as a characteristic object.

Alternatively, the control unit (250) may exclude dynamic objects (people, cars, bicycles, animals, etc.) from the objects within the image. On the other hand, the control unit (250) determines fixed objects (public telephone booths, patterns on building exterior walls, signs, etc.) from the objects within the image as characteristic objects.

Alternatively, the control unit (250) determines an object placed close to the image as a characteristic object with priority over an object placed far away.

Figure 6 illustrates the arrangement of bounding boxes around a characteristic object according to one embodiment of the present invention. Following the process described above, the control unit (250) selects a characteristic object within the image, as shown in Figure 6.

First, the control unit (250) determines the areas indicated by 21, 22, 23, and 24 as candidates for characteristic objects.

And for each candidate, the control unit (250) performs a judgment as shown in Fig. 5. As a result, the control unit (250) judges 21 as a person (dynamic area) and 22 as a vehicle (dynamic area).

Meanwhile, the control unit (250) determines that 24 is a nearby object, a public telephone booth, and that 23 is a distant object, the exterior of a building. As a result, the control unit (250) determines that 24, a nearby and static object among the candidate characteristic objects (21, 22, 23, 24), is the characteristic object. In addition, the bounding box can be set to include the boundary of the public telephone booth.

The control unit (250) can store the bounding box and the corresponding image as a single image or separately. The bounding box can be determined by coordinate values within the image. Accordingly, the control unit (250) can store the image and separately store the coordinate values for the bounding box.

Bounding boxes are necessary for identifying the locations of images and characteristic objects stored on the map during the robot's future position estimation process, when the robot searches for and compares characteristic objects within the image from the map storage unit. Therefore, the map storage unit can store coordinate information indicating the location and size of the bounding boxes of characteristic objects within the image.

For example, the map storage unit (210) may store the upper left and lower right coordinates of the bounding box within the image. Alternatively, the map storage unit (210) may store the upper left coordinates (or specific vertex coordinates such as the lower right coordinates) and width/height information of the bounding box within the image.

Meanwhile, the robot can determine a characteristic object within a single image, but it can also determine a characteristic object among two or more consecutive images.

Figure 7 illustrates a process by which a robot, according to one embodiment of the present invention, determines a characteristic object. Figure 7 illustrates a process by which the robot's control unit (250) verifies an object commonly included in a first image acquired at a first time point and a second image acquired at a second time point subsequent to the first time point as a characteristic object.

26 shows four images acquired during the movement of the robot (100) and characteristic objects (28) within the images. During the movement, time t varies from 1 to 4. The image acquired at time t=1 is image1, the image acquired at time t=2 is image2, the image acquired at time t=3 is image3, and the image acquired at time t=4 is image4.

And 26 shows that the position of the characteristic object (28) within each image changes. This is because the relative position between the characteristic object and the robot changes as the robot (100) moves.

Accordingly, the robot (100) judges an object (28) that appears commonly in four images as a characteristic object and stores it.

At this time, the method of storing in the map storage unit (210) is as shown in 27. 27 is an example of a SLAM pose graph for each time point. The pose graph includes nodes (N1 to N9) corresponding to each time point and edges between each node, as shown in 27. Three pieces of information are stored in each node (N1 to N4) of the pose graph.

In one embodiment, an image acquired at time t=1, i.e., a raw image, is stored, and a characteristic object (feature) within the raw image is stored separately. The characteristic object stored in the map storage unit (210) may be stored as an image extracted along the outline of the characteristic object within the raw image.

The bounding box coordinates can also be stored in the map storage, indicating where the characteristic object is located within the raw image.

In order to store the poses in which the 28 characteristic objects were photographed, the map storage unit (210) stores the characteristic objects, raw images, and bounding box coordinates corresponding to nodes (N1 to N4) for each time point.

The edges between nodes between each time point (t=1 to 4) can store wheel odometry generated during the robot's movement between each node, or information acquired through other sensors.

Likewise, the robot (100) can acquire images at different points in space (t=k~ t=k+2), extract new characteristic objects (29) in the same manner as the aforementioned process, and store them in the map storage.

29 is a newly identified characteristic object. Similarly, the control unit (250) stores the location and image information where the characteristic object is identified in the map storage unit (210). The control unit (250) stores the characteristic object (29) and the raw images (image_k to image_k2) from N7 to N9, and the bounding box coordinates indicating the location of the characteristic object (29) within the raw images in the map storage unit (210).

The information stored as in 27 of Fig. 7 is compared with the images acquired during the movement of the robot (100).

Fig. 7 corresponds to the mapping process. In this process, the control unit (250) determines the characteristic object or candidate regions that can be the characteristic object through an object detection method in the first images (t=1 or t=k) in which the characteristic object is confirmed. In this process, "YOLOv3", "Mask R-CNN", etc. can be applied as embodiments for object detection. Once the candidate regions that can be the characteristic object are selected, the control unit (250) performs verification on the characteristic object as shown in Fig. 5 and selects the characteristic object to be stored in the map storage unit (210) among the candidate regions.

Alternatively, the interface unit (290) outputs candidate bounding boxes indicating candidate areas. Then, the administrator can select a box suitable for a characteristic object from among these output boxes.

Alternatively, even if the administrator does not make a separate selection, the interface unit (290) displays the area to be saved as a characteristic object in the image as a bounding box so that the administrator of the robot (100) can check the process of creating a map and saving a characteristic object.

When a number of characteristic objects are identified within an image, as shown in FIG. 5, the control unit (250) can select the characteristic objects based on the location within the image or the dynamic or fixed properties of the objects themselves. In addition, the control unit (250) can store a history of characteristic objects that are frequently used within the map due to their high similarity to images selected in the past during map creation or stored in a general database of characteristic objects, in the map storage unit (210) or a separate storage medium. In addition, the control unit (250) stores a portion of the image suitable as a characteristic object as a characteristic object.

Alternatively, the interface unit (290) may recommend more useful characteristic objects to enable the administrator to select a bounding box. The bounding box of Figure 24 in Figure 6 may be output with a thicker or clearer line to encourage the administrator to select the bounding box of Figure 24.

When a characteristic object is extracted from the first image (t=1 or t=k), the control unit (250) can predict the next location of the characteristic object in subsequent images (t=2, 3, …, or t=k+1, k+2, …) using a multiple-object tracking method. Then, the interface unit (290) can output a bounding box at the predicted location. Alternatively, the control unit (250) can automatically store the area of the predicted location as a characteristic object.

The Simple Online and Realtime Tracking (SORT) or Person of Interest (POI) method can be used for multi-object tracking. If the output bounding box deviates from the actual characteristic object's perimeter, the administrator can touch the interface (290) to correct it.

Figure 8 illustrates an example of comparing an image acquired by a robot during a movement process according to one embodiment of the present invention with an image stored in a map storage unit. Figure 31 shows the result of the robot acquiring the image and extracting a characteristic object. Figure 32 shows the process of the robot confirming its current location by comparing it with the information stored in the map storage unit.

The robot (100) obtains image_New. The robot (100) distinguishes a characteristic object, such as 30, within the obtained image. In addition, the robot (100) can extract the location coordinates of the characteristic object (30) within image_New.

The robot (100) searches for the extracted characteristic object (30) in the map storage. As shown in FIG. 7, the characteristic object is stored separately, so 30 can be compared with the characteristic objects stored in the map storage (210). As a result of the comparison, the robot finds that the characteristic object (35) stored in N8 is identical to 30. In addition, the robot (100) can extract the bounding box coordinates of the searched characteristic object (35).

And as a result of comparing the bounding box coordinates of the N8 node and the location coordinates of the characteristic object (30) in image_New, the robot (100) determines that the current location is the N8 node stored in the map storage.

In the process described in FIGS. 7 and 8, the robot (100) creates a map and stores characteristic objects, and then, in the slam process, the location of the robot can be confirmed based on the characteristic objects.

Figure 9 shows a localization process for generating pose information from a map storage unit using a characteristic object according to one embodiment of the present invention.

The camera sensor (230) of the robot (100) acquires an image (S35). Then, the control unit (250) generates a query image from the acquired image (S36). The query image refers to a portion of the image acquired by the camera sensor (230) for comparison with the images stored in the map storage unit (210). One example of the query image includes an image in which a characteristic object is extracted from an image acquired by the camera sensor (230).

The control unit (250) extracts candidate images similar to the query image from the map storage unit (210) (S37). In one embodiment, when a query image is generated, the control unit (250) extracts candidate images similar to the query image using a clustering method such as VocTree (Vocabulary Tree).

The control unit (250) then compares the query image and the bounding box area among the candidate images and calculates the node information, i.e., pose information, corresponding to the image determined to be most similar. The process of Fig. 9 corresponds to localization, i.e., the process of position estimation.

When performing SLAM based on characteristic objects, the accuracy of SLAM can be improved by preventing errors in location confirmation due to image similarity.

Figure 10 shows a process in which a robot detects movement of an object in continuously captured images according to one embodiment of the present invention and determines a characteristic object.

As illustrated in FIG. 7, the control unit (250) moves at a constant speed and captures surrounding images. Therefore, there is a high probability that the same object will be captured in two consecutive images, and FIG. 7 illustrates an embodiment of tracking such objects and storing them as characteristic objects.

Meanwhile, the control unit (250) can detect objects in two or more consecutive images, and then determine whether the objects in the two images are the same object by reflecting the change in the position of the objects in the two images and the movement distance of the robot.

In Fig. 10, an image (41) captured at a first time point (t=1) includes one object (51, first object). And a first embodiment of an image captured at a second time point (t=2) is 42. In the image indicated by 42, the dotted line indicates the position of the first object (51) in the first image (41).

On the other hand, the object (52, second object) in the second image (42) has moved horizontally by the size "dist1" from the position 51. If the similarity between the second object (52) in the second image (42) and the first object (51) in the first image (41) is above the standard, and the change called "dist1" is of a size corresponding to the movement distance of the robot, the control unit (250) determines that the second object (52) in the second image (42) and the first object (51) in the first image (41) are the same object. Then, the control unit (250) verifies whether these two objects (51, 52) are characteristic objects. As a result of the verification, the control unit (250) can store the corresponding object as a characteristic object in the map.

Meanwhile, the second embodiment of the image captured at the second time point (t=2) is 43. In the image indicated by 43, the dotted line indicates the position of the first object (51) in the first image (41).

On the other hand, the object (53, third object) in the third image (43) moved from position 51 by the size "dist2", but it did not move horizontally but in an inclined direction. Although the similarity between the third object (53) in the third image (43) and the first object (51) in the first image (41) is above the standard, the change called "dist2" may be of a size that does not correspond to the movement distance of the robot, or the movement direction of the object may be a completely different direction from the movement of the robot.

In this case, the control unit (250) determines that the third object (53) in the third image (43) and the first object (51) in the first image (41) are different objects. Then, the control unit (250) verifies whether each of these two objects (51, 53) is a characteristic object. Based on the verification result, the control unit (250) may or may not store the corresponding objects in the map as characteristic objects.

Because the first image (41) and the third image (43) at two points in time (t1, t2) contain different objects, both 51 and 53 are likely to be dynamic objects, so the control unit (250) does not determine them as characteristic objects.

To summarize, it is as follows.

When two images (images 1 and 2) are captured at different shooting times and locations, as shown in Figure 7, objects likely to be selected as distinctive objects are more likely to be included in both images. Furthermore, the locations of these objects within the two images also correspond to the robot's movement distance and direction.

To this end, the control unit (250) detects a first object within a first image and a second object within a second image. Furthermore, the control unit (250) reflects changes in the positions of the first and second objects and the distance traveled by the robot to determine whether the first and second objects are the same object. If they are the same object, the control unit (250) verifies whether the first and second objects are characteristic objects and stores the results in the map storage unit (210).

When verified as a characteristic object, the control unit (250) stores the first object, the first image, the bounding box for the first object, and the location information where the first image was acquired as a single unit in the map storage unit (210). Similarly, the control unit (250) stores the second object, the second image, the bounding box for the second object, and the location information where the second image was acquired as a single unit in the map storage unit (210).

Figure 11 illustrates a process in which the interface unit of a robot according to one embodiment of the present invention outputs a characteristic object and receives an input for selecting the object. The interface unit (290) outputs predetermined visual and auditory information. Furthermore, the interface unit (290) includes a touch screen, so that when a person selects the output information or moves the displayed information by dragging or other means, the interface unit (290) can process such input information.

The control unit (250) detects an object within an image (S61). The control unit (250) can then calculate the location of a bounding box surrounding the detected object. The interface unit (290) then outputs the detected object and the bounding box positioned around the object (S62).

This allows the interface unit (290) to output an area that can be a characteristic object at the current location, so that a user or administrator can modify or confirm it.

In particular, when there are two or more objects detected in the image, the interface unit can output two or more bounding boxes corresponding to the two or more objects, and then input selection information for one of the two or more bounding boxes (S63).

Here, the selection information includes information about the user touching the area around or inside the bounding box. Alternatively, the selection information includes information about the user touching and dragging the area around or inside the bounding box to modify the position or size of the bounding box.

The robot's control unit (250) stores objects within a bounding box into which selection information has been input as characteristic objects. In this process, the control unit (250) can extract objects based on the modified bounding box if the location or size of the bounding box has been modified.

Figure 12 shows the result of modifying a bounding box according to one embodiment of the present invention. The control unit (250) identifies a public telephone booth and creates a bounding box in the rectangle indicated by 68, and the interface unit (290) outputs the first bounding box (68).

Here, the user or administrator confirms that the first bounding box (68) displayed in the interface section (290) includes only a portion of the actual public telephone booth, and then drags or resizes the first bounding box (68) to modify it into the second bounding box (69).

As this modification process progresses, the control unit (250) extracts a characteristic object centered on the second bounding box, and then stores the characteristic object, the second bounding box, and the image of FIG. 12 and the location information of the robot from which the image was acquired in the map storage unit (210).

Based on the above-described embodiments, the artificial intelligence unit (255) of the robot (100) or the artificial intelligence unit (355) of the cloud server (300) can classify the input image when extracting an object from an image, determining whether the object is a dynamic object or a fixed object, or determining the perspective of the object.

Alternatively, the artificial intelligence unit (255) of the robot (100) or the artificial intelligence unit (355) of the cloud server (300) can determine the boundary of a characteristic object in an input image and output information about the area of a bounding box. The configuration of the artificial intelligence unit (255) of the robot (100) or the artificial intelligence unit (355) of the cloud server (300) will be examined in more detail.

Artificial intelligence (AI) is the study of artificial intelligence or the methodologies for creating it, while machine learning (ML) defines various problems in the field of AI and studies the methodologies for solving them. Machine learning is also defined as an algorithm that improves performance on a task through consistent experience.

An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

Supervised learning refers to a method for training an artificial neural network when given labels for the training data. The labels can refer to the correct answer (or output value) that the artificial neural network must infer when the training data is input to the artificial neural network. Unsupervised learning can refer to a method for training an artificial neural network when the training data is not given labels. Reinforcement learning can refer to a learning method in which an agent defined within a given environment is trained to select actions or action sequences that maximize the cumulative reward in each state.

Machine learning implemented with a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, the term "machine learning" is used to encompass deep learning.

The robot (100) can perform artificial intelligence functions through the artificial intelligence unit (255), which is a sub-component of the control unit (250) discussed above. The artificial intelligence unit (255) within the control unit (250) can be configured as software or hardware.

In this case, the communication unit (280) of the robot (100) can transmit and receive data with robots that provide other AI functions or external devices such as a cloud server (300) as discussed in FIG. 13 using wired or wireless communication technology. For example, the communication unit (280) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

At this time, the communication technologies used by the communication unit (280) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

The interface unit (290) can obtain various types of data.

At this time, the interface unit (290) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input unit for receiving information from a user, etc. Here, the information acquired by the lidar sensor (220), camera sensor (230), or microphone refers to sensing data or sensor information, etc.

The interface unit (290) and various sensors, as well as the wheel encoder (260) of the moving unit, can obtain input data to be used when obtaining output using learning data and learning models for model learning. The aforementioned components can also obtain unprocessed input data, in which case the control unit (250) or the artificial intelligence unit (255) can extract input features as preprocessing for the input data.

The artificial intelligence unit (255) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network can be referred to as a learning model. The learning model can be used to infer result values for new input data other than the training data, and the inferred values can be used as a basis for determining whether the robot (100) should perform a certain action.

At this time, the artificial intelligence unit (255) of the robot (100) can perform AI processing together with the artificial intelligence unit (355) of the cloud server (300).

At this time, the artificial intelligence unit (255) of the robot (100) may include a memory integrated or implemented in the robot (100). Alternatively, the artificial intelligence unit (255) of the robot (100) may be implemented using a separate memory, an external memory coupled to the robot (100), or a memory maintained in an external device.

The robot (100) can obtain at least one of internal information of the robot (100), information about the surrounding environment of the robot (100), and user information by using various sensors.

The memory built into the robot (100) can store data that supports various functions of the robot (100). For example, it can store input data, learning data, learning models, learning history, etc. acquired by various sensors or interface units (290) built into the robot (100).

The control unit (250) can determine at least one executable operation of the robot (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm. Then, the control unit (250) can control components of the robot (100) to perform the determined operation.

To this end, the control unit (250) can request, search, receive or utilize data from the artificial intelligence unit or memory, and control components of the robot (100) to execute at least one predicted action among executable actions or an action determined to be desirable.

At this time, if connection of an external device is required to perform a determined operation, the control unit (250) can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

The control unit (250) can obtain intention information for user input and determine the user's requirements based on the obtained intention information.

Meanwhile, the control unit (250) can extract feature points from sensor data acquired in real time, such as image sensor data or lidar sensor data. To this end, the FEM sub-module of the artificial intelligence unit (255) can be configured as an artificial neural network trained according to a machine learning algorithm. In addition, the artificial intelligence unit (255) of the robot (100) can be trained, or trained by the artificial intelligence unit (355) of the cloud server (300), or trained by distributed processing of these.

The control unit (250) can collect history information including the operation details of the robot (100) or the user's feedback on the operation, and store it in the memory or artificial intelligence unit (255), or transmit it to an external device such as a cloud server (300). The collected history information can be used to update the learning model.

Figure 13 shows another configuration of a cloud server according to one embodiment of the present invention.

A cloud server (300) that provides the functionality of an artificial intelligence server, or AI server, may refer to a device that trains an artificial neural network using a machine learning algorithm or utilizes a trained artificial neural network. Here, the cloud server (300) may be comprised of multiple servers to perform distributed processing, and may be defined as a 5G network.

The cloud server (300) includes a communication unit (380), a server control unit (350), an artificial intelligence unit (355), etc., each of which has been described in FIG. 3. Meanwhile, the cloud server (300) may further include a memory (330).

The memory (330) may include a model storage unit (331). The model storage unit (331) may store a model (or artificial neural network, 331a) being learned or learned through the artificial intelligence unit (355).

The artificial intelligence unit (355) can train an artificial neural network (331a) using learning data. The learning model can be used while mounted on the cloud server (300) of the artificial neural network, or can be mounted on an external device such as a robot (100).

The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (330).

The server control unit (350) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

Although all components constituting the embodiments of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to such embodiments, and within the scope of the present invention, all components may be selectively combined and operated one or more times. In addition, although all of the components may be implemented as individual independent hardware, some or all of the components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions combined in one or more hardware pieces. The codes and code segments constituting the computer program may be easily inferred by those skilled in the art of the present invention. Such a computer program may be stored in a computer-readable storage medium and read and executed by a computer, thereby implementing the embodiments of the present invention. Storage media for computer programs include magnetic recording media, optical recording media, and storage media including semiconductor recording devices. In addition, a computer program implementing an embodiment of the present invention includes a program module that is transmitted in real time through an external device.

While the above description focuses on specific embodiments of the present invention, various modifications and variations can be made within the scope of those skilled in the art. Therefore, it should be understood that such modifications and variations are within the scope of the present invention, as long as they do not depart from its scope.

1: Robot 150: Control module
210: Map storage 220: Lidar sensor
230: Camera sensor 250: Control unit
300: Cloud Server 350: Server Control Unit

Claims (20)

A moving part that moves the robot;
A camera sensor that acquires images necessary for the robot to perform simultaneous localization and mapping (SLAM);
A map storage unit that stores the necessary information for the robot to perform a slam; and
A control unit that verifies and extracts a characteristic object from the image and stores the location information where the image was acquired, the characteristic object, and the coordinate information of the characteristic object within the image in the map storage unit,
A robot that performs slam based on a characteristic object in an image, wherein the control unit extracts a characteristic object from the image and predicts the location of the characteristic object in subsequent images using a multi-object tracking method. In the first paragraph,
The control unit detects one or more objects within the image and verifies whether the detected objects correspond to characteristic objects.
A robot that performs slam based on a characteristic object in an image, wherein the control unit stores the object as a characteristic object in the map storage unit according to the verified result.
In the second paragraph,
A robot that performs slam based on a characteristic object in an image, wherein the control unit determines whether each object is a dynamic object or a fixed object, and if the object is a fixed object, determines it as a characteristic object.
In the second paragraph,
A robot that performs slam based on characteristic objects in an image, wherein the control unit determines whether each object is placed close to the robot or far away, and calculates the suitability of the characteristic objects of the objects according to the order in which they are placed closest among the one or more objects.
In the second paragraph,
The above control unit compares each object with objects stored in a database or previously selected characteristic objects and calculates the similarity.
A robot that performs slam based on characteristic objects in an image, wherein the control unit calculates the suitability for characteristic objects of the objects according to the similarity.
In the first paragraph,
A robot that performs slam based on a characteristic object in an image, wherein the control unit verifies an object commonly included in a first image acquired at a first time point and a second image acquired at a second time point subsequent to the first time point as a characteristic object.
◈Claim 7 was waived upon payment of the registration fee.◈ In paragraph 6,
The above control unit detects a first object within the first image,
The above control unit detects a second object within the second image,
A robot that performs a slam based on a characteristic object in an image, which verifies whether the first object and the second object are characteristic objects when the first object and the second object are the same object by reflecting the change in the position of the first object and the second object and the movement distance of the robot.
In the first paragraph,
A robot that performs a slam based on a characteristic object within an image, wherein the coordinate information includes information about the position and size of a bounding box containing the characteristic object within the image.
◈Claim 9 was waived upon payment of the registration fee.◈ In paragraph 8,
The above robot further includes an interface unit for outputting information,
After the above control unit detects an object in the image,
A robot that performs slam based on a characteristic object in an image, wherein the interface unit outputs the image, the detected object, and a bounding box placed around the object.
◈Claim 10 was waived upon payment of the registration fee.◈ In paragraph 9,
There are more than two objects detected in the above image,
The above interface unit outputs two or more bounding boxes corresponding to the two or more objects, and then receives selection information for one of the two or more bounding boxes.
A robot that performs slam based on a characteristic object in an image, wherein the control unit stores an object within a bounding box into which the selection information is input as a characteristic object.
A communication unit that receives images required to perform simultaneous localization and mapping (SLAM) from a robot;
A map storage unit that stores the necessary information for the robot to perform a slam; and
A server control unit that verifies and extracts a characteristic object from the image and stores the location information where the image was acquired, the characteristic object, and the coordinate information of the characteristic object within the image in the map storage unit,
The above server control unit is a cloud server that extracts a characteristic object from the image and predicts the location of the characteristic object in subsequent images using a multi-object tracking method. In Article 11,
The server control unit detects one or more objects within the image and verifies whether the detected objects correspond to characteristic objects.
A cloud server in which the server control unit stores the object as a characteristic object in the map storage unit according to the verified result.
◈Claim 13 was waived upon payment of the registration fee.◈ In Article 12,
A cloud server in which the server control unit determines whether each object is a dynamic object or a fixed object, and if the object is a fixed object, determines it as a characteristic object, or the server control unit determines whether each object is an object placed close to the robot or an object placed far away, and calculates the suitability of the objects for the characteristic object according to the order of placement of the closest objects among the one or more objects.
◈Claim 14 was waived upon payment of the registration fee.◈ In Article 12,
The above server control unit compares each object with objects stored in the database or previously selected characteristic objects and calculates the similarity.
A cloud server in which the server control unit calculates the suitability for characteristic objects of the objects based on the similarity.
In Article 11,
A cloud server in which the server control unit verifies an object commonly included in a first image acquired at a first time point and a second image acquired at a second time point subsequent to the first time point as a characteristic object.
delete delete delete delete delete