KR102838414B1 - Method and device for recognizing toys - Google Patents

Abstract

A toy recognition method performed by a processor is disclosed. The toy recognition method includes a step of receiving an image of a toy captured by a camera, and a step of applying the image to a first neural network to identify a type of the toy.

Description

{Method and device for recognizing toys}

The present invention relates to a method and device for recognizing a toy, and more particularly, to a method and device capable of recognizing a toy using a neural network.

If you receive donated toys, you can give them away if they are reusable, and if they are not reusable, you can recycle them into plastic materials. In the past, there was a problem that people had to sort the donated toys one by one, which took a lot of time and manpower.

Therefore, a method to classify toys more quickly using artificial intelligence is required.

Korean Patent Publication No. 10-1979122 (May 9, 2019)

The technical problem to be achieved by the present invention is to provide a method and device capable of efficiently recognizing a toy using a neural network.

A toy recognition method performed by a processor according to an embodiment of the present invention includes the steps of receiving an image of a toy captured by a camera, and the steps of applying the image to a first neural network to identify the type of the toy.

According to an embodiment, the toy recognition method may further include a step of applying the image to a second neural network to distinguish a state of the toy.

When the output of the first neural network is below a threshold, a step of applying the image to the second neural network to distinguish the state of the toy can be performed.

According to an embodiment, the toy recognition method may further include a step of distinguishing the toy as a new type of toy when the state of the toy is not distinguishable.

A computing device according to an embodiment of the present invention includes a processor for executing toy recognition commands, and a memory for storing the commands.

The above commands are implemented to receive an image of a toy captured by a camera and apply the image to a first neural network to identify the type of the toy.

According to an embodiment, the above commands may be further implemented to apply the image to a second neural network to distinguish the state of the toy.

When the output of the first neural network is below a threshold, commands for applying the image to the second neural network to distinguish the state of the toy can be executed.

The above commands may be further implemented to distinguish the toy into a new toy type when the state of the toy is not distinguishable.

The above toy recognition method includes a step of determining whether the year of manufacture of the toy is older than a first specific year based on a first specific year when the year of manufacture of the toy is identified by applying the image to a first neural network; a step of applying an image corresponding to the toy whose year of manufacture is identified to a third neural network when the year of manufacture of the toy is determined to be older than the first specific year and distinguishing a state of the toy included in the image corresponding to the toy whose year of manufacture is identified; and a step of applying the image corresponding to the toy whose year of manufacture is identified to a fourth neural network when the year of manufacture of the toy is determined to be not older than the first specific year and older than the second specific year and distinguishing a state of the toy included in the image corresponding to the toy whose year of manufacture is identified.

The number of hidden layers and the number of nodes in the hidden layers of the third neural network are greater than the number of hidden layers and the number of nodes in the hidden layers of the fourth neural network.

A toy recognition method and device according to an embodiment of the present invention has the effect of identifying the type of a toy by applying an image of the toy captured by a camera to a neural network.

In addition, the toy recognition method and device according to an embodiment of the present invention have the effect of identifying the state of a toy by applying an image of the toy captured by a camera to a neural network.

In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.
Figure 1 shows a block diagram of a toy recognition system according to an embodiment of the present invention.
FIG. 2 illustrates a conceptual diagram for identifying the type of toy performed by the computing device illustrated in FIG. 1.
Figure 3 shows an exemplary image of a toy predicted by the neural network illustrated in Figure 2.
Figure 4 illustrates a conceptual diagram for distinguishing status grades of a toy performed by the computing device illustrated in Figure 1.
Figure 5 illustrates another conceptual diagram for distinguishing status grades of a toy performed by the computing device illustrated in Figure 1.
Figure 6 shows a flow chart of a toy recognition method according to an embodiment of the present invention.

Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely exemplified for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

Since embodiments according to the concept of the present invention can have various changes and can have various forms, the embodiments are illustrated in the drawings and described in detail in the present specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, but includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

The terms first or second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only intended to distinguish one component from another, for example, without departing from the scope of the rights according to the concept of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When it is said that an element is "connected" or "connected" to another element, it should be understood that it may be directly connected or connected to that other element, but that there may be other elements in between. On the other hand, when it is said that an element is "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Other expressions that describe the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. It should be understood that, as used herein, the terms "comprises" or "has" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common use dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the attached drawings.

Figure 1 shows a block diagram of a toy recognition system according to an embodiment of the present invention.

Referring to FIG. 1, the toy recognition system includes a computing device (10) and a camera (20) for recognizing a toy.

The computing device (10) includes a processor (11) that executes commands for recognizing a toy and a memory (13) that stores the commands. The computing device (10) may be implemented as various electronic devices such as a smart phone, a tablet PC, a laptop, a server, or a personal PC. Hereinafter, the operation of the processor (11) may be understood as the operation of the computing device (10).

The camera (20) can capture an image of a toy (40) moving on a conveyor belt (30). The toy (40) may be a toy that has been used by another user and may have some parts missing or may be in a state of use that makes it difficult to reuse.

The purpose of the present invention is to photograph a toy (40) with a camera (20) and have a computing device (10) identify the image generated to distinguish the type of the toy (40) and identify its status.

FIG. 2 illustrates a conceptual diagram for identifying the type of toy performed by the computing device illustrated in FIG. 1.

Referring to FIGS. 1 and 2, the processor (11) receives an image (50) of a toy (40) captured by a camera (20).

The processor (11) applies the image (50) to the first neural network (60) to identify the type of toy (40). The first neural network (60) can be implemented with the YOLOv8 Object Detection algorithm.

Training images are used to train the first neural network (60). The training images are created by accurately setting up a toy (40) on a conveyor belt (30) and using a camera (20) to photograph the toy (40) from various angles.

The type of the toy (40) can be identified by applying the image (50) to the trained first neural network (60). For example, in FIG. 2, Class 1 can be an 'a' toy of any 'A' brand, and Class 2 can be a 'b' toy of any 'B' brand. In another embodiment, in FIG. 2, Class 1 can be identified as a block, and Class 2 can be identified as a dinosaur toy. The identification method for the type of the toy (40) can vary depending on the training of the first neural network (60).

Class 1 and Class 2 shown in Fig. 2 are outputs of the first neural network (60) and represent the types of toys (40). In Fig. 2, only Class 1 and Class 2 are shown for convenience of explanation, but the number of outputs of the first neural network (60) may vary depending on the embodiment.

The outputs of the first neural network (60) can be expressed as values between 0 and 1. The brand of the toy (40) corresponding to the output having the largest value among the outputs of the first neural network (60) is recognized. For example, when the number of outputs of the first neural network (60) is 4 and each output is 0.1, 0.2, 0.2, and 0.5, the brand of the toy (40) corresponding to the output of the first neural network (60) of 0.5 is predicted.

According to an embodiment, when an image (50) is input to the first neural network (60), the processor (11) can identify not only the brand of the toy (40) but also the year of manufacture. Training images tagged with the year of manufacture of the toy (40) can be used for the first neural network (60) to identify the year of manufacture of the toy (40). The year of manufacture means the year of manufacture. The toy (40) can be renewed every year. Therefore, the shape of the toy (40) can be different depending on the year of manufacture.

Figure 3 shows an exemplary image of a toy predicted by the neural network illustrated in Figure 2.

Referring to FIGS. 1 to 3, when an image (50) is input to a first neural network (60), a toy identified in the image (50) is displayed. A plurality of toys may be included in one image (50). When a plurality of toys are included in one image (50), an identification code and a probability of a toy brand may be displayed for each of the plurality of toys. For example, in the image illustrated in FIG. 3, “tfr_y_2” means an identification code of a toy, and “0.95” indicates the probability that the image of the toy illustrated in FIG. 3 is “tfr_y_2”. Depending on the embodiment, the identification code of the toy may be variously labeled.

Figure 4 illustrates a conceptual diagram for distinguishing status grades of a toy performed by the computing device illustrated in Figure 1.

Referring to FIGS. 1, 2, and 4, the processor (11) applies the image (51) to the second neural network (70) to distinguish the state of the toy (40).

The state of the toy (40) can be distinguished as a state in which the battery cover of the toy (40) is not present and a state in which the velcro is attached to the toy (40). Depending on the embodiment, the state of the toy (40) can be distinguished in various ways. For example, the state of the toy (40) can be distinguished as a state in which it can be recycled and a state in which it cannot be recycled. In FIG. 4, “Status1” can mean a state in which it can be recycled, and “Status2” can mean a state in which it cannot be recycled. In addition, the state of the toy (40) can be distinguished as a new product state, a nearly new product state, a recyclable state, and a state in which it cannot be recycled.

The second neural network (70) can be implemented with the YOLOv8 Object Segmentation algorithm.

When the image (51) is applied to the first neural network (60) and the output of the first neural network (60) is less than a threshold (e.g., 0.5), the processor (11) applies the image (51) to the second neural network (70) to distinguish the state of the toy (40). The image (51) means an image including a toy (40) with a specific part missing. For example, the image (51) may be a robot toy with an arm missing. Even if the image (51) including the toy (40) with a specific part missing is applied to the first neural network (60), the toy (40) may lose the specific part, so the first neural network (60) may not properly recognize the toy (40). That is, the output of the first neural network (60) may be less than a threshold (e.g., 0.5). Therefore, when the image (51) is applied to the first neural network (60) and the output of the first neural network (60) is less than a threshold (e.g., 0.5), the processor (11) applies the image (51) to the second neural network (70) to distinguish the state of the toy (40). That is, the processor (11) applies the image (51) to the second neural network (70) to determine whether the toy (40) is reusable. At this time, even if the output of the first neural network (60) is less than the threshold by applying the image (51) to the first neural network (60), the identified brand of the toy (40) corresponding to the output of the first neural network (60) is recognized as the toy (40) included in the image (51).

For reference, the image (50) shown in Fig. 2 means an image including a toy (40) in which a specific part is not lost.

In order to train the second neural network (70), training images labeled as non-recyclable for robots without arms or robots without legs, and training images labeled as recyclable for toys without battery covers or toys with Velcro attached, may be used. Depending on the embodiment, the states of various toys may be labeled as recyclable or recyclable.

When the image (50) is applied to the first neural network (60) and the output of the first neural network (60) is greater than the threshold (e.g., 0.5), the processor (11) applies the image (50) to the second neural network (70) and classifies the toy (40) included in the image (50) as a recyclable state without distinguishing the state of the toy (40). This is because when the type of the toy (40) is identified through the first neural network (60), the toy (40) is properly identified because a specific part of the toy (40) is not lost, and thus can be classified as a recyclable state. Therefore, when the image (50) is applied to the first neural network (60) and the output of the first neural network (60) is greater than the threshold, the state of the toy (40) included in the image (50) can be classified as a recyclable state without applying the image (50) to the second neural network (70).

According to an embodiment, the processor (11) applies the image (50) to the first neural network (60) and, even if the output of the first neural network (60) is greater than a threshold (e.g., 0.5), randomly applies the image (50) to the second neural network (70) to distinguish the state of the toy (40) included in the image (50). The state of the toy (40) may be a recyclable state or a non-recyclable state.

Figure 5 illustrates another conceptual diagram for distinguishing status grades of a toy performed by the computing device illustrated in Figure 1.

Referring to FIGS. 1 and 5, when an image (53) is applied to a first neural network (60) according to an embodiment and the production year of a toy (40) included in the image (53) is identified from the output of the first neural network (60), the processor (11) determines whether the production year of the toy (40) is older than a first specific year (e.g., 2010) based on the first specific year.

When it is determined that the production year of the toy (40) is older than the first specific year, the processor (11) can apply the image (53) corresponding to the toy (40) with the identified production year to the third neural network (80) to distinguish the condition of the toy (40) included in the image (53) corresponding to the toy (40) with the identified production year. At this time, the condition of the toy (40) is distinguished according to the degree of use of the toy (40). If it is determined that the degree of use of the toy (40) is high, it can be determined that the condition of the toy (40) is not recyclable. Conversely, if it is determined that the degree of use of the toy (40) is low, it can be determined that the condition of the toy (40) is recyclable. The degree of use refers to the degree of wear and tear of the toy (40) due to heavy use.

The processor (11) can convert the image (53) to grayscale before applying the image (53) to the third neural network (80).

In order to judge the condition of a toy (40) according to the usability of the toy (40), the third neural network (80) uses training images with different pixel values. The different training images refer to images of the same toy (40) with different pixel values. The processor (11) can change the pixel values corresponding to the toy (40) in one training image including the toy (40) to generate the different training images, thereby generating images with different pixel values. It is determined that the larger the pixel value, the less the usability, and the smaller the pixel value, the greater the usability. In order to generate the training images with different pixel values, the processor (11) can convert one training image into grayscale.

A third neural network (80) is trained using training images so that any pixel value becomes a criterion for determining whether it can be reused or not.

When a toy included in an image has an arbitrary pixel value, the processor (11) can use the arbitrary pixel value as a criterion for a state in which it can be recycled or not. When the pixel value has a range from 0 to 255, the arbitrary pixel value can be 80. When the pixel value of the toy (40) included in the image (53) is smaller than the arbitrary pixel value, the processor (11) applies the image (53) to the third neural network (80), and the output of the third neural network (80) determines that the toy (40) included in the image (53) is not recyclable. This is because the processor (11) determines that the toy (40) included in the image (53) has been used for a long time.

Conversely, when the pixel value of the toy (40) included in the image (53) is greater than the arbitrary pixel value, the processor (11) applies the image (53) corresponding to the toy (40) to the third neural network (80), and the output of the third neural network (80) determines that the toy (40) included in the image (53) is in a state where it can be reused. This is because the processor (11) determines that the toy (40) included in the image (53) has little wear and tear.

According to an embodiment, when the image (55) is applied to the first neural network (60) and the production year of the toy (40) included in the image (50) is identified from the output of the first neural network (60), and the production year of the toy (40) is determined to be not older than the first specific year, the processor (11) determines whether the production year of the toy (40) is older than the second specific year (e.g., 2015). The second specific year (e.g., 2015) means a year later than the first specific year (e.g., 2010).

When it is determined that the production year (e.g., 2017) of the toy (40) is not older than the first specific year (e.g., 2010) and older than the second specific year (e.g., 2015), the processor (11) applies the image (55) corresponding to the toy (40) with the identified production year to the fourth neural network (90) to distinguish the state of the toy (40) included in the image (55) corresponding to the toy (40) with the identified production year.

The third neural network (80) or the fourth neural network (90) can be implemented using a CNN (Convolutional Neural Network) algorithm. The third neural network (80) and the fourth neural network (90) can be trained using the same training images. In Fig. 5, the outputs of the third neural network (80), “Status1” and “Status2,” represent states in which reuse is possible and states in which reuse is not possible. In addition, the outputs of the fourth neural network (90) in Fig. 5, “Status1” and “Status2,” represent states in which reuse is possible and states in which reuse is not possible.

However, the number of hidden layers and the number of nodes in the hidden layers of the third neural network (80) are greater than the number of hidden layers and the number of nodes in the hidden layers of the fourth neural network (90). The year of manufacture of the toy (40) included in the image (55) applied to the fourth neural network (90) is more recent than the year of manufacture of the toy (40) included in the image (53) applied to the third neural network (80). Therefore, the toy (40) included in the image (55) applied to the fourth neural network (90) is more likely to have less wear than the toy (40) included in the image (53) applied to the third neural network (80). Therefore, the number of hidden layers of the third neural network (80) and the number of nodes in the hidden layers are made larger than the number of hidden layers of the fourth neural network (90), thereby allowing more precise prediction of the state of the toy (40) for a toy (40) expected to have been used a lot, and allowing quick prediction of the state of the toy (40) for a toy (40) expected to have been used a little.

According to an embodiment, when the ratio (e.g., 60%) of a state in which recyclability is predicted through the third neural network (80) or the fourth neural network (90) is greater than a certain ratio (e.g., 70%), the processor (11) changes an arbitrary pixel value that serves as a criterion for determining a state in which recyclability is possible or non-recyclable, and retrains the third neural network (80) or the fourth neural network (90) so that the ratio of a state in which recyclability is possible is lowered. Retraining means that the third neural network (80) or the fourth neural network (90) is retrained so that the ratio of a state in which recyclability is possible is lowered by changing the arbitrary pixel value. When the arbitrary pixel value is 80, the arbitrary pixel value may be changed to 100.

Lowering the percentage of recyclable toys is necessary for consumer satisfaction. In other words, if there are many toys in good condition, the percentage of recyclable toys is lowered so that only toys in relatively better condition can be recycled, which provides better satisfaction to consumers when reselling or distributing them. On the other hand, if there are few toys in good condition, the percentage of recyclable toys is not lowered in order to provide consumers with a certain number of recyclable toys.

In an embodiment, when the state of the toy (40) is not distinguished by applying the image (51) to the second neural network (70), the processor (11) adds the toy (40) as a new type of toy, and the first neural network (60) can be retrained so that the new type of toy is added.

Figure 6 shows a flow chart of a toy recognition method according to an embodiment of the present invention.

Referring to FIGS. 1 to 3, the processor (11) receives an image (50) of a toy (40) captured by a camera (20) (S10).

The processor (11) applies the image (50) to the first neural network (60) to identify the type of toy (40) (S20). The first neural network (60) can be implemented with the YOLOv8 Object Detection algorithm.

The processor (11) applies the image (51) to the second neural network (70) to distinguish the state of the toy (40) (S30). The second neural network (70) can be implemented with the YOLOv8 Object Segmentation algorithm.

Although the present invention has been described with reference to one embodiment shown in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the appended claims.

10: Computing device;
11: Processor;
13: Memory;
20: Camera;
30: Conveyor belt;
40: Toys;
50: Image;
60: First neural network;
70: Second neural network;

Claims (8)

In a toy recognition method performed by a processor,
A step of receiving an image of a toy captured by a camera;
A step of applying the above image to the first neural network to identify the type and year of manufacture of the toy;
A step of determining whether the manufacturing year of the above toy is older than the first specific year;
When it is determined that the production year of the above toy is older than the first specific year, a step of applying an image corresponding to the toy with the identified production year to a third neural network to distinguish the state of the toy included in the image corresponding to the toy with the identified production year;
When it is determined that the production year of the toy is not older than the first specific year, a step of determining whether the production year of the toy is older than the second specific year;
When it is determined that the production year of the above toy is not older than the first specific year and is older than the second specific year, a step of applying an image corresponding to the toy with the identified production year to a fourth neural network to distinguish the state of the toy included in the image corresponding to the toy with the identified production year is included.
The third neural network or the fourth neural network uses training images with different pixel values to determine the condition of the toy according to the usability of the toy.
The third neural network is trained using the training images so that any pixel value becomes a criterion for determining whether it can be reused or not.
A toy recognition method in which the number of hidden layers of the third neural network and the number of nodes of the hidden layers are greater than the number of hidden layers of the fourth neural network and the number of nodes of the hidden layers, so as to more precisely predict the state of the toy expected to have been used a lot, and to quickly predict the state of the toy expected to have been used a little. In the first paragraph, the toy recognition method.
A toy recognition method further comprising a step of applying the above image to a second neural network to distinguish the state of the toy. A toy recognition method, wherein in the second paragraph, when the output of the first neural network is below a threshold, a step of applying the image to the second neural network to distinguish the state of the toy is performed. In the second paragraph, the toy recognition method.
A toy recognition method further comprising a step of distinguishing the toy as a new type of toy when the state of the toy is not distinguished. a processor for executing toy recognition commands; and
Contains memory for storing the above commands,
The above commands are,
The toy receives images captured by the camera,
Applying the above image to the first neural network, the type and year of manufacture of the toy are identified.
Determine whether the production year of the above toy is older than the first specific year.
When it is determined that the production year of the above toy is older than the first specific year, the image corresponding to the toy with the identified production year is applied to the third neural network to distinguish the state of the toy included in the image corresponding to the toy with the identified production year.
When it is determined that the production year of the above toy is not older than the first specific year, it is determined whether the production year of the above toy is older than the second specific year.
When it is determined that the production year of the above toy is not older than the first specific year and is older than the second specific year, the image corresponding to the toy with the identified production year is applied to the fourth neural network to distinguish the state of the toy included in the image corresponding to the toy with the identified production year.
The third neural network or the fourth neural network uses training images with different pixel values to determine the condition of the toy according to the usability of the toy.
The third neural network is trained using the training images so that any pixel value becomes a criterion for determining whether it can be reused or not.
A computing device in which the number of hidden layers of the third neural network and the number of nodes of the hidden layers are greater than the number of hidden layers of the fourth neural network and the number of nodes of the hidden layers, so as to more precisely predict the state of the toy expected to have a lot of use, and to quickly predict the state of the toy expected to have a little use. In paragraph 5, the above commands,
A computing device further implemented to apply the above image to a second neural network to distinguish the state of the toy. A computing device in which, in the sixth paragraph, when the output of the first neural network is below a threshold, commands for applying the image to the second neural network to distinguish the state of the toy are executed. In paragraph 6, the above commands,
A computing device further implemented to distinguish said toy as a new type of toy when the state of said toy is not distinguishable.