KR102665453B1 - Apparatus and method for providing customized content based on gaze recognition - Google Patents

Abstract

A method of providing customized content based on gaze recognition according to an embodiment of the present invention is a method of providing customized content based on gaze recognition that is executed by at least one processor of a content providing device, wherein the user's face is detected based on an image sensor. Obtaining a recognition image; Obtaining a front view image of the user's face by controlling the image sensor based on the acquired facial recognition image; Generating gaze information including gaze direction and gaze fixation time information of the user based on the acquired front view image of the face; determining a display area corresponding to the generated gaze information as a region of interest; detecting an item matching the determined area of interest; and providing product information content for the detected item.

Description

Device and method for providing customized content based on gaze recognition {APPARATUS AND METHOD FOR PROVIDING CUSTOMIZED CONTENT BASED ON GAZE RECOGNITION}

The present invention relates to an apparatus and method for providing customized content based on gaze recognition. More specifically, it relates to an apparatus and method for providing customized content by estimating the level of interest of a device user in real time based on gaze recognition of the device user looking at a product.

Among the various advertising methods in the information age, advertising through display devices is a method of unilaterally providing advertising to a specific majority of viewers regardless of their level of interest.

The effectiveness of advertising varies greatly depending on the viewer's interest in the advertised product or personal preference. To increase the effectiveness of advertising, the viewer's tastes and consumption patterns are collected to provide personalized advertising.

However, providing customized advertisements by estimating interest based on this information has the problem that it cannot reflect the interest felt by viewers when actually consuming media.

In addition, in cases where advertisements can be displayed on a given device, there is a method of providing advertisements for products of interest through the selection of the person using the device, but since this requires user action, advertisements are provided unilaterally or automatically. Compared to other methods, the exposure of advertisements is bound to be low.

Therefore, when a user consumes media, there is a need for a method to automatically display advertisements by immediately reflecting the user's level of interest.

KR 10-0020002 B1

The present invention was conceived to solve the problems of the prior art as described above, and its purpose is to provide an apparatus and method for providing customized content based on gaze recognition.

Additionally, the present invention seeks to provide a device and method for providing customized content based on gaze recognition that automatically adjusts the viewing angle of a camera attached to the device in real time according to the user's physical characteristics.

Additionally, the present invention seeks to provide an apparatus and method for providing customized content based on gaze recognition that outputs content about a product the user is looking at.

Additionally, the present invention seeks to provide an apparatus and method for providing customized content based on gaze recognition by dividing the content output screen according to the number of users.

However, the technical problems to be achieved by the present invention and embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

A method of providing customized content based on gaze recognition according to an embodiment of the present invention is a method of providing customized content based on gaze recognition that is executed by at least one processor of a content providing device, wherein the user's face is detected based on an image sensor. Obtaining a recognition image; Obtaining a front view image of the user's face by controlling the image sensor based on the acquired facial recognition image; Generating gaze information including gaze direction and gaze fixation time information of the user based on the acquired front view image of the face; determining a display area corresponding to the generated gaze information as a region of interest; detecting an item matching the determined area of interest; and providing product information content for the detected item.

At this time, the step of controlling the image sensor based on the acquired face recognition image to obtain a front view image of the user's face includes providing the face recognition image as input data to a previously learned deep learning neural network; , Obtaining facial pose information of the user using output data of the deep learning neural network, and controlling the image sensor based on the acquired facial pose information.

In addition, the step of determining the display area corresponding to the generated gaze information as the interest area includes, if the user's gaze fixation time information satisfies a preset time or more, the display area corresponding to the user's gaze direction is selected as the interest area. It includes the step of determining the area.

In addition, the step of detecting an item matching the determined area of interest includes obtaining a user input for selecting an item page to be output based on the display area, and selecting an item matched to the area of interest within the selected item page. It includes a detection step.

Additionally, the step of determining the display area corresponding to the gaze information as the area of interest includes updating the area of interest with the display area corresponding to the changed gaze information when the gaze information changes.

In addition, the step of detecting an item matching the determined area of interest includes detecting at least one pixel in a display area matching the area of interest, and detecting a pre-stored item for the detected at least one pixel. Includes steps.

In addition, a method of providing customized content based on gaze recognition according to an embodiment of the present invention includes obtaining a user input for selecting at least one pixel in the display area, and selecting a pre-stored item for the selected at least one pixel. It further includes detecting and outputting product information content for the detected item.

Meanwhile, an apparatus for providing customized content based on gaze recognition according to an embodiment of the present invention includes at least one image sensor that acquires a face recognition image of a user; At least one display that outputs product information content based on the acquired facial recognition image; at least one processor; It includes, wherein the processor controls the image sensor based on the face recognition image to obtain a front view image of the user's face, and determines the user's gaze direction and gaze fixation based on the obtained front view image of the face. Generate gaze information including time information, determine the display area corresponding to the generated gaze information as an area of interest, detect an item matching the determined area of interest, and product information content for the detected item. provides.

Additionally, the display includes an item display that outputs a representative image for at least one item, and a content display that outputs the product information content.

In addition, the processor acquires facial pose information of the user based on deep learning based on the facial recognition image, and controls the image sensor based on the acquired facial pose information to produce a front view image of the user's face. Acquire.

An apparatus and method for providing customized content based on gaze recognition according to an embodiment of the present invention performs initial settings tailored to the user each time the user changes by automatically adjusting the viewing angle of the camera attached to the device in real time according to the user's physical characteristics. It has the effect of shortening the time required.

In addition, the device and method for providing customized content based on gaze recognition according to an embodiment of the present invention intuitively identifies the product of interest to the user by outputting content about the product the user is looking at, thereby improving the user's satisfaction according to content exposure. It has an effect.

In addition, the device and method for providing customized content based on gaze recognition according to an embodiment of the present invention divides the content output screen according to the number of users, thereby efficiently using the content output screen to deliver advertisements and information to each user. It has the effect of increasing .

However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the description below.

1 is an internal block diagram of a device that provides customized content based on gaze recognition according to an embodiment of the present invention.
Figure 2 is an example of a diagram for explaining a device that provides customized content based on gaze recognition according to an embodiment of the present invention.
Figure 3 is a flowchart illustrating a method of providing customized content based on gaze recognition according to an embodiment of the present invention.
Figure 4 is an example of a diagram for explaining a method of estimating a face key point according to an embodiment of the present invention.
Figure 5 is an example of a drawing for explaining gaze information for tracking gaze direction according to an embodiment of the present invention.
Figure 6 is an example of a diagram for explaining a method of setting a region of interest based on gaze information according to an embodiment of the present invention.
Figure 7 is a flowchart illustrating a method of providing customized content based on gaze recognition according to another embodiment of the present invention.
Figure 8 is an example showing a plurality of customized contents based on gaze recognition being output according to another embodiment of the present invention.

Since the present invention can be modified in various ways and can have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used not in a limiting sense but for the purpose of distinguishing one component from another component. Additionally, singular expressions include plural expressions unless the context clearly dictates otherwise. Additionally, terms such as include or have mean that the features or components described in the specification exist, and do not preclude the possibility of adding one or more other features or components. Additionally, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are shown arbitrarily for convenience of explanation, so the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and redundant description thereof will be omitted. .

A device that provides customized content based on gaze recognition according to an embodiment of the present invention automatically adjusts the angle of a camera attached to the device in real time based on images captured of the user, recognizes the user's gaze, and It is possible to provide customized content based on gaze recognition, which is content containing product information about the item being gazed at.

In the embodiment, items refer to goods and services that can be sold to users using a gaze recognition-based customized content provision device.

Additionally, in the embodiment, product information means advertisements and additional information about items that can be sold to users using the device.

- Device for providing customized contents based on gaze recognition (100)

Figure 1 is an internal block diagram of a device that provides customized content based on gaze recognition (hereinafter referred to as content providing device 100) according to an embodiment of the present invention.

Referring to FIG. 1, the content providing device 100 according to an embodiment of the present invention may be a computing device installed with a processor assembly 120 that provides customized content based on gaze recognition.

In an embodiment, the content providing device 100 may provide a purchasing environment in which a user can purchase a selected item.

Additionally, in an embodiment, the content providing device 100 may provide a content output environment that outputs content including advertisements and additional information about the item the user is looking at on a display included in the device.

Additionally, in the embodiment, the content providing device 100 may acquire an image of the user's front view.

In the most preferred embodiment, this content providing device 100 may be implemented as a vending machine (hereinafter referred to as a vending machine) or a kiosk (Kiosk) that is easy to use in an environment where consumers generally purchase items. In the embodiment, the content providing device 100 is described based on the vending machine type, but is not limited thereto.

In an embodiment, the processor assembly 120 may be installed in the content providing device 100 described above and perform various functional operations performed by the content providing device 100.

Additionally, in the embodiment, the content providing device 100 may further include a server computing device to provide a content output environment.

Meanwhile, referring further to FIG. 1, from a functional perspective, the content providing device 100 includes a memory 110, a processor assembly 120, a communication processor 130, a payment unit 140, an input system 150, and a sensor. It may include a system 160, a display system 170, and a discharge unit 180. These components may be configured to be included within the housing of the content providing device 100.

In detail, a product information database 111 is stored in the memory 110, and any one or more of various applications, data, and commands for providing a content output environment may be stored in the product information database 111.

At this time, in the embodiment, the product information database 111 refers to a data set in which product information for each item sold through the content providing device 100 is pre-stored.

That is, the memory 110 may store commands and data that can be used to create a content output environment.

Additionally, the memory 110 may include a program area and a data area.

Here, the program area according to the embodiment may be linked between the operating system (OS: Operating System) and functional elements that boot the content providing device 100, and the data area may be used for use of the content providing device 100. Data generated accordingly can be stored.

Additionally, the memory 110 may include at least one non-transitory computer-readable storage medium and a temporary computer-readable storage medium.

For example, the memory 110 may be a variety of storage devices such as ROM, EPROM, flash drive, hard drive, etc., and web storage that performs the storage function of the memory 110 on the Internet. may include.

The processor assembly 120 may include at least one processor capable of executing instructions of the product information database 111 stored in the memory 110 to perform various tasks for providing a purchasing environment and content output environment. there is.

In an embodiment, the processor assembly 120 may control the overall operation of components through the product information database 111 of the memory 110 to provide customized content.

This processor assembly 120 may be a system-on-chip (SOC) suitable for the content providing device 100 that includes a central processing unit (CPU) and/or a graphics processing unit (GPU), and is installed in the memory 110. The stored operating system (OS) and/or application programs can be executed, and each component mounted on the content providing device 100 can be controlled.

Additionally, the processor assembly 120 may internally communicate with each component through a system bus and may include one or more bus structures, including a local bus.

In addition, the processor assembly 120 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and other electrical units for performing functions.

The communication processor 130 may include one or more devices for communicating with external devices. This communication processor 130 can communicate through a wireless network.

In detail, the communication processor 130 may communicate with a predetermined computing device that stores a content source for implementing a content output environment, and may communicate with various user input components such as a controller that receives user input.

In an embodiment, the communication processor 130 may transmit and receive various data related to customized content to another content providing device 100 and/or an external server.

In detail, in the embodiment, the communication processor 130 may receive content for the item selected by the user by communicating with other external devices and external servers using OTT (Over The Top, online video service).

This communication processor 130 uses technical standards or communication methods for mobile communication (e.g., Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5G New Radio (NR), WIFI). Alternatively, data can be transmitted and received wirelessly with at least one of a base station, an external content providing device 100, and an arbitrary server on a mobile communication network established through a communication device capable of performing short-distance communication.

The sensor system 160 may include various sensors, such as an image sensor 161 and an audio sensor 165.

Here, the image sensor 161 may capture images and/or videos of the physical space around the content providing device 100.

In an embodiment, the image sensor 161 may capture and obtain images related to customized content (e.g., a user's face recognition image, a front view image of the face, etc.).

In addition, the image sensor 161 is disposed on the front or/and rear of the content providing device 100 and can acquire an image by photographing the direction in which it is placed, and is disposed toward the outside of the content providing device 100. You can photograph physical space through a camera.

At this time, in the embodiment, the camera may be a camera implemented in a form capable of adjusting a predetermined angle so as to recognize the position of the user's face and adjust the angle.

Additionally, in an embodiment, the camera may further include a motor to adjust a predetermined angle.

This image sensor 161 may include an image sensor device and an image processing module. In detail, the image sensor 161 can process still images or moving images obtained by an image sensor device (eg, CMOS or CCD).

Additionally, the image sensor 161 may use an image processing module to process still images or moving images obtained through the image sensor device to extract necessary information and transmit the extracted information to the processor.

This image sensor 161 may be a camera assembly including at least one camera. The camera assembly may include a general camera that photographs a visible light band, and may further include a special camera such as an infrared camera or a stereo camera.

In addition, the image sensor 161 as described above may be included and operated in the content providing device 100, depending on the embodiment, or may be included in an external device (e.g., an external server, etc.) to operate as the above-described communication processor 130. It may also operate through linkage based on etc.

The audio sensor 165 can recognize sounds around the content providing device 100.

In detail, the audio sensor 165 may include a microphone capable of detecting a voice input from a user using the content providing device 100 and/or an audio signal detected from the surrounding environment of the content providing device 100. there is.

The payment unit 140 may perform all actions related to payment for the cost of purchasing and using a certain item selected by the user through the content providing device 100.

The payment unit 140 must be equipped with any vending machine. In the embodiment, the payment unit 140 may include a cash slot, a card slot, a card reader, and an input amount recognition unit, but performs actions related to payment. Any configuration can be added.

Input system 150 may detect a user's input (e.g., a gesture, voice command, actuation of a button, or other type of input) related to customized content.

In detail, the input system 150 may include a predetermined button, a touch sensor, and/or an image sensor 161 that receives a user motion input.

The input system 150 must be equipped with any vending machine. In the embodiment, the input system 150 may be implemented with a button, a touch sensor, and a touch screen, but it can be used to input an item selected by the user. Any configuration can be added.

The display system 170 can output various information related to customized content as graphic images.

In an embodiment, the display system 170 may display a user's face recognition image, a frontal view image of the face, shooting location feedback information, a feedback message, and item content using a display.

These displays include liquid crystal display (LCD), thin film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), and flexible display. , a 3D display, or an e-ink display.

The above components may be disposed within the housing of the content providing device 100, and the user interface may include a touch sensor 173 on the display 171 configured to receive user touch input.

In detail, the display system 170 may include a display 171 that outputs an image and a touch sensor 173 that detects a user's touch input.

For example, the display 171 may be implemented as a touch screen by forming a mutual layer structure or being integrated with the touch sensor 173. This touch screen can function as a user input unit that provides an input interface between the content providing device 100 and the user, and at the same time, can provide an output interface between the content providing device 100 and the user.

The discharge unit 180 may discharge a predetermined product that can be provided to a user using the content providing device 100 to the outside of the device.

The discharge unit 180 must be provided in any vending machine. In the embodiment, the discharge unit 180 can discharge items purchased by the user, change, receipts, etc., to the outside of the device and perform actions related to the discharge. It can be implemented in any configuration.

Meanwhile, the content providing device 100 according to an embodiment of the present invention can perform deep learning necessary for providing customized content by linking with a deep-learning neural network.

Here, the deep learning neural network according to the embodiment is a convolutional neural network (CNN), Regions with CNN features (R-CNN), Fast R-CNN, Faster R-CNN, and/or Mask R- It may include CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing embodiments described later, and in embodiments of the present invention, such deep learning neural network itself is limited or restricted. I don't.

In detail, in the embodiment, the content providing device 100 may perform deep learning necessary for providing customized content using the Pose Estimation Model and Gaze Estimation Model based on the deep learning neural network described above.

At this time, the Pose Estimation Model is a model designed using the papers Whole-Body Human Pose Estimation in the Wild and RMPE: Regional Multi-Person Pose Estimation.

Additionally, the Gaze Estimation Model is a model designed using the papers Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings and Gaze360: Physically Unconstrained Gaze Estimation in the Wild.

Using this deep learning technology, the content providing device 100 according to the embodiment can automatically adjust the user's initial camera position in real time.

In addition, using such deep learning technology, the content providing device 100 according to the embodiment can determine the user who is the target of gaze tracking in real time and automatically track the gaze of the user.

Figure 2 is an example of a diagram for explaining a device 100 that provides customized content based on gaze recognition according to an embodiment of the present invention.

Referring to FIG. 2 , the content providing device 100 may include a payment unit 140, an image sensor 161, a display 171, and a discharge unit 180 on one side.

The image sensor 161 may be included on the same side as the side where the items and item images to be sold to the user 1 are displayed in order to recognize the gaze of the user 1.

Additionally, the display 171 may include a content display 171-1 and an item display 171-2.

In the embodiment, the content display 171-1 refers to a display that recognizes the gaze of the user 1 and outputs content including product information about the item that the user 1 gazes at.

Additionally, in the embodiment, the item display 171-2 refers to a display on which an image representing an item to be sold to the user 1 is output. At this time, a predetermined button may be located within the item display 171-2 area, or a touch sensor 173 may be built into part or all of the area.

That is, according to this, the content providing device 100 can provide a user interface capable of receiving input from the user 1 based on the item display 171-2.

Additionally, the payment unit 140 and the discharge unit 180 may be located anywhere on the content providing device 100 as long as it is convenient for the user 1 to use, and are not limited to what is shown.

- How the content providing device 100 provides customized content based on gaze recognition

Hereinafter, a method of providing customized content based on gaze recognition by the content providing device 100 executed by the processor assembly 120 including at least one processor according to an embodiment of the present invention will be described in FIGS. 3 to 7. Please refer to and explain in detail.

Figure 3 is a flowchart illustrating a method of providing customized content based on gaze recognition according to an embodiment of the present invention.

Referring to FIG. 3, the content providing device 100 can recognize the approach of the user 1. (S101)

In detail, in the embodiment, the content providing device 100 may recognize the approach of the user 1 using the image sensor 161 attached to the device 100.

At this time, in the embodiment, the content providing device 100 may change the state of the user 1 to comply with preset normal access standard information.

Here, the normal access standard information according to the embodiment may include initial location information and initial appearance information required for the user 1 who wants to use the content providing device 100.

At this time, the initial location information may be information that guides the placement status of the user 1 (eg, location and angle, etc.). Additionally, the initial appearance information may be information that guides the appearance state of the user 1 (for example, appearance for eye and face recognition).

In detail, the content providing device 100 may construct predetermined initial location information and initial appearance information according to the settings of the device manager and a preset process.

In an embodiment, the initial location information may be preset to guide a predetermined location where the face of the user 1 can be photographed. Additionally, the initial appearance information may be preset to guide the user 1 to a predetermined appearance state (e.g., not wearing a hat or sunglasses, etc.) that the user 1 is required to maintain while using the content providing device 100.

Additionally, in the embodiment, the content providing device 100 may generate access feedback information based on the preset normal access standard information as described above, output the access feedback information through the display 171, and provide it to the user 1.

Here, the approach feedback information refers to adjusting/adjusting the status of the user 1 (in the embodiment, position, angle, appearance, etc.) to obtain a face recognition image at a location that satisfies the shooting environment conditions when photographing the user 1. It may refer to information guiding change.

At this time, the shooting environment conditions are conditions for easily estimating the facial area of the user 1 using the content providing device 100, and in the embodiment, conditions are conditions for easily obtaining the facial key points and gaze information of the user 1. This may be possible, and a detailed explanation of this will be provided later.

For example, referring again to FIG. 2, the content providing device 100 may display commands such as “Locate within the area” and “Look at the image sensor 161” related to shooting environment conditions. and “Please take off your hat.” The position, angle, and appearance of the user 1 can be adjusted by outputting approach feedback information such as text through the display 171.

In this way, in the embodiment, the content providing device 100 can photograph and obtain the facial recognition image 10 of the user 1 using the image sensor 161.

At this time, in the embodiment, the face recognition image 10 refers to an image that recognizes the user's face when the user approaches the content providing device 100 at a predetermined distance in order to determine the user's face area.

Next, in the embodiment, the content providing device 100 may obtain a front view image of the face by adjusting the camera angle. (S103)

In detail, in the embodiment, the content providing device 100 determines the user's face area through a deep learning model based on the face recognition image 10 and adjusts the position and angle of the camera accordingly using a motor included in the camera. By adjusting, a frontal view image of the face can be obtained.

At this time, the content providing device 100 may detect the facial keypoint of the user 1 using the Pose Estimation Model among deep learning models.

FIG. 4 is an example of a diagram illustrating a method for estimating a facial key point 30 according to an embodiment of the present invention.

Referring to FIG. 4, the content providing device 100 performs deep learning analysis based on the acquired facial recognition image 10 of the user 1 to detect the facial key point 30 in the facial recognition image 10. can do.

Here, the facial key point 30 according to the embodiment may include at least one key point and a sub key point 31 that can specify the facial shape of the user 1.

In an embodiment, the face keypoint 30 includes an eye keypoint that specifies the eye shape of the applicant, a nose keypoint that specifies the nose shape, a mouth keypoint that specifies the mouth shape, and/or an ear keypoint that specifies the ear shape, etc. And, at least one key point described above may be connected to a sub key point 31.

That is, in the embodiment, the content providing device 100 may perform detection deep learning to extract facial keypoints of the user 1 from the facial recognition image 10 in conjunction with a predetermined deep learning neural network.

In addition, the content providing device 100 provides the face information including the direction and angle of the face of the user 1 based on the plurality of facial key points of the extracted user 1 as output data of the deep learning neural network. Pose information can be obtained.

That is, the shooting environment conditions described above in step S101 include conditions including all facial key points 30 that specify the facial shape of the user 1, and pixel areas based on the facial key points 30 set to a predetermined standard (e.g. , a preset pixel area or more, a preset ratio to the total pixel area of the face recognition image 10, etc.) may be included.

Additionally, in the embodiment, the content providing device 100 may obtain facial pose information, which is information including the direction and angle of the face of the user 1, based on the detected facial keypoint of the user 1.

Additionally, in the embodiment, the content providing device 100 may control the image sensor based on the acquired facial pose information of the user 1.

In order to perform image sensor control, the content providing device 100 may set standard frontal facial area information based on a predetermined facial key point.

In an embodiment, the content providing device 100 may determine a clustering degree between predetermined facial keypoints (e.g., a standard interval of a predetermined length (e.g., 0.1 cm interval and ±5% error range, etc.), and a predetermined area (e.g., , standard area of 3 square centimeters (cm ² ) and error range of ±5%, etc.), standard frontal facial area information including the number and/or presence or absence of certain facial key points, etc. can be preset.

That is, the content providing device 100 may compare the acquired facial pose information of the user 1 and preset standard frontal facial area information and obtain the facial pose information according to the error value.

Additionally, the content providing device 100 may obtain a front view image of the face including the front view of the face of the user 1 by controlling the above-described image sensor based on the acquired facial pose information.

In addition, according to the conditions of the shooting environment, in the embodiment, the content providing device 100 may select a plurality of facial areas (FA) of the user 1 recognized in the face recognition image 10. ), a facial area (FA) smaller than a predetermined size (e.g., 3 square centimeters (cm ² )) may be excluded from the judgment.

For example, when there are two or more users who want to use the content providing device 100 and the face area of the user who is not in their turn to use is recognized, the face area of the user who is not in their turn to use is excluded from the judgment, so that the face area of the user who is not in their turn to use is recognized. Only the user's facial area (FA) can be recognized.

Additionally, the content providing device 100 can determine the facial area (FA) of the user 1 in real time and automatically, and change the camera angle in real time and automatically accordingly.

In other words, the content providing device 100 changes the viewing angle of the camera using a motor included in the camera to normally recognize the determined facial area (FA) of the user 1 in real time and automatically. You can adjust the camera angle to suit.

Additionally, in the embodiment, the content providing device 100 may obtain a front view image of the face of the user 1 by adjusting the angle of the camera.

At this time, the front view image 20 of the face is used to obtain gaze information 40 of the user 1, and the angle of the camera is such that the user's eyes, nose, and mouth are clearly visible based on the face recognition image 10. It may refer to an image including the front view of the face of the user 1 obtained by adjusting .

Next, in the embodiment, the content providing device 100 may generate gaze information 40 by tracking the gaze of the user 1. (S105)

In detail, in the embodiment, the content providing device 100 tracks the gaze of the user 1 based on the eye key point 30-E included in the face area (FA) of the user 1 and provides gaze information 40. can be created.

Here, the gaze information 40 is information that estimates the gaze position, gaze direction, and gaze fixation time (e.g., seconds) of the user 1, and may include various information related to the gaze of the user 1. You can.

That is, in the embodiment, the content providing device 100 generates the gaze information 40 of the user 1 by tracking the gaze direction of the user 1 based on the front view image 20 of the user 1's face. can do.

Figure 5 is an example of a drawing for explaining gaze information 40 for tracking gaze direction according to an embodiment of the present invention.

Referring to FIG. 5 , in the embodiment, the content providing device 100 may track the gaze direction of the user 1 based on the eye keypoint 30-E included in the front view image 20 of the face.

In detail, in the embodiment, the content providing device 100 selects the eye keypoint 30-E, which provides gaze information of the user 1, from among the plurality of facial keypoints 30 included in the frontal view image 20 of the face. As a basis, the gaze direction of the user (1) can be tracked.

At this time, the face front view image 10 includes the user 1 while making a purchase decision while using the content providing device 100, and the eye key point 30-E is a predetermined deep view in step S103. It can be detected by estimating the position of the eyes in the face area (FA) through a learning process (in the example, using the Pose Estimation Model).

Additionally, in the embodiment, the content providing device 100 may generate gaze information 40, which is information that continuously tracks the gaze of the user 1 through the detected eye keypoint 30-E.

Here, in the embodiment, the content providing device 100 estimates the gaze position and direction of the user 1 through a predetermined deep learning learning process (in the embodiment, using a Gaze Estimation Model) and provides gaze information 40. can be created.

At this time, in the embodiment, if the gaze fixation time of the user 1 included in the generated gaze information 40 is longer than a predetermined time (for example, 0.5 seconds or longer, etc.), the content providing device 100 fixes the gaze. You can set the time as the time of interest.

Here, in the embodiment, the content providing device 100 may consider and set only the longest time among the gaze fixation times set as the time of interest as the time of interest.

Additionally, in the embodiment, the content providing device 100 may determine the area of interest based on the generated gaze information 40. (S107)

In detail, in the embodiment, the content providing device 100 may determine the area of interest based on the time of interest included in the generated gaze information 40.

Figure 6 is an example of a diagram for explaining a method of setting a region of interest based on gaze information 40 according to an embodiment of the present invention.

Referring to FIG. 6, the content providing device 100 provides gaze information ( The area of interest can be determined based on the interest time included in 40).

In other words, in the embodiment, the area of interest is an area where the gaze of the user 1 stays for more than a predetermined time, and can be determined based on the time of interest.

In detail, for convenience of explanation, FIG. 6 may show gaze information 40 from the perspective of the user 1 rather than from the perspective of the image sensor attached to the content providing device 100. In other words, the gaze information 40 shown in FIG. 6 can be regarded as the user's gaze.

At this time, (a) and (b) included in the gaze information 40 may be areas of interest determined based on the time of interest. The sizes of (a) and (b) are proportional to the time of interest, and as described above, in the embodiment, the content providing device 100 may determine the area where (b), where a larger time of interest is set, is formed as the area of interest. .

Additionally, the region of interest determined here may be located on at least one item image (IMG) output on the item display 171-2 of the content providing device 100.

Additionally, when there is at least one user 1, the content providing device 100 may generate the gaze information 40 in a number proportional to the number of users 1.

Returning again, in the embodiment, the content providing device 100 may derive a pixel area (PA) that has a high matching rate with the determined area of interest. (S109)

In detail, in the embodiment, the content providing device 100 determines that the matching rate is the highest among the plurality of pixel areas (PA) by comparing the matching rate with the determined area of interest and the pixel area (PA) included in the item display 171-2. A high first pixel area (PA-1) can be derived.

Here, the pixel area (PA) refers to a predetermined area where each item is located in the item display 171-2 where a plurality of items are displayed or an item image (IMG) is exposed.

That is, for this purpose, in the embodiment, the content providing device 100 may preset at least one pixel area (PA) by individually matching the item and the item image (IMG) for each pixel area (PA).

In addition, the item image (IMG) refers to a representative image for an item that can be sold to a user using the content providing device 100, and at least one image is displayed on the item display 171-2. It can be.

Accordingly, the pixel area (PA) and item image (IMG) may exist in numbers corresponding to the number of items provided by the content providing device 100.

In addition, in the embodiment, the content providing device 100 divides the item display 171-2 by a preset ratio (e.g., 9 divisions) and sets each divided area as a pixel area (PA), and each item image ( The pixel area (PA) can be preset using a method such as setting a predetermined area (for example, a rectangular area containing the item image (IMG)) adjacent to the IMG as the pixel area (PA).

In an embodiment of the present invention, the content providing device 100 is shown and described in the drawing based on a method of presetting the pixel area (PA) using the latter method. However, the method of presetting the pixel area (PA) is It is not limited to this and various embodiments may exist.

In the embodiment, the content providing device 100 compares the matching rate of the plurality of pixel areas (PA) included in the item display 171-2 and the determined area of interest of the user 1, and selects the first pixel with the highest matching rate. The area (PA-1) can be derived.

Referring again to FIG. 6 , in the embodiment, the content providing device 100 may preset a number of pixel areas (PA) corresponding to the number of items being sold (for example, 9 as of FIG. 6 ). At this time, each pixel area (PA) may include one item and an item image (IMG).

Additionally, in the embodiment, the content providing device 100 may derive the first pixel area (for example, PA-1 in FIG. 6) with the highest matching rate with (b), which is the user's area of interest.

At this time, in the embodiment, the content providing device 100 may derive the first pixel area (PA-1) that has the highest matching rate with the region of interest (e.g., matching rate of 80% or more) among the plurality of pixel areas (PA). .

Next, in the embodiment, the content providing device 100 may detect an item matching the derived pixel area (PA). (S111)

To this end, in the embodiment, the content providing device 100 may detect the item matched for each pixel area (PA) by determining that it is the item that the user 1 is gazing at.

That is, in the embodiment, the content providing device 100 derives the first pixel area (PA-1) that has the highest matching rate with the region of interest among the plurality of pixel areas (PA), and the derived first pixel area (PA-1) Items matching 1) can be detected.

At this time, the detected item may include product information, which is advertising and additional information for the item. To this end, in the embodiment, the content providing device 100 individually matches product information included in each item sold to the user 1 using the content providing device 100 to each item to create a product information database 111. ) can be pre-stored.

Meanwhile, even when a plurality of item pages exist due to a limit to the number of item images (IMG) that can be exposed at one time on the item display 171-2, the image matching the predetermined pixel area (PA) in the corresponding item page Items can be detected.

To this end, in the embodiment, the content providing device 100 may detect the user 1's item page change input using the input system 150.

Additionally, in the embodiment, the content providing device 100 may change the item and item image (IMG) matched to a predetermined pixel area (PA) in the corresponding item page according to the user's (1) item page change input. .

Accordingly, the content providing device 100 can detect an item matching the pixel area (PA) derived on the item page changed by the input of the user 1.

For example, referring to FIG. 6 again, when the first item page is changed to the second item page by detecting the input of the user 1, the content providing device 100 matches the first pixel area (PA-1). The first item can be changed to the second item.

At this time, the item image (IMG) exposed for each pixel area (PA) of a plurality of item pages changes and the detected item is also different accordingly, so for this purpose, the content providing device 100 stores the item in the preset pixel area (PA). May contain page information.

That is, even if the content providing device 100 derives the same pixel area (PA) by the user 1 staring at the same item display 171-2 area, the content may be different depending on the item page information included in the pixel area (PA). Items can be detected.

Subsequently, in the embodiment, the content providing device 100 may output content for the detected item. (S113)

In detail, in the embodiment, the content providing device 100 may output product information about the item detected based on the product information database 111 as content to the content display 171-1.

In addition, when product information for the detected item does not exist in the product information database 111, in the embodiment, the content providing device 100 provides advertisements and Content including additional information may be acquired and output on the content display 171-1.

According to the embodiment, the content providing device 100 detects an item matching the pixel area from the product information database 111, and sends product information for the detected item to a disclosed OTT service (Over-the-top). It can also be obtained and provided through the Internet based on media service.

In summary, the content providing device 100 determines and detects the item that the user 1 is looking at among the plurality of items exposed to the item display 171-2 and displays advertisements and additional information for the item on the content display ( Content can be provided to the user 1 by printing it on 171-1).

Additionally, the content providing device 100 may provide content to the user 1 by detecting the user 1's item selection input using the input system 150 as well as the user's 1 gaze information.

To this end, the content providing device 100 detects the user 1's item selection input to select a pixel area (PA) containing at least one item among the plurality of items exposed on the item display 171-2. can do.

According to the input, the content providing device 100 may derive the pixel area (PA) selected by the user 1, detect an item matching the derived pixel area (PA), and provide information about the detected item. Product information content can be printed.

Additionally, in the embodiment, the content providing device 100 manages (in the embodiment, plays) content exposed to the content display 171-1 based on the gaze information 40 of the user 1 generated in step S105. , change, stop, and terminate).

For example, 1) When the gaze information 40 of the user 1 is not collected on the item display 171-2, in the embodiment, the content providing device 100 allows the user 1 to display the content 171-1. By determining that the user is continuously looking at the content being played, the content being played can be maintained on the content display 171-1.

In addition, for example, 2) when gaze information 40 of the user 1 is collected on the item display 171-2 and the area of interest included in the gaze information 40 is changed, in the embodiment, the content providing device 100 ) can stop the content being played, return to step S109, and change the content to the content for the item that the user 1 is newly looking at.

In addition, for example, 3) when the gaze information 40 of the user 1 is not collected on the item display 171-2 and the face recognition image 10 acquired using the image sensor 161 does not exist, In an embodiment, the content providing device 100 may determine that the user 1 does not wish to purchase the item and has stopped using the content providing device 100, and may end the content being played.

In the case of 3) described above, a general advertisement can be output on the content display 171-1 targeting general passers-by rather than the user 1, thereby increasing the use efficiency of the content providing device 100. There is a possible effect.

Meanwhile, the content providing device 100 according to another embodiment may provide customized content based on gaze recognition to each of a plurality of users even when a plurality of users use the content providing device 100.

In other embodiments below, only content that is different from the above-described embodiment will be described, and the method of the above-described embodiment will be considered to apply to content not described.

Figure 7 is a flowchart illustrating a method of providing customized content based on gaze recognition according to another embodiment of the present invention.

Referring to FIG. 7 , in another embodiment, the content providing device 100 may recognize a plurality of user facial areas (FA) based on the facial recognition image 10. (S301)

In detail, in another embodiment, the content providing device 100 has a plurality of face areas (FA) of the user 1 recognized in the face recognition image 10, and all of the recognized face areas (FA) have a predetermined size (e.g. , 3 square centimeters (cm2) or more, multiple user face areas (FA) can be recognized.

For convenience of explanation, the description is based on the case where there are two users (1), but the present invention is not limited to this and various embodiments in which there are fewer or more users (1) may be possible.

For example, if there are two users (1) who want to use the content providing device 100 and both recognized face areas (FA) are larger than a predetermined size, the content providing device 100 It can be determined that there are multiple users (1) trying to use the device.

Subsequently, in another embodiment, the content providing device 100 may adjust the camera angle according to the recognized facial area (FA). (S303)

In detail, in another embodiment, since there are a plurality of recognized face areas (FA), the content providing device 100 may adjust the camera angle by deriving an angle at which all of the plurality of face areas (FA) are recognized normally.

For example, the content providing device 100 may adjust the camera angle so that the camera center point is located at the exact center of the two recognized face areas (FA).

Additionally, in another embodiment, the content providing device 100 may generate first gaze information and second gaze information by tracking the gaze of a plurality of users. (S305)

At this time, in another embodiment, the content providing device 100 may generate first gaze information, which is gaze information that tracks the gaze of the first user, and second gaze information, which is gaze information that tracks the gaze of the second user. .

Additionally, in another embodiment, the content providing device 100 may determine a first region of interest and a second region of interest based on a plurality of generated gaze information. (S307)

At this time, in another embodiment, the content providing device 100 may determine the first area of interest from the generated first gaze information and determine the second area of interest from the second gaze information.

That is, in another embodiment, the content providing device 100 may generate a number of gaze information and areas of interest proportional to the number of users.

Additionally, in another embodiment, the content providing device 100 may derive a first pixel area and a second pixel area that have a high matching rate with the plurality of determined regions of interest. (S309)

At this time, in another embodiment, if the derived first pixel area and the second pixel area are the same, the content providing device 100 determines that the item that the plurality of users are looking at is the same and displays the content in the same manner as S111 and S113. Only one content can be output at (171-1).

In other words, in another embodiment, when the derived first pixel area and the second pixel area are different, the content providing device 100 determines that the items that the plurality of users are looking at are different and displays the plurality of contents in the content display 171- Step S311 can be performed to output in 1).

Additionally, in another embodiment, the content providing device 100 may detect a first item and a second item that match the derived plurality of pixel areas. (S311)

Subsequently, in another embodiment, the content providing device 100 may split the output screen on the content display 171-1. (S313)

At this time, the output screen of the content display 171-1 may be divided into numbers corresponding to the number of recognized users.

Additionally, in another embodiment, the content providing device 100 may output first content and second content respectively on divided output screens. (S315)

In detail, in another embodiment, the content providing device 100 may output the first content and the second content on the output screen of the divided content display 171-1 located close to each user.

Figure 8 is an example showing a plurality of customized content based on gaze recognition being output according to another embodiment of the present invention.

Referring to FIG. 8, in another embodiment, the content providing device 100 tracks the gaze of the first user 1-A and the second user 1-B and includes them in the item display 171-2. The first pixel area (PA-A) and the second pixel area (PA-B) can be derived.

At this time, in another embodiment, the content providing device 100 provides the first user (1-A) and the second user regardless of the positions of the first pixel area (PA-A) and the second pixel area (PA-B). Based on the location of (1-B), content can be output on a split output screen close to each user.

For example, the content providing device 100 may display content about an item corresponding to the first pixel area (PA-A) that the first user (1-A) is looking at on the A' split output screen close to the first user. Contents for items corresponding to the second pixel area (PA-B) that the second user (1-B) is gazing at can be output on the split output screen B' close to the second user.

Additionally, in another embodiment, the content providing device 100 may designate a split output screen in correspondence with the location of the user face area (FA) recognized in step S301.

As described above, the device and method for providing customized content based on gaze recognition according to an embodiment of the present invention automatically adjusts the viewing angle of the camera attached to the device in real time according to the user's physical characteristics, so that each time the user changes, the device and method provide an initial value tailored to the user. It has the effect of shortening the time required to perform settings.

In addition, the device and method for providing customized content based on gaze recognition according to an embodiment of the present invention intuitively identifies the product of interest to the user by outputting content about the product the user is looking at, thereby improving the user's satisfaction according to content exposure. It has an effect.

In addition, the device and method for providing customized content based on gaze recognition according to an embodiment of the present invention divides the content output screen according to the number of users, thereby efficiently using the content output screen to deliver advertisements and information to each user. It has the effect of increasing .

The embodiments according to the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. A hardware device can be converted into one or more software modules to perform processing according to the invention, and vice versa.

The specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, those skilled in the art or those skilled in the art will understand the spirit of the present invention as described in the patent claims to be described later. It will be understood that the present invention can be modified and changed in various ways without departing from the technical scope. Therefore, the technical scope of the present invention should not be limited to what is described in the detailed description of the specification, but should be defined by the scope of the claims.

Claims (10)

In a method of providing customized content based on gaze recognition executed by at least one processor of a content providing device,
Obtaining a face recognition image including a facial area of at least one user based on an image sensor;
controlling the image sensor based on the acquired facial recognition image to obtain a front view image of the face of the at least one user;
Generating at least one piece of gaze information including gaze direction and gaze fixation time information of the at least one user based on the acquired front view image of the face;
determining a display area corresponding to the generated gaze information as a region of interest;
detecting an item matching the determined area of interest; and
Comprising the step of providing product information content for the detected item,
The step of providing product information content for the detected item is,
dividing the display area into an item display area and a content display area;
determining an area of interest corresponding to at least one gaze information generated in the item display area;
If the determined areas of interest are two or more, dividing the content display area to correspond to the number of areas of interest;
Among the divided content display areas, the first product information content for the first item corresponding to the first area of interest according to the first gaze information of the first user is output in the first content display area, and in the second content display area. Including the step of outputting second product information content for the second item corresponding to the second area of interest according to the second gaze information of the second user, respectively.
A method of providing customized content based on gaze recognition. According to claim 1,
The step of controlling the image sensor based on the acquired facial recognition image to obtain a front view image of the face of the at least one user,
Providing the face recognition image as input data of a pre-trained deep learning neural network, obtaining facial pose information of the at least one user as output data of the deep learning neural network, and the acquired face Comprising controlling the image sensor based on pose information.
A method of providing customized content based on gaze recognition. According to claim 1,
The step of determining the display area corresponding to the generated gaze information as the area of interest is,
When the user's gaze fixation time information meets a preset time or more, determining a first item area included in the item display area corresponding to the user's gaze direction as the area of interest.
A method of providing customized content based on gaze recognition. According to claim 1,
The step of detecting an item matching the determined area of interest is,
Obtaining a user input for selecting an item page to be output based on the item display area;
Comprising the step of detecting an item matching the area of interest within the selected item page.
A method of providing customized content based on gaze recognition. According to claim 1,
The step of determining the display area corresponding to the gaze information as the area of interest is,
When first gaze information is changed to second gaze information, updating the region of interest with at least one of the item display area and the content display area corresponding to the changed second gaze information,
The step of updating the region of interest is,
If the updated area of interest is an item display area, changing the first product information content matched to the first gaze information played in the content display area into a second product information content matched to the second gaze information; ,
If the updated area of interest is a content display area, maintaining playback of the first product information content played in the content display area.
A method of providing customized content based on gaze recognition. According to claim 1,
The step of detecting an item matching the determined area of interest is,
detecting at least one pixel in a display area matching the region of interest;
Comprising the step of detecting a pre-stored item for the detected at least one pixel.
A method of providing customized content based on gaze recognition. According to claim 1,
Obtaining a user input for selecting at least one pixel in the item display area, detecting a pre-stored item for the selected at least one pixel, and displaying product information content for the detected item Further comprising the step of outputting to the area
A method of providing customized content based on gaze recognition. At least one image sensor that acquires a face recognition image including a facial area of at least one user;
At least one display that outputs product information content based on the acquired facial recognition image;
At least one processor;
The processor,
Controlling the image sensor based on the acquired facial recognition image to obtain a front view image of the face of the at least one user,
Generating at least one piece of gaze information including gaze direction and gaze fixation time information of the at least one user based on the obtained front view image of the face,
Determine the display area corresponding to the generated gaze information as the area of interest,
Detect an item matching the determined area of interest,
Provide product information content for the detected item,
Provision of product information content for the detected item,
Dividing the display area into an item display area and a content display area,
Determine an area of interest corresponding to at least one gaze information generated in the item display area,
If the determined areas of interest are two or more, dividing the content display area to correspond to the number of areas of interest,
Among the divided content display areas, the first product information content for the first item corresponding to the first area of interest according to the first gaze information of the first user is output in the first content display area, and in the second content display area. Including outputting second product information content for the second item corresponding to the second area of interest according to the second gaze information of the second user, respectively.
A device that provides customized content based on gaze recognition. According to clause 8,
The display is,
Comprising an item display that outputs a representative image for at least one of the items, and a content display that outputs the product information content.
A device that provides customized content based on gaze recognition. According to clause 8,
The processor,
Obtaining facial pose information of the user based on deep learning based on the facial recognition image,
Obtaining a front view image of the user's face by controlling the image sensor based on the acquired facial pose information
A device that provides customized content based on gaze recognition.