KR102862950B1 - Electronic device for providing augmented reality service and operating method thereof - Google Patents

Abstract

Various embodiments of the present invention relate to a device and method for providing an augmented reality service in an electronic device by linking with another electronic device. A first electronic device includes a communication circuit, a camera module, a sensor module, a display device, and a processor operatively connected to the communication circuit, the display device, the sensor module, and the camera module, wherein the processor communicates with a second electronic device through the communication circuit, obtains at least one object information included in an area corresponding to a field of view of a user wearing the first electronic device through at least one of the camera module and the sensor module, obtains gaze information of the user, determines whether an augmented reality service is linked with the second electronic device based on the gaze information of the user and the at least one object information, and, if it is determined that the augmented reality service is linked with the second electronic device, transmits the linkage information of the augmented reality service to the second electronic device through the communication circuit. Other embodiments may also be possible.

Description

Electronic device for providing augmented reality service and operating method thereof {ELECTRONIC DEVICE FOR PROVIDING AUGMENTED REALITY SERVICE AND OPERATING METHOD THEREOF}

Various embodiments of the present invention relate to a device and method for providing augmented reality services in an electronic device.

Advances in information and communication technology and semiconductor technology are evolving various electronic devices into multimedia devices that provide a variety of multimedia services. Multimedia services may include at least one of the following: voice call services, messaging services, broadcasting services, wireless Internet services, camera services, electronic payment services, or music playback services.

Electronic devices can provide diverse experiences to users. For example, electronic devices can provide augmented reality (AR) services that superimpose virtual information (e.g., virtual objects) onto real-world spaces.

AR devices can be provided by mobile devices, projectors, or wearable devices. Electronic devices providing AR services can be configured in various forms to enable users to experience realistic AR services. For example, an electronic device providing AR services can be configured in the form of glasses and worn on the user's face.

An electronic device (e.g., a wearable device) that provides AR services can provide AR services independently from an electronic device (e.g., a mobile device) such as a smartphone or mobile terminal. For example, when an application (e.g., an AR application) installed on the electronic device is running, the electronic device that provides AR services can display virtual information (e.g., a virtual object) by overlaying it on at least a portion of the real space corresponding to the user's field of view.

Electronic devices such as smartphones and mobile terminals have become popular and can be used by a relatively large number of users. Accordingly, electronic devices that provide AR services require methods to link with electronic devices such as smartphones and mobile terminals to provide users with a wider variety of AR services.

Various embodiments of the present invention disclose devices and methods for providing augmented reality services in conjunction with another electronic device (e.g., an electronic device providing augmented reality services) on an electronic device (e.g., a mobile device).

According to various embodiments, a first electronic device includes a communication circuit, a camera module, a sensor module, a display device, and a processor operatively connected to the communication circuit, the display device, the sensor module, and the camera module, wherein the processor communicates with a second electronic device through the communication circuit, obtains at least one object information included in an area corresponding to a field of view of a user wearing the first electronic device through at least one of the camera module and the sensor module, obtains gaze information of the user, determines whether to link an augmented reality service with the second electronic device based on the gaze information of the user and the at least one object information, and when it is determined to link the augmented reality service with the second electronic device, transmits linkage information of the augmented reality service to the second electronic device through the communication circuit.

According to various embodiments, a second electronic device includes a communication circuit, an input device, and a processor operatively connected to the communication circuit and the input device, wherein the processor is configured to communicate with the first electronic device through the communication circuit, and, when an input is detected through the input device, determine an input mode of the second electronic device, and, when the input mode is the first mode, transmit the input information to the first electronic device through the communication circuit, and, when the input mode is the second mode, perform an operation corresponding to the input information.

According to various embodiments, a second electronic device includes a communication circuit, a display device, and a processor operatively connected to the communication circuit and the display device, wherein the processor receives, from the first electronic device through the communication circuit, at least one piece of object information included in an area corresponding to a field of view of a user wearing the first electronic device, and determines whether an object related to the second electronic device exists in the area corresponding to the field of view of the user wearing the first electronic device based on the at least one piece of object information, and when an object related to the second electronic device exists in the area corresponding to the field of view of the user wearing the first electronic device, determines whether the user's gaze gazes at the object related to the second electronic device, and when the user's gaze gazes at another object different from the object related to the second electronic device, determines that the first electronic device and an augmented reality service are linked.

According to various embodiments of the present invention, a first electronic device (e.g., an electronic device providing an augmented reality service) can selectively link with a second electronic device (e.g., a mobile device) based on at least one of a user's line of sight related to the augmented reality service, object information included in an area corresponding to the user's field of view, or location information of the first electronic device, thereby providing various control (or operation) methods corresponding to the augmented reality service.

According to various embodiments, a second electronic device (e.g., a mobile device) selectively provides input information acquired by the second electronic device to the first electronic device (e.g., an electronic device providing an augmented reality service) based on an input mode corresponding to whether or not the second electronic device is linked with an augmented reality service, thereby allowing a user to easily control (or operate) the augmented reality service.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
FIG. 2 is a configuration of electronic devices linked for augmented reality services according to various embodiments.
FIG. 3 is a block diagram of a first electronic device for providing augmented reality services according to various embodiments.
FIG. 4 is a block diagram of a second electronic device for providing augmented reality services according to various embodiments.
FIG. 5 is a flowchart for interfacing a first electronic device with a second electronic device according to various embodiments.
FIGS. 6A and 6D are screen configurations related to an augmented reality service displayed on a first electronic device according to various embodiments.
FIG. 7 is a flowchart for linking a first electronic device with a second electronic device based on gaze information related to an object according to various embodiments.
FIGS. 8A to 8D are screen configurations of a first electronic device for interfacing with a second electronic device based on gaze information related to an object according to various embodiments.
FIG. 9 is a flowchart for linking a first electronic device with a second electronic device based on gaze information related to an object associated with the second electronic device according to various embodiments.
FIGS. 10A to 10D illustrate screen configurations of a first electronic device for interfacing with a second electronic device based on user gaze information according to various embodiments.
FIGS. 11A to 11C are screen configurations of a first electronic device for interfacing with a second electronic device based on gaze information related to an object associated with the second electronic device according to various embodiments.
FIG. 12 is a flowchart for interfacing with a second electronic device based on FOV information in a first electronic device according to various embodiments.
FIG. 13 is a screen configuration of a first electronic device for interfacing with a second electronic device based on FOV information of the first electronic device according to various embodiments.
FIG. 14 is a flowchart for linking a first electronic device with a second electronic device based on location information according to various embodiments.
FIG. 15 is a flowchart for processing input information provided from a second electronic device in a first electronic device according to various embodiments.
FIGS. 16A to 16C are screen configurations of a first electronic device for displaying a functional UI based on input information provided from a second electronic device according to various embodiments.
FIGS. 17A to 17C are screen configurations of a first electronic device for displaying an operation UI based on input information provided from a second electronic device according to various embodiments.
FIGS. 18A and 18B are screen configurations of a first electronic device for controlling video content based on input information provided from a second electronic device according to various embodiments.
FIGS. 19A and 19B are screen configurations of a first electronic device for providing guide information according to various embodiments.
FIGS. 20A and 20B are screen configurations of a first electronic device for displaying a functional UI based on the shape of a second electronic device according to various embodiments.
FIG. 21 is a flowchart for interfacing a second electronic device with a first electronic device according to various embodiments.
FIG. 22 is a flowchart for interfacing with a first electronic device based on control of the first electronic device in a second electronic device according to various embodiments.
FIG. 23 is an example for changing a display method based on linkage with a first electronic device in a second electronic device according to various embodiments.
FIG. 24 is an example of changing content displayed on a display device based on linkage with a first electronic device in a second electronic device according to various embodiments.
FIG. 25 is a flowchart for determining linkage with a first electronic device in a second electronic device according to various embodiments.
FIG. 26 is a flowchart for providing IoT services in a first electronic device according to various embodiments.
FIG. 27a and FIG. 27b are screen configurations of a first electronic device for registering an IoT device according to various embodiments.
FIG. 28 is a flowchart for providing an IoT service based on a first electronic device in a second electronic device according to various embodiments.

Various embodiments are described in detail below with reference to the accompanying drawings. The embodiments and terms used herein are not intended to limit the technology described in this document to specific embodiments, but rather to encompass various modifications, equivalents, and/or alternatives of the embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar components. Singular expressions may include plural expressions unless the context clearly dictates otherwise.

FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the display device (160) or the camera module (180)), or may have one or more other components added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., a fingerprint sensor, an iris sensor, or an ambient light sensor) may be implemented embedded in a display device (160) (e.g., a display).

The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may load a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) into a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) and a secondary processor (123) (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith. Additionally or alternatively, the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part of it.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display device (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)).

The memory (130) can store various data used by at least one component (e.g., a processor (120) or a sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., a program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134).

The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input device (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input device (150) can include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

The audio output device (155) can output audio signals to the outside of the electronic device (101). The audio output device (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback, and the receiver may be used to receive incoming calls. In one embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

A display device (160) can visually provide information to an external party (e.g., a user) of an electronic device (101). The display device (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display device (160) may include touch circuitry configured to detect a touch, or a sensor circuit (e.g., a pressure sensor) configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through an input device (150), or output sound through an audio output device (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or headphone)) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information stored in the subscriber identification module (196) (e.g., an international mobile subscriber identity (IMSI)) to identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include one antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., an RFIC) may be additionally formed as a part of the antenna module (197).

At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, or client-server computing technologies may be utilized, for example.

Electronic devices according to various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the aforementioned devices.

The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or portion of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^™ ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

FIG. 2 is a configuration of electronic devices linked for augmented reality services according to various embodiments.

Referring to FIG. 2, according to various embodiments, a first electronic device (200) may be worn on a user's head and provide an image to the user. According to one embodiment, the first electronic device (200) may provide an augmented reality (AR) service that adds virtual information (e.g., a virtual object) to at least a portion of an image (e.g., a preview image) corresponding to a real space acquired through a camera of the first electronic device (200) (e.g., the camera module (180) of FIG. 1). According to one embodiment, the first electronic device (200) may provide an augmented reality service that displays virtual information (e.g., a virtual object) by overlapping it to at least a portion of a real space corresponding to a field of view (FOV) of a user wearing the first electronic device (200). For example, the first electronic device (200) may provide an augmented reality service by adding virtual information (e.g., a virtual object) to be displayed in at least a portion of a real space to light entering the eyes of a wearer of the first electronic device (200). For example, the first electronic device (200) may be configured in the form of at least one of glasses, goggles, a helmet, or a hat, but is not limited thereto.

According to various embodiments, the second electronic device (210) may detect user input. For example, the second electronic device (210) may include, but is not limited to, at least one of a portable communication device (e.g., a smartphone) or a portable multimedia device that is movable and held by a user.

According to various embodiments, the first electronic device (200) and the second electronic device (210) may be connected wirelessly and/or wiredly and may be linked. According to one embodiment, the first electronic device (200) and the second electronic device (210) may be connected wirelessly and/or wiredly and may be linked if they are authenticated by the same user. According to one embodiment, the first electronic device (200) may provide an augmented reality service based on input information provided from the second electronic device (210) connected wirelessly. For example, the input information provided from the second electronic device (210) may include at least one of a touch input, a physical button input, or movement information of the second electronic device (210). For example, the first electronic device (200) and the second electronic device (210) may be linked through at least one of short-range communication and cellular communication. For example, short-range communication may include at least one of Bluetooth, BLE (Bluetooth Low Energy), or wireless LAN (e.g., WiFi Direct). For example, cellular communication may include at least one of LTE (long term evolution), LTE-A (LTE Advance), 5G (or NR (new radio)), CDMA (code division multiple access), WCDMA (wideband CDMA), or GSM (global system for mobile communication).

FIG. 3 is a block diagram of a first electronic device for providing augmented reality services according to various embodiments. For example, the first electronic device (200) may be the electronic device (101) of FIG. 1.

Referring to FIG. 3, according to various embodiments, the first electronic device (200) may include a processor (300), a camera module (310), a display device (320), a communication circuit (330), and/or a sensor module (340). According to one embodiment, the processor (300) may be the same as the processor (120) of FIG. 1 or may be included in the processor (120). The camera module (310) may be the same as the camera module (180) of FIG. 1 or may be included in the camera module (180). The display device (320) may be the same as the display device (160) of FIG. 1 or may be included in the display device (160). The communication circuit (330) may be the same as the communication module (190) of FIG. 1 or may be included in the communication module (190). The sensor module (340) may be the same as the sensor module (176) of FIG. 1 or may be included in the sensor module (176).

According to various embodiments, the processor (300) may provide an augmented reality service. According to one embodiment, the processor (300) may control the display device (320) to display at least one virtual object so that at least one virtual object is added to at least a portion of a real space corresponding to a field of view of a user wearing the first electronic device (200). According to one embodiment, the processor (300) may control the display device (320) to display an augmented reality image in which at least one virtual object is added to at least a portion of a real-time image (e.g., a preview image) acquired through the camera module (310). For example, the at least one virtual object may include at least one of information related to at least one object located in a real space acquired through at least one of the camera module (310) or the sensor module (340) (e.g., a space sensor) or content related to the execution of an application.

According to various embodiments, the processor (300) may obtain information about the user's gaze. According to one embodiment, the processor (300) may determine the user's gaze based on at least one of head gaze and eye gaze. For example, the head gaze corresponds to the movement of the head of the user wearing the first electronic device (200), and may be set based on the center of the display device (320) or the center of a real-time image acquired through the camera module (310) (e.g., an external camera). For example, the eye gaze may be set based on the movement of the pupil acquired through the sensor module (340) (e.g., an eye tracking sensor).

According to various embodiments, the processor (300) may determine whether to link with the second electronic device (210) based on at least one object included in an area corresponding to the field of view of a user wearing the first electronic device (200). According to one embodiment, the processor (300) may determine whether to link with the second electronic device (210) based on a maintenance time of the user's gaze corresponding to an object included in an area corresponding to the field of view of a user wearing the first electronic device (200). For example, the processor (300) may determine that the first electronic device (200) is linked with the second electronic device (210) when the user's gaze corresponding to an object included in an area corresponding to the field of view of a user wearing the first electronic device (200) is continuously maintained for a reference time. For example, if the user's gaze corresponding to an object included in an area corresponding to the field of view of the user wearing the first electronic device (200) is not maintained for a reference time, the processor (300) may determine that the second electronic device (210) is not connected. For example, the object included in the area corresponding to the field of view of the user wearing the first electronic device (200) may include at least one of an object in a real space (e.g., an Internet of Things device) or a virtual object detected through at least one of a sensor module (340) (e.g., a spatial sensor) or a camera module (310).

According to one embodiment, the processor (300) may determine whether to link with the second electronic device (210) based on whether an object related to the second electronic device (210) is included in an area corresponding to a field of view of a user wearing the first electronic device (200). For example, if an object related to the second electronic device (210) is included in an area corresponding to a field of view of a user wearing the first electronic device (200), the processor (300) may determine not to link with the second electronic device (210). For example, if an object related to the second electronic device (210) does not exist in an area corresponding to a field of view of a user wearing the first electronic device (200), the processor (300) may determine to link with the second electronic device (210). For example, when the processor (300) obtains object information corresponding to an appearance configuration greater than a reference range among the entire appearance configuration of the second electronic device (210) through at least one of the camera module (310) or the sensor module (340) (e.g., a spatial sensor), the processor (300) may determine that an object related to the second electronic device (210) is included in an area corresponding to the field of view of a user wearing the first electronic device (200). For example, the object related to the second electronic device (210) may include at least one of an object related to the appearance of the second electronic device (210) obtained through at least one of the camera module (310) or the sensor module (340) (e.g., a spatial sensor) or an object related to content displayed on a display device of the second electronic device (210) (e.g., a display device (440) of FIG. 4).

According to one embodiment, the processor (300) may determine whether to link with the second electronic device (210) based on the user's gaze corresponding to an object related to the second electronic device (210) included in an area corresponding to the field of view of the user wearing the first electronic device (200). For example, if the processor (300) confirms the user's gaze corresponding to an object related to the second electronic device (210) included in an area corresponding to the field of view of the user wearing the first electronic device (200), the processor (300) may determine not to link with the second electronic device (210). For example, if the user's gaze corresponding to an object related to the second electronic device (210) included in an area corresponding to the field of view of the user wearing the first electronic device (200) is maintained for a reference time, the processor (300) may determine not to link with the second electronic device (210). For example, the processor (300) may determine that the second electronic device (210) is connected when there is no object related to the second electronic device (210) within an area corresponding to the field of view of the user wearing the first electronic device (200), or when the user's gaze is identified to correspond to another object different from the object related to the second electronic device (210).

According to one embodiment, the processor (300) may determine whether to link with the second electronic device (210) based on location information of the first electronic device (200) and the second electronic device (210). For example, the processor (300) may determine that the second electronic device (210) is linked with the second electronic device (210) if the second electronic device (210) is located within a reference range of the first electronic device (200). For example, the processor (300) may determine that the second electronic device (210) is not linked with the second electronic device (210) if the second electronic device (210) is located outside the reference range of the first electronic device (200). For example, the location information of the first electronic device (200) may be acquired based on the sensor module (340). The location information of the second electronic device (210) can be provided from the second electronic device (210) via a wireless path with the second electronic device (210).

According to various embodiments, when the processor (300) determines that the second electronic device (210) is linked, the processor (300) may control (or manage) the augmented reality service based on input information provided from the second electronic device (210). According to one embodiment, when the processor (300) determines that the second electronic device (210) is linked, the processor (300) may transmit linkage information to the second electronic device (210). When the processor (300) receives input information from the second electronic device (210) in response to the linkage information, the processor (300) may control the augmented reality service based on the input information provided from the second electronic device (210). For example, the processor (300) may control the display device (320) to display a functional user interface (UI) corresponding to an object included in an area corresponding to a field of view of a user wearing the first electronic device (200) based on the input information provided from the second electronic device (210). The processor (320) may control the operation of the corresponding object based on a menu corresponding to additional input information provided from the second electronic device (210) among at least one menu included in the functional UI. For example, the object on which the functional user interface is displayed may include an object corresponding to the user's gaze at the time of receiving input information from the second electronic device (210) among objects included in an area corresponding to the field of view of the user wearing the first electronic device (200). For example, the processor (300) may provide a manipulation user interface corresponding to an object included in an area corresponding to the field of view of the user wearing the first electronic device (200) based on the input information provided from the second electronic device (210). For example, the processor (300) may adjust the display form (e.g., size, position, or angle) of an object corresponding to the user's gaze based on the input information provided from the second electronic device (210). For example, the input information provided from the second electronic device (210) may include at least one of a user input detected through an input device of the second electronic device (210) (e.g., an input device (420) of FIG. 4) or sensing data acquired through a sensor module of the second electronic device (210) (e.g., a sensor module (430) of FIG. 4). According to one embodiment, if the processor (300) determines that it is not linked with the second electronic device (210), it may transmit non-linking information to the second electronic device (210).

According to various embodiments, when the processor (300) determines that the second electronic device (210) is linked, the processor (300) may set a display method related to the augmented reality service based on the direction of the second electronic device (210). For example, when the direction of the second electronic device (210) is configured in a longitudinal shape and the relatively long axis is horizontal to the direction of gravity, the second electronic device (210) may be determined to be in a vertical direction, and when the relatively long axis is perpendicular to the direction of gravity, the second electronic device (210) may be determined to be in a horizontal direction. According to one embodiment, the processor (300) may set a display method of at least one virtual object for the augmented reality service based on the direction of the second electronic device (210). For example, when the second electronic device (210) is in a vertical direction, the display device (320) may display a virtual object in a vertical direction. For example, the display device (320) may display a virtual object in a horizontal direction when the second electronic device (210) is in a horizontal direction. According to one embodiment, the processor (300) may set the shape of the functional UI differently based on the direction of the second electronic device (210). For example, the shape of the functional UI may include at least one of the type of menu included in the functional UI, the arrangement of the menu, or the method of switching the menu. For example, the direction of the second electronic device (210) may be confirmed based on sensing data provided from the second electronic device (210).

According to various embodiments, the processor (300) may register an Internet of Things (IoT) device in conjunction with the second electronic device (210). According to one embodiment, the processor (300) may detect an Internet of Things device located in an area corresponding to the field of view of a user wearing the first electronic device (200) by using at least one of a camera module (310) or a sensor module (340) (e.g., a spatial sensor). If registration information of the Internet of Things device does not exist, the processor (300) may transmit information related to the Internet of Things device to the second electronic device (210). For example, if the processor (300) receives registration request information related to the Internet of Things device from the second electronic device (210), the processor (300) may control the display device (320) to display a registration screen of the Internet of Things device. When the processor (300) detects a registration input corresponding to a registration screen displayed on the display device (320), the processor (300) can register the Internet of Things device at the location (e.g., living room) of the first electronic device (200). For example, when the processor (300) receives registration information related to the Internet of Things device from the second electronic device (210), the processor (300) can control the display device (320) to display a virtual object (e.g., information on the Internet of Things device) corresponding to the Internet of Things device. For example, the information related to the Internet of Things device may include information on the type and/or location of the device.

According to various embodiments, the processor (300) may change the display method and/or display information of the display device (320) based on the user's gaze information. According to one embodiment, if the processor (300) determines that the gaze of the user wearing the first electronic device (200) is gazing at an object related to the second electronic device (210) included in an area corresponding to the user's field of view, the processor (300) may adjust the transparency and/or brightness of a virtual object displayed on the display device (440). For example, if the processor (300) determines that the gaze of the user is gazing at an object related to the second electronic device (210), the processor (300) may control the display device (320) to relatively brighten the brightness of at least one virtual object displayed on the display device (320) or relatively transparently adjust the transparency so that the user can focus on the display screen of the second electronic device (210).

According to various embodiments, the camera module (310) can capture still images and/or videos. According to one embodiment, the camera module (310) can include an external camera (not shown) for capturing real-time images corresponding to the field of view of a user wearing the first electronic device (200).

According to various embodiments, the display device (320) may display information processed in the first electronic device (200). According to one embodiment, the display device (320) may display at least one virtual object within an area corresponding to the user's field of view while the user wearing the first electronic device (200) views an actual space through the display device (320). For example, the user's field of view may include an angle or range that allows the user to recognize an object without turning their head or eyes while looking straight ahead. For example, the area corresponding to the user's field of view may be set to be equal to or smaller than the entire display area of the display device (320). Additionally, the area corresponding to the user's field of view may be determined or adjusted based on the distance from the user's eyes to the display device (320). According to one embodiment, the display device (320) may display an image in the form of augmented reality (AR) by adding at least one virtual object to a real-time image acquired through the camera module (310).

According to various embodiments, the communication circuit (330) may support communication using a wired and/or wireless communication channel between the first electronic device (200) and the second electronic device (210). For example, the wireless channel between the first electronic device (200) and the second electronic device (210) may be established based on a short-range communication method or a cellular communication method.

According to various embodiments, the sensor module (340) may detect the operating state of the first electronic device (200) or the external environmental state and provide the detected information to the processor (300). According to one embodiment, the sensor module (340) may include a motion sensor for detecting the movement of the first electronic device (200) (or the user). For example, the motion sensor may include at least one of an acceleration sensor, a tilt sensor, a gyro sensor, or a 3-axis magnetic sensor. According to one embodiment, the sensor module (340) may include a position sensor (e.g., a global navigation satellite system (GNSS)) for detecting position information of the first electronic device (200). According to one embodiment, the sensor module (340) may include a detection sensor (e.g., a proximity sensor) for determining whether the first electronic device (200) is worn by a user. According to one embodiment, the sensor module (340) may include a spatial sensor for recognizing an object included in an area corresponding to the field of view of a user wearing the first electronic device (200). For example, the spatial sensor may include at least one of a light sensor (e.g., an infrared sensor) or a depth sensor.

FIG. 4 is a block diagram of a second electronic device for providing augmented reality services according to various embodiments. For example, the second electronic device (210) may be the electronic device (101) of FIG. 1.

Referring to FIG. 4, according to various embodiments, the second electronic device (210) may include a processor (400), a communication circuit (410), an input device (420), a sensor module (430), and/or a display device (440). According to one embodiment, the processor (400) may be the same as the processor (120) of FIG. 1 or may be included in the processor (120). The communication circuit (410) may be the same as the communication module (190) of FIG. 1 or may be included in the communication module (190). The input device (420) may be the same as the input device (150) of FIG. 1 or may be included in the input device (150). The sensor module (430) may be the same as the sensor module (176) of FIG. 1 or may be included in the sensor module (176). The display device (440) may be the same as the display device (160) of FIG. 1 or may be included in the display device (160).

According to various embodiments, the processor (400) may set the input mode of the second electronic device (210). According to one embodiment, the processor (400) may set the input mode of the second electronic device (210) based on the linkage information provided from the first electronic device (200). For example, when the processor (400) receives linkage information from the first electronic device (200) through the communication circuit (410), the processor (400) may determine that the second electronic device (210) is linked to the first electronic device (200) to provide an augmented reality service. In this case, the processor (400) may set the input mode of the second electronic device (210) to the first input mode for sharing input information with the first electronic device (200). For example, if the processor (400) receives non-linking information from the first electronic device (200) through the communication circuit (410), it may determine that linking with the first electronic device (200) for providing an augmented reality service is restricted. In this case, the processor (400) may set the input mode of the second electronic device (210) to a second input mode in which input information sharing with the first electronic device (200) is restricted.

According to one embodiment, the processor (400) may set the input mode of the second electronic device (210) based on information related to the augmented reality service provided from the first electronic device (200). For example, the information related to the augmented reality service may include at least one of object information included in an area corresponding to the field of view of a user wearing the first electronic device (200), gaze information of the user wearing the first electronic device (200), or location information of the first electronic device (200). For example, the processor (400) may determine whether the second electronic device (210) will be linked with the first electronic device (200) to provide the augmented reality service based on object information included in an area corresponding to the field of view of a user wearing the first electronic device (200). For example, the processor (400) may determine whether to link with the first electronic device (200) based on at least one of: a duration of a user's gaze corresponding to an object included in an area corresponding to the field of view of a user wearing the first electronic device (200), whether an object related to the second electronic device (210) is included in an area corresponding to the field of view of a user wearing the first electronic device (200), or user gaze information corresponding to an object related to the second electronic device (210). For example, the processor (400) may determine whether the second electronic device (210) will link with the first electronic device (200) to provide an augmented reality service based on location information of the first electronic device (200). For example, the input mode of the second electronic device (210) may be set to the first input mode when the second electronic device (210) links with the first electronic device (200) to provide an augmented reality service. For example, the input mode of the second electronic device (210) may be set to the second input mode when the second electronic device (210) is limited in its connection with the first electronic device (200) for providing an augmented reality service.

According to various embodiments, the processor (400) may control to selectively share input information with the first electronic device (200). According to one embodiment, when the processor (400) detects an input through the input device (420) and/or the sensor module (430), the processor (400) may check the input mode of the second electronic device (210). For example, when the input mode of the second electronic device (210) is the first input mode, the processor (400) may control the communication circuit (410) to transmit the input information to the first electronic device (200). In this case, the processor (400) may restrict the second electronic device (210) from performing an operation corresponding to the input. For example, when the input mode of the second electronic device (210) is the second input mode, the processor (400) may perform an operation corresponding to the input information.

According to various embodiments, the processor (400) may change the display method and/or display information of the display device (440) when the second electronic device (210) is linked with the first electronic device (200) to provide an augmented reality service. According to one embodiment, when the processor (400) determines that the second electronic device (210) is linked with the first electronic device (200) to provide an augmented reality service, the processor (400) may adjust the active mode and/or brightness of the display device (440). For example, when the processor (400) determines that the second electronic device (210) is linked with the first electronic device (200), the processor (400) may control the display device (440) to switch to an inactive mode. For example, when the processor (400) determines that the second electronic device (210) is linked with the first electronic device (200), the processor (400) may control the display device (440) to darken the display area of the display device (440). According to one embodiment, if the processor (400) determines that the second electronic device (210) is linked with the first electronic device (200) to provide an augmented reality service, the processor (400) may change the content of the display device (440). For example, if the processor (400) determines that the second electronic device (210) is linked with the first electronic device (200), the processor (400) may control the display device (440) to display content related to the augmented reality service provided by the first electronic device (200).

According to various embodiments, the processor (400) may register an Internet of Things (IoT) device in conjunction with the first electronic device (200). According to one embodiment, the processor (400) may receive information related to the Internet of Things device from the first electronic device (200) through the communication circuit (410). The processor (400) may confirm registration information related to the Internet of Things device provided from the first electronic device (200) through a memory (e.g., memory (130) of FIG. 1). For example, when the processor (400) confirms registration information of the Internet of Things device, the processor (400) may control the communication circuit (410) to transmit the registration information of the Internet of Things device to the first electronic device (200). For example, when the registration information of the Internet of Things device is not confirmed, the processor (400) may control the communication circuit (410) to transmit registration request information to the first electronic device (200). When the processor (400) receives registration information of an Internet of Things device from the first electronic device (200) through the communication circuit (410), the processor (400) can register the Internet of Things device at the location of the first electronic device (200) (or the second electronic device (210)). For example, the information related to the Internet of Things device can include the type of device recognized by the first electronic device (200) and/or location information of the first electronic device (200).

According to various embodiments, the communication circuit (410) may support communication using a wired and/or wireless communication channel between the first electronic device (200) and the second electronic device (210). For example, the wireless channel between the first electronic device (200) and the second electronic device (210) may be established based on a short-range communication method such as Bluetooth, BLE, or wireless LAN.

According to various embodiments, the input device (420) may receive commands or data related to the second electronic device (210) from an external source (e.g., a user) of the second electronic device (210). For example, the input device (420) may include a microphone, a mouse, a keyboard, a digital pen (e.g., a stylus pen), or a touch circuit capable of detecting touch input.

According to various embodiments, the sensor module (430) may detect the operating state of the second electronic device (210) or the external environmental state and provide the detected information to the processor (400). According to one embodiment, the sensor module (430) may include a motion sensor for detecting the movement of the second electronic device (210) (or the user). According to one embodiment, the sensor module (430) may include a location sensor (e.g., GNSS) for detecting location information of the second electronic device (210). According to one embodiment, the sensor module (430) may also include a detection sensor (e.g., a proximity sensor) for determining whether the second electronic device (210) is worn by a user.

According to various embodiments, the display device (440) may display information processed in the second electronic device (210). According to one embodiment, the display device (440) may display content related to an application running in the second electronic device (210).

According to various embodiments, the first electronic device (200) may provide an augmented reality service based on the control of the second electronic device (210) or an external server (e.g., the server (108) of FIG. 1). According to one embodiment, the first electronic device (200) may display at least one virtual object so that at least one virtual object is added to at least a portion of a real space corresponding to a field of view of a user wearing the first electronic device (200) based on information related to the virtual object provided from the second electronic device (210) or an external server. For example, the information related to the virtual object may include at least one of a location for displaying the virtual object, a type of the virtual object, or a configuration of the virtual object. According to one embodiment, the first electronic device (200) may transmit at least one of object information (or spatial information) included in an area corresponding to the field of view of a user wearing the first electronic device (200) or the user's gaze information, obtained through at least one of a camera module (310) or a sensor module (340) (e.g., a spatial sensor), to the second electronic device (210) or an external server. According to one embodiment, the second electronic device (210) or the external server may determine the input mode of the second electronic device (210) based on at least one of object information (or spatial information) included in an area corresponding to the field of view of a user wearing the first electronic device (200) or the user's gaze information, provided from the first electronic device (200). For example, if the second electronic device (210) or the external server determines that the user's gaze corresponds to an object related to the second electronic device (210) included in an area corresponding to the field of view of the user wearing the first electronic device (200), the second electronic device (210) or the external server may set the input mode of the second electronic device (210) to the second input mode. For example, if the object related to the second electronic device (210) does not exist in the area corresponding to the field of view of the user wearing the first electronic device (200), or the user's gaze corresponds to a different object than the object related to the second electronic device (210), the second electronic device (210) or the external server may set the input mode of the second electronic device (210) to the first input mode. In this case, the second electronic device (210) may determine the input of the second electronic device (210) as an input for controlling an augmented reality service.

According to various embodiments, a first electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 2) includes a communication circuit, a camera module, a sensor module, a display device, and a processor operatively connected to the communication circuit, the display device, the sensor module, and the camera module, wherein the processor communicates with a second electronic device through the communication circuit, obtains at least one object information included in an area corresponding to a field of view of a user wearing the first electronic device through at least one of the camera module and the sensor module, obtains gaze information of the user, determines whether to link an augmented reality service with the second electronic device based on the gaze information of the user and the at least one object information, and when it is determined to link the augmented reality service with the second electronic device, transmits linking information of the augmented reality service to the second electronic device through the communication circuit.

According to various embodiments, the processor may obtain gaze information of the user based on at least one of a head gaze or an eye gaze of the first electronic device.

According to various embodiments, the processor may control the display device to display at least one virtual object in at least a portion of an area corresponding to a field of view of a user wearing the first electronic device.

According to various embodiments, the processor may determine that the augmented reality service is linked with the second electronic device when the user's gaze corresponding to any one of at least one object included in an area corresponding to a field of view of a user wearing the first electronic device or any one of the at least one virtual object is continuously maintained for a reference time, and may determine that the augmented reality service is not linked with the second electronic device when the user's gaze corresponding to any one of the objects is not maintained for the reference time.

According to various embodiments, when an object related to the second electronic device exists in an area corresponding to a field of view of a user wearing the first electronic device, the processor may determine whether the user's gaze is directed at the object related to the second electronic device, and when the user's gaze is directed at another object different from the object related to the second electronic device, determine that the second electronic device is linked to an augmented reality service, and when the user's gaze is directed at the object related to the second electronic device, determine that the second electronic device is not linked to an augmented reality service.

According to various embodiments, when the processor determines to link the second electronic device with an augmented reality service, the processor may determine whether input information is received from the second electronic device through the communication circuit, and when input information is received from the second electronic device, perform an operation corresponding to the input information.

According to various embodiments, the input information received from the second electronic device may include at least one of a touch input, a physical button input, or movement information of the second electronic device.

According to various embodiments, when the processor receives input information from the second electronic device, the processor can control the display device to identify an object corresponding to the user's gaze and display a control menu related to the object corresponding to the user's gaze.

According to various embodiments, a second electronic device (e.g., the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 2) includes a communication circuit, an input device, and a processor operatively connected to the communication circuit and the input device, wherein the processor is configured to communicate with the first electronic device through the communication circuit, and, when an input is detected through the input device, determine an input mode of the second electronic device, and, when the input mode is the first mode, transmit the input information to the first electronic device through the communication circuit, and, when the input mode is the second mode, perform an operation corresponding to the input information.

According to various embodiments, when the processor receives linkage information related to an augmented reality service from the first electronic device through the communication circuit, the processor may set the input mode of the second electronic device to the first mode, and when linkage information related to an augmented reality service is not received from the first electronic device through the communication circuit or non-linkage information is received, the processor may set the input mode of the second electronic device to the second mode.

According to various embodiments, the display device further includes, wherein the processor can control the display device to change display information and/or a display method of the display device when linkage information related to an augmented reality service is received from the first electronic device through the communication circuit.

According to various embodiments, the input device may include at least one of a touch circuit or a physical button.

According to various embodiments, a second electronic device (e.g., the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 2) includes a communication circuit, a display device, and a processor operatively connected to the communication circuit and the display device, wherein the processor receives, from the first electronic device through the communication circuit, at least one piece of object information included in an area corresponding to a field of view of a user wearing the first electronic device, and determines whether an object related to the second electronic device exists in the area corresponding to the field of view of the user wearing the first electronic device based on the at least one piece of object information, and when an object related to the second electronic device exists in the area corresponding to the field of view of the user wearing the first electronic device, determines whether the user's gaze gazes at the object related to the second electronic device, and when the user's gaze gazes at another object different from the object related to the second electronic device, determines that the first electronic device and an augmented reality service are linked.

According to various embodiments, the processor may obtain gaze information of the user based on at least one of a head gaze or an eye gaze of the first electronic device.

According to various embodiments, the processor may control the first electronic device to display at least one virtual object in at least a portion of an area corresponding to a field of view of a user wearing the first electronic device.

According to various embodiments, the processor may determine to link the first electronic device with an augmented reality service when no object related to the second electronic device exists in an area corresponding to a field of view of a user wearing the first electronic device.

According to various embodiments, the processor may determine not to link the first electronic device with the augmented reality service when the user's gaze is directed at an object associated with the second electronic device.

According to various embodiments, when the processor determines to link the first electronic device with an augmented reality service, the processor may transmit information related to an operation corresponding to input information acquired from the second electronic device to the first electronic device.

According to various embodiments, the input information obtained from the second electronic device may include at least one of a touch input, a physical button input, or movement information of the second electronic device.

According to various embodiments, when the processor detects the input information, the processor may identify an object corresponding to the user's gaze based on the at least one object information, and transmit a control menu related to the object corresponding to the user's gaze to the first electronic device.

FIG. 5 is a flowchart for linking a first electronic device with a second electronic device according to various embodiments. In the following embodiments, the respective operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 5 will be described with reference to FIGS. 6A and 6D. FIGS. 6A and 6B illustrate screen configurations related to an augmented reality service displayed on a first electronic device according to various embodiments.

Referring to FIG. 5, according to various embodiments, a first electronic device (e.g., a processor (120) of FIG. 1, a communication module (190) of FIG. 1, a processor (300) of FIG. 3, or a communication circuit (330) of FIG. 3) may connect a communication link (or communication channel) with a second electronic device (e.g., a second electronic device (210) of FIG. 2) in operation 501. According to one embodiment, when user information set in the first electronic device (200) and the second electronic device (210) are the same, the processor (300) may control the communication module (330) to establish a communication link (or communication channel) with the second electronic device (210).

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may output information related to the augmented reality service at operation 503. According to one embodiment, the processor (300) can recognize at least one object existing in a real space corresponding to the field of view of a user wearing the first electronic device (200) through at least one of a camera module (310) or a sensor module (340 (e.g., a space sensor). The processor (300) can control the display device (320) to display at least one virtual object related to at least one object existing in the real space. For example, the display device (320) can display at least one virtual object (614 and/or 616) within an area (610) corresponding to the field of view of the user while the user wearing the first electronic device (200) views the real space through the display device (320), as shown in FIG. 6A. For example, the first virtual object (614) can include information (e.g., status information) related to an object (612) in a real space (e.g., a television). For example, the second virtual object (616) can include information related to the first electronic device (200). It may include information (e.g., a playback screen) related to an application (e.g., a music application) running on the device (200). According to one embodiment, the processor (300) may control the display device (320) to display an augmented reality image in which at least one virtual object is added to at least a portion of a real-time image (e.g., a preview image) acquired through the camera module (310). According to one embodiment, the processor (300) may set a display method of the virtual object based on the direction of the second electronic device (210) connected to the communication link with the first electronic device (200). For example, as shown in FIG. 6b, when the second electronic device (210) is in a vertical direction (620), the processor (300) may control the display device (320) to display a plurality of virtual objects (632) in a vertical direction in an area (610) corresponding to the field of view of a user wearing the first electronic device (200). For example, as shown in FIG. 6c, the processor (300) may control the second electronic device (210) to display a plurality of virtual objects (632) in a vertical direction. When the electronic device (210) is in a horizontal orientation (640), the display device (320) can be controlled to display a plurality of virtual objects (652) in a horizontal orientation in an area (610) corresponding to the field of view of a user wearing the first electronic device (200).

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may determine whether to link with a second electronic device (e.g., the second electronic device (210) of FIG. 2) while providing an augmented reality service in operation 505. According to one embodiment, the processor (300) may determine whether to link with the second electronic device based on object information included in an area corresponding to a field of view of a user wearing the first electronic device (200). For example, the object information included in the area corresponding to the field of view of the user may include at least one of: a maintenance time of a user's gaze corresponding to an object included in an area corresponding to the field of view of the user wearing the first electronic device (200), whether an object related to the second electronic device (210) is included in an area corresponding to the field of view of the user wearing the first electronic device (200), or user gaze information corresponding to an object related to the second electronic device (210). According to one embodiment, the processor (300) can determine whether to link with the second electronic device based on location information of the first electronic device (200) and the second electronic device (210).

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) determines that an augmented reality service is to be linked with a second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 505), in operation 507, the processor (300) may control the communication circuit (330) to transmit information indicating linkage of the augmented reality service with the second electronic device (210) to the second electronic device (210).

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) determines that an augmented reality service is not linked with a second electronic device (e.g., second electronic device (210) of FIG. 2) (e.g., 'No' in operation 505), the first electronic device may transmit information indicating non-linkage of the augmented reality service to the second electronic device. According to one embodiment, when the processor (300) determines that an augmented reality service is not linked with the second electronic device (210), the processor (300) may control the communication circuit (330) to transmit information indicating non-linkage of the augmented reality service to the second electronic device (210).

According to various embodiments, when providing an augmented reality service, the first electronic device (200) may display an object corresponding to the user's gaze so as to be distinguished from other objects. For example, the first electronic device (200) may display a virtual object related to the user's gaze around an object (612) corresponding to the user's gaze, as shown in FIG. 6A. For example, the virtual object related to the user's gaze may be displayed around the object (612) corresponding to the user's gaze if the object (612) corresponding to the user's gaze is an active object. For example, the first electronic device (200) may display the object (600) (e.g., a virtual object) corresponding to the user's gaze to be relatively larger than other objects, as shown in FIG. 6D. For example, the active object may include at least one of an object in a real space or a virtual object defined to provide various information based on the augmented reality service among a plurality of objects included in an area corresponding to the field of view of a user wearing the first electronic device (200).

According to various embodiments, the first electronic device (200) may determine whether an augmented reality service is linked with the second electronic device (210) based on a request from the second electronic device (210). According to one embodiment, when the second electronic device (210) detects input information, the second electronic device (210) may transmit a signal for determining whether an augmented reality service is linked to the first electronic device (200). According to one embodiment, the first electronic device (200) may determine whether an augmented reality service is linked with the second electronic device based on object information included in an area corresponding to a field of view of a user wearing the first electronic device (200) at the time of receiving the signal for determining whether an augmented reality service is linked. According to one embodiment, the first electronic device (200) may determine whether an augmented reality service is linked with the second electronic device based on location information of the first electronic device (200) and the second electronic device (210) at the time of receiving the signal for determining whether an augmented reality service is linked.

FIG. 7 is a flowchart for linking with a second electronic device based on gaze information related to an object in a first electronic device according to various embodiments. According to one embodiment, the operations of FIG. 7 may be detailed operations of operation 505 of FIG. 5. In the following embodiments, the respective operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 7 will be described with reference to FIGS. 8A to 8D. FIGS. 8A to 8D are screen configurations of the first electronic device for linking with a second electronic device based on gaze information related to an object according to various embodiments.

Referring to FIG. 7, according to various embodiments, a first electronic device (e.g., the processor (120) of FIG. 1 or the processor (300) of FIG. 3) may, in operation 701, identify an object corresponding to the gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2). In one embodiment, when providing an augmented reality service, the processor (300) may control the display device (320) to display at least one virtual object in an overlapping manner in a real space corresponding to the field of view of the user wearing the first electronic device (200). For example, a user wearing the first electronic device (200) may identify multiple objects (810 and 820) through an area (800) corresponding to the field of view of the user wearing the first electronic device (200), as illustrated in FIG. 8A. For example, the object corresponding to the user's gaze may include the first object (810) in which the user's gaze (830) is displayed in FIG. 8A. For example, the first object (810) may include a virtual object displayed on the display device (320), and the second object (820) may include an object (e.g., an Internet of Things device) existing in a real space corresponding to the user's field of view. For example, the area (800) corresponding to the field of view of the user wearing the first electronic device (200) may be set to be equal to or smaller than the display area of the display device (320).

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may, in operation 703, determine whether an object corresponding to the gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is an active object. For example, the active object may include at least one of an object in a real space or a virtual object defined to provide various information based on an augmented reality service among a plurality of objects included in an area corresponding to the field of view of a user wearing the first electronic device (200). For example, the active object may be set or removed based on a user's input.

According to various embodiments, when an object corresponding to a gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is an active object (e.g., 'Yes' in operation 703), in operation 705, the first electronic device (e.g., the processor (300)) may determine whether the user's gaze on the active object is continuously maintained for a reference time. According to one embodiment, the processor (300) may determine whether the user's gaze (830) corresponding to the first object (810) of FIG. 8A is continuously maintained for a reference time. For example, the user's gaze may be set based on at least one of a head gaze or an eye gaze.

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may determine to link with the second electronic device (e.g., the second electronic device (210) of FIG. 2) an augmented reality service if the user's gaze on the active object is continuously maintained for a reference time (e.g., 'yes' in operation 705) in operation 707. According to one embodiment, the processor (300) may determine to set the second electronic device (210) as an input device of the augmented reality service if the user's gaze on the active object included in an area corresponding to the field of view of the user wearing the first electronic device (200) is continuously maintained for a reference time.

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may determine not to link the augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2) in operation 709 if an object corresponding to the gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is not an active object (e.g., 'No' of operation 703) or the user's gaze on the active object is not maintained for a reference time (e.g., 'No' of operation 705).

According to various embodiments, the first electronic device (200) may determine whether to link with the second electronic device (210) an augmented reality service based on a request from the second electronic device (210). According to one embodiment, when the second electronic device (210) detects input information, it may transmit a signal to the first electronic device (200) for determining whether to link with the augmented reality service. When the first electronic device (200) receives a signal from the second electronic device (210) for determining whether to link with the augmented reality service while the user's gaze is maintained on an active object, it may determine that the second electronic device (210) and the augmented reality service are linked. In this case, the first electronic device (200) may determine that the augmented reality service is linked with the second electronic device (210) regardless of the time for which the user maintains gaze on the active object.

According to various embodiments, when the first electronic device (200) determines that the second electronic device (210) is linked to the augmented reality service, the first electronic device (200) may control the augmented reality service based on input information provided from the second electronic device (210). According to one embodiment, when the processor (300) receives information of a tap input (841) from the second electronic device (210), as shown in FIG. 8B , the processor (300) may control the display device (320) to display a function UI (850) of a first object (810) corresponding to the user's gaze. For example, the function UI (850) may be displayed so as to overlap or be adjacent to the first object (810) at least partially. For example, the display device (320) may display a cursor on a first menu (851) (e.g., a cancel menu) among a plurality of menus corresponding to the first object (810) included in the function UI (850). For example, when the processor (300) receives an additional input (e.g., a tap input) from the second electronic device (210), the processor (300) may control the display device (320) to remove the display of the function UI (850) based on the first menu (851) on which the cursor is displayed. According to one embodiment, when the processor (300) receives information of a touch motion input (843) (e.g., a drag) in a first direction (e.g., a leftward direction) from the second electronic device (210) as shown in FIG. 8C, the processor (300) may move the cursor displayed on the function UI (850) of the first object (810) in the first direction (e.g., a leftward direction). For example, the display device (320) may display the cursor on the second menu (852) (e.g., a time menu) among a plurality of menus corresponding to the first object (810) included in the function UI (850) due to the cursor movement in the first direction (e.g., the leftward direction). For example, when the processor (300) receives an additional input (e.g., a tap input) from the second electronic device (210), the processor (300) may control the display device (320) to display time information based on the second menu (852) on which the cursor is displayed. According to one embodiment, when the processor (300) receives information of a touch motion input (845) (e.g., a drag) in a second direction (e.g., a rightward direction) from the second electronic device (210), as shown in FIG. 8D , the processor (300) may move the cursor displayed on the function UI (850) of the first object (810) in the second direction (e.g., a rightward direction). For example, the display device (320) may display the cursor on a third menu (854) (e.g., a location menu) among a plurality of menus corresponding to the first object (810) included in the function UI (850) due to the cursor movement in the second direction (e.g., the rightward direction). For example, when the processor (300) receives additional input (e.g., tap input) from the second electronic device (210), it can control the display device (320) to display location information based on the third menu (854) where the cursor is displayed.

FIG. 9 is a flowchart for linking a first electronic device with a second electronic device based on gaze information related to an object related to the second electronic device according to various embodiments. According to one embodiment, the operations of FIG. 9 may be detailed operations of operation 505 of FIG. 5. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 9 will be described with reference to FIGS. 10A to 10D and FIGS. 11A to 11C. FIGS. 10A to 10D are screen configurations of the first electronic device for linking with a second electronic device based on gaze information of a user according to various embodiments. FIGS. 11A to 11C are screen configurations of a first electronic device for interfacing with a second electronic device based on gaze information related to an object associated with the second electronic device according to various embodiments.

Referring to FIG. 9, according to various embodiments, a first electronic device (e.g., the processor (120) of FIG. 1 or the processor (300) of FIG. 3) may, in operation 901, identify an object corresponding to the line of sight of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2). In one embodiment, the user wearing the first electronic device (200) may identify an object (e.g., an Internet of Things device) (1010) existing in a real space through an area (1000) corresponding to the field of view of the user wearing the first electronic device (200), as shown in FIG. 10A. The processor (300) may recognize an object (1010) located in a real space corresponding to the field of view of the user through at least one of a camera module (310) or a sensor module (340) (e.g., a space sensor). For example, an object corresponding to a user's gaze may include an object (1010) indicating the user's gaze (1020) in FIG. 10a.

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may, in operation 903, determine whether an object corresponding to the gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is an object related to a second electronic device (e.g., the second electronic device (210) of FIG. 2). For example, the object related to the second electronic device (210) may include an object related to content displayed on a display device (440) of the second electronic device (210) detected by analyzing an image acquired through a camera module (310). For example, the object related to the second electronic device (210) may include an object related to an appearance of the second electronic device (210) recognized through at least one of the camera module (310) or the sensor module (340) (e.g., a spatial sensor).

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may determine, in operation 905, that the augmented reality service is linked with the second electronic device (e.g., the second electronic device (210) of FIG. 2) if the object corresponding to the gaze of the user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is not an object related to the second electronic device (e.g., 'No' of operation 903). According to one embodiment, when the processor (300) determines that the gaze of a user wearing the first electronic device (200) is gazing (1020) at an object (1010) (e.g., an Internet of Things device) other than an object related to the second electronic device (210), as shown in FIG. 10a, the processor (300) may determine that the second electronic device (210) and the augmented reality service are linked.

According to various embodiments, when an object corresponding to the gaze of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) is an object related to a second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 903), the first electronic device (e.g., the processor (120 or 300)) may determine not to link the augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2)) in operation 907. According to one embodiment, when the processor (300) determines that the gaze of the user wearing the first electronic device (200) is gazing at an object (1110) related to the second electronic device (210) included in an area (1100) corresponding to the user's field of view, as shown in FIG. 11A, the processor (300) may determine not to link the augmented reality service with the second electronic device (210).

According to various embodiments, when the first electronic device (200) determines that the second electronic device (210) is linked to the augmented reality service, the first electronic device (200) may control the augmented reality service based on input information provided from the second electronic device (210). According to one embodiment, when the processor (300) receives information of a tap input (1030) from the second electronic device (210), as shown in FIG. 10B , the processor (300) may control the display device (320) to display a function UI (1040) of an object (1010) corresponding to the user's gaze. For example, the function UI (1040) may be displayed to overlap at least partly with the object (1010). For example, the display device (320) may display a cursor on a first menu (1041) (e.g., a cancel menu) among a plurality of menus related to the object (1010) included in the function UI (1040). According to one embodiment, when the processor (300) receives information of a touch motion input (1032) (e.g., a drag) in a second direction (e.g., a rightward direction) from the second electronic device (210), as shown in FIG. 10c, the processor (300) may move a cursor displayed on the function UI (1040) in the second direction (e.g., a rightward direction). For example, the display device (320) may display a cursor on a third menu (1042) (e.g., a location menu) among a plurality of menus related to an object (1010) included in the function UI (1040) due to the cursor movement in the second direction (e.g., a rightward direction). According to one embodiment, when the processor (300) receives information of a touch motion input (1034) (e.g., a drag) in a first direction (e.g., a leftward direction) from the second electronic device (210), as shown in FIG. 10d, the processor (300) may move a cursor displayed on the function UI (1040) in the first direction (e.g., a leftward direction). For example, the display device (320) may display a cursor on a fourth menu (1044) (e.g., a switch menu) among a plurality of menus related to an object (1010) included in the function UI (1040) due to the cursor movement in the first direction (e.g., a leftward direction). According to one embodiment, when the processor (300) receives an additional input (e.g., a tap input) from the second electronic device (840), the processor (300) may perform an operation corresponding to a menu (1041, 1042, or 1043) on which a cursor is displayed. For example, when the processor (300) receives additional input (e.g., tap input) from the second electronic device (840) while the cursor is displayed on the fourth menu (1044), the processor (300) may control the communication circuit (330) to transmit an activation signal (or deactivation signal) to the Internet of Things device corresponding to the object (1010).

According to various embodiments, if the second electronic device (210) determines not to link the augmented reality service with the first electronic device (210), the second electronic device (210) may process input information from the second electronic device (210). According to one embodiment, the processor (400) may control the display device (440) to display a list of images stored in the second electronic device (210) based on the execution of the gallery application, as shown in FIG. 11A. If the processor (400) detects a tap input (1120) through the input device (420) (e.g., a touch circuit), as shown in FIG. 11B, the processor (400) may enlarge and display (1130) a first image corresponding to the tap input (1120), as shown in FIG. 11C.

FIG. 12 is a flowchart for interworking with a second electronic device based on FOV information in a first electronic device according to various embodiments. According to one embodiment, the operations of FIG. 12 may be detailed operations of operation 505 of FIG. 5. In the following embodiments, the respective operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 12 will be described with reference to FIG. 13. FIG. 13 is a screen configuration of a first electronic device for interworking with a second electronic device based on FOV information of the first electronic device according to various embodiments.

Referring to FIG. 12, according to various embodiments, a first electronic device (e.g., the processor (120) of FIG. 1 or the processor (300) of FIG. 3) may, in operation 1201, determine whether an object related to a second electronic device (e.g., the second electronic device (210) of FIG. 2) is included in an area corresponding to a field of view of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2). According to one embodiment, the processor (300) may recognize at least one object existing in a real space corresponding to the field of view of the user through at least one of a camera module (310) or a sensor module (340) (e.g., a space sensor). In this case, the processor (300) may determine whether an object related to the second electronic device (210) exists among the at least one object existing in the real space. For example, an object related to the second electronic device (210) may include at least one of an object related to the external appearance of the second electronic device (210) or an object related to content displayed on a display device (440) of the second electronic device (210).

According to various embodiments, if an object related to a second electronic device (e.g., the second electronic device (210) of FIG. 2) is not included in an area corresponding to a field of view of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'No' in operation 1201), the first electronic device (e.g., the processor (120 or 300)) may determine to link the augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2)) in operation 1203. According to one embodiment, if an object related to the second electronic device (210) is not present in an area (1000) corresponding to a field of view of a user wearing the first electronic device (200), as in FIG. 10A, the processor (300) may determine to link the augmented reality service with the second electronic device (210).

According to various embodiments, when an object related to a second electronic device (e.g., the second electronic device (210) of FIG. 2) is included in an area corresponding to a field of view of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'Yes' in operation 1201), the first electronic device (e.g., the processor (120 or 300)) may determine not to link the augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2)) in operation 1205. According to one embodiment, as shown in FIG. 13, when an object (1310) related to the second electronic device (210) exists in an area (1300) corresponding to a field of view of a user wearing the first electronic device (200), the processor (300) may determine not to link the augmented reality service with the second electronic device (210).

FIG. 14 is a flowchart illustrating a method for linking a first electronic device with a second electronic device based on location information according to various embodiments. According to one embodiment, the operations of FIG. 14 may be detailed operations of operation 505 of FIG. 5 . In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3 .

Referring to FIG. 14, according to various embodiments, a first electronic device (e.g., a processor (120) of FIG. 1 or a processor (300) of FIG. 3) may, in operation 1401, check location information of a first electronic device (e.g., a second electronic device (210) of FIG. 2) connected to a communication link (or communication channel) with the first electronic device (e.g., a first electronic device (200) of FIG. 2). According to one embodiment, the processor (300) may periodically receive location information of the second electronic device (210) from the second electronic device (210) through a communication circuit (330). According to one embodiment, the processor (300) may receive changed location information of the second electronic device (210) from the second electronic device (210) through the communication circuit (330) at a time when the location of the second electronic device (210) changes.

According to various embodiments, a first electronic device (e.g., processor (120 or 300)) may, in operation 1403, determine whether a second electronic device (e.g., the second electronic device (210) of FIG. 2) is included within a reference range of the first electronic device (e.g., the first electronic device (200) of FIG. 2). For example, the reference range may be set based on location information of the first electronic device (200).

According to various embodiments, when a second electronic device (e.g., the second electronic device (210) of FIG. 2) is included within a reference range of the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'Yes' of operation 1403), the first electronic device (e.g., the processor (120 or 300)) may determine, in operation 1405, to link an augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2)).

According to various embodiments, if a second electronic device (e.g., the second electronic device (210) of FIG. 2) is not included within a reference range of the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'No' in operation 1403), the first electronic device (e.g., the processor (120 or 300)) may determine, in operation 1407, not to link the augmented reality service with the second electronic device (e.g., the second electronic device (210) of FIG. 2)).

FIG. 15 is a flowchart for processing input information provided from a second electronic device in a first electronic device according to various embodiments. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 15 will be described with reference to FIGS. 16A to 16C, FIGS. 17A to 17C, FIGS. 18A to 18B, FIGS. 19A to 19B, and FIGS. 20A to 20B. FIGS. 16A to 16C are screen configurations of the first electronic device for displaying a functional UI based on input information provided from the second electronic device according to various embodiments. FIGS. 17A to 17C are screen configurations of a first electronic device for displaying a manipulation UI based on input information provided from a second electronic device according to various embodiments. FIGS. 18A and 18B are screen configurations of a first electronic device for controlling video content based on input information provided from a second electronic device according to various embodiments. FIGS. 19A and 19B are screen configurations of a first electronic device for providing guide information according to various embodiments. FIGS. 20A and 20B are screen configurations of a first electronic device for displaying a function UI based on the shape of a second electronic device according to various embodiments.

Referring to FIG. 15, according to various embodiments, a first electronic device (e.g., the processor (120) of FIG. 1 or the processor (300) of FIG. 3) may determine, in operation 1501, whether to link an augmented reality service with a second electronic device (e.g., the second electronic device (210) of FIG. 2) connected to a communication link (or communication channel). According to one embodiment, the processor (300) may determine whether to link the augmented reality service with the second electronic device (210) based on a maintenance time of a user's gaze corresponding to an active object, as in operations 701 to 709 of FIG. 7. According to one embodiment, the processor (300) may determine whether to link the augmented reality service with the second electronic device (210) based on whether the user's gaze gazes at an object related to the second electronic device (210), as in operations 901 to 907 of FIG. 9. For example, when providing an augmented reality service, the processor (300) may control the display device (320) to display a virtual object (e.g., a playback screen) (1610) related to an application (e.g., a music application) running on the first electronic device (200) in at least a portion of an area (1600) corresponding to the field of view of a user wearing the first electronic device (200), as shown in FIG. 16A. When the user's gaze gazes at the virtual object (1610), the processor (300) may set the second electronic device (210) as an input device for the augmented reality service. According to one embodiment, the processor (300) may determine whether to link the augmented reality service with the second electronic device (210) based on whether an object related to the second electronic device (210) is included in an area corresponding to the field of view of a user wearing the first electronic device (200), as shown in operations 1201 to 1205 of FIG. 12. According to one embodiment, the processor (300) may determine whether to link the augmented reality service with the second electronic device (210) based on the location information of the first electronic device (200) and the second electronic device (210), as in operations 1401 to 1407 of FIG. 14.

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) determines that an augmented reality service is to be linked with a second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 1501), in operation 1503, it may determine whether input information is received from the second electronic device (e.g., the second electronic device (210) of FIG. 2)).

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) receives input information from a second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 1503), in operation 1505, the first electronic device may perform a function corresponding to the input information received from the second electronic device (e.g., the second electronic device (210) of FIG. 2). In one embodiment, when the processor (300) receives information related to a tap input (1622) from the second electronic device (210), as shown in FIG. 16b, the processor (300) may control the display device (320) to display a function UI (1630) of a virtual object (1610) corresponding to the user's gaze. For example, the function UI (1630) may be displayed at least partially overlapping or adjacent to the virtual object (1610). For example, the function UI (1630) may display a cursor on a first menu (1631) (e.g., a cancel menu) among a plurality of menus corresponding to a virtual object (1610). According to one embodiment, when the processor (300) receives information of a touch movement input (1624) (e.g., a drag) in a third direction (e.g., a left diagonal direction) from the second electronic device (210) while the function UI (1630) is displayed, as shown in FIG. 16C, the processor (300) may move the cursor displayed on the function UI (1630) in the third direction (e.g., a left diagonal direction). For example, the cursor of the function UI (1630) may be displayed on a fifth menu (1632) (e.g., a settings menu) among a plurality of menus corresponding to the virtual object (1610) included in the function UI (1630) due to movement in the third direction (e.g., a left diagonal direction). According to one embodiment, when the processor (300) receives information of a selection input (e.g., tap input) from the second electronic device (210) while the function UI (1630) is displayed, as in FIG. 16b or FIG. 16c, the processor (300) may perform an operation corresponding to the menu (1631 or 1632) on which the cursor is displayed (e.g., removing the function UI or displaying a settings menu).

According to various embodiments, when the first electronic device (e.g., the processor (120 or 300)) determines that the augmented reality service is not linked with the second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'No' in operation 1501) or input information is not received from the second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'No' in operation 1503), the first electronic device (e.g., the first electronic device (200) of FIG. 2)) may control (or manage) the augmented reality service based on input information detected by the first electronic device.

According to various embodiments, when the first electronic device (200) determines that the second electronic device (210) is linked to provide an augmented reality service, the first electronic device (200) may control the display form (e.g., size, position, or angle) of a virtual object displayed on the display device (320) based on input information provided from the second electronic device (210). In one embodiment, when providing an augmented reality service, the processor (300) may control the display device (320) to display at least one virtual object (1710) in at least a portion of an area (1700) corresponding to a field of view of a user wearing the first electronic device (200), as shown in FIG. 17A. For example, the user wearing the first electronic device (200) may check an image object (1710) displayed in at least a portion of a real space through an area (1700) corresponding to a field of view of the user wearing the first electronic device (200), as shown in FIG. 17A. For example, when the user's gaze is directed at the image object (1710), gaze information (1702) may be displayed on the image object (1710). According to one embodiment, when the processor (300) receives information of a multi-tap input (1722) from the second electronic device (210) at the time when the user's gaze is directed at the image object (1710), as shown in FIG. 17A, the processor (300) may control the display device (320) to enlarge the size of the image object (1710) as shown in FIG. 17B (1730). For example, the size of the image object (1710) may be enlarged based on a predefined ratio or a ratio corresponding to the multi-tap input (1722). According to one embodiment, when the processor (300) receives movement information (1724) of the second electronic device (210) in a first direction (e.g., leftward direction) from the second electronic device (210) while the user's gaze is focused on the image object (1710), as shown in FIG. 17c, the processor (300) may control the display device (320) to move the position of the image object (1710) in the first direction (e.g., leftward direction) (1740). For example, the movement distance of the image object (1710) may be set based on the movement distance of the electronic device (210). For example, the processor (300) may control the display device (320) to move the position of the image object (1710) based on the movement information (1724) of the second electronic device (210) while the multi-tap input (1722) is maintained.

According to various embodiments, when the first electronic device (200) determines that the second electronic device (210) is linked to provide an augmented reality service, the first electronic device (200) may control the operation of an application (e.g., a video application) running on the first electronic device (200) based on input information provided from the second electronic device (210). In one embodiment, when providing an augmented reality service, the processor (300) may control the display device (320) to display an execution screen of an application played on the first electronic device (200) in an area (1800) corresponding to a field of view of a user wearing the first electronic device (200), as shown in FIG. 18A. According to one embodiment, when the processor (300) receives information of a tap input (1812) from the second electronic device (210) while the execution screen of the application is displayed on the display device (320), as shown in FIG. 18A, the processor (300) may control the display device (320) to display a function UI (or control menu) (1820) of the application. For example, the function UI (1820) may display a cursor on a sixth menu (e.g., a playback menu) (1822) among a plurality of menus included in the function UI (1820). According to one embodiment, when the processor (300) receives information of a touch movement input (1814) (e.g., drag) in a first direction (e.g., leftward) from the second electronic device (210) while the function UI (or control menu) (1820) of the application is displayed as in FIG. 18a, the processor (300) may move the cursor displayed on the function UI (1820) in the first direction (e.g., leftward), as in FIG. 18b. For example, the cursor of the function UI (1820) may be displayed on a seventh menu (1824) (e.g., backward menu) included in the function UI (1820) due to movement in the first direction (e.g., leftward). According to one embodiment, when the user gazes at the seventh menu (1824) while the function UI (or control menu) (1820) of the application is displayed as in FIG. 18a, the processor (300) may control the display device (320) to display the cursor displayed on the function UI (1820) on the seventh menu (1824) (e.g., a back menu). According to one embodiment, when the processor (300) receives information on a selection input (e.g., a tap input) from the second electronic device (210) while the function UI (1820) is displayed as in FIG. 18a or FIG. 18b, the processor (300) may perform an operation (e.g., play or back) corresponding to the menu (1822 or 1824) on which the cursor is displayed.

According to various embodiments, when the first electronic device (200) determines that the second electronic device (210) is linked to provide an augmented reality service, the first electronic device (200) may provide guide information corresponding to a user input. In one embodiment, when providing an augmented reality service, the processor (300) may control the display device (320) to display at least one virtual object in an area (1900) corresponding to the field of view of a user wearing the first electronic device (200), as shown in FIG. 19A or FIG. 19B . For example, the display device (320) may display at least one virtual object (1910) arranged horizontally, as shown in FIG. 19A . For example, the display device (320) may display at least one virtual object (1930) arranged in a grid pattern, as shown in FIG. 19B . According to one embodiment, the processor (300) may control the display device (320) to display user gaze information in a form corresponding to the form in which the at least one virtual object is displayed. For example, the display device (320) may display user gaze information (1920) in a form indicating that movement to the left/right is possible when at least one object (1910) is arranged horizontally, as shown in FIG. 19a. For example, the display device (320) may display user gaze information (1940) in a form indicating that movement to the up/down/left/right is possible when at least one object (1930) is arranged in a grid form, as shown in FIG. 19b. As an example, the display device (320) may display the size of an object corresponding to a cursor among at least one virtual object displayed in an area (1900) corresponding to the field of view of a user wearing the first electronic device (200) relatively large, or may display the depth differently.

According to various embodiments, the first electronic device (200) may set a different function UI based on motion information (e.g., direction) of the second electronic device (210). In one embodiment, a user wearing the first electronic device (200) may check an object (2002) in a real space through an area (2000) corresponding to a field of view of the user wearing the first electronic device (200), as shown in FIG. 20A. For example, the object (2002) may correspond to an external device supporting an Internet of Things service. When the processor (300) receives input information (e.g., tap input) from the second electronic device (210) at a time when the user gazes at the object (2002), the processor (300) may control the display device (320) to display a control menu (e.g., function UI) corresponding to the object (2002). For example, the display device (320) may display a control menu (2020) corresponding to a relatively simply configured object (2002) when the second electronic device (210) is in a portrait orientation (2010), as shown in FIG. 20A. For example, the control menu (2020) may include a switch menu and/or a brightness adjustment menu. For example, the display device (320) may display a control menu (2040) corresponding to a relatively specifically configured object (2002) when the second electronic device (210) is in a landscape orientation (2030), as shown in FIG. 20B. For example, the control menu (2040) may include a menu for adjusting the color of the object (2002).

FIG. 21 is a flowchart illustrating a method for interfacing a first electronic device with a second electronic device according to various embodiments. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 4.

Referring to FIG. 21, according to various embodiments, a second electronic device (e.g., the processor (120) or the communication module (190) of FIG. 1 or the processor (400) or the communication circuit (410) of FIG. 4) may connect a communication link (or communication channel) with a first electronic device (e.g., the first electronic device (200) of FIG. 2) in operation 2101. According to one embodiment, when user authentication corresponding to the communication link connection between the first electronic device (200) and the second electronic device (210) is successful, the processor (400) may control the communication module (410) to establish a communication link (or communication channel) with the first electronic device (200). For example, the communication link (or communication channel) may be established based on a short-range communication method such as Bluetooth, BLE, or wireless LAN (e.g., WiFi direct).

According to various embodiments, the second electronic device (e.g., processor (120 or 400)) may determine whether input information is detected in operation 2103. For example, the input information may include at least one of user input information detected through the input device (420) and/or an operating state (e.g., movement information) of the second electronic device (210) detected through the sensor module (430).

According to various embodiments, the second electronic device (e.g., processor (120 or 400)) may determine whether input information is detected if no input information is detected (e.g., 'No' in operation 2103).

According to various embodiments, when input information is detected (e.g., 'Yes' in operation 2103), the second electronic device (e.g., processor (120 or 400)) may, in operation 2105, determine whether the input mode of the second electronic device (e.g., the second electronic device (210) of FIG. 2) is set to the first input mode. For example, the first input mode may include an input mode for sharing input information with the first electronic device (200).

According to various embodiments, the second electronic device (e.g., the processor (120 or 400)) may transmit input information to the first electronic device (e.g., the first electronic device (200) of FIG. 2) that links the augmented reality service when the first input mode is set (e.g., 'Yes' in operation 2105) in operation 2107. For example, the input information transmitted to the first electronic device (200) may include command information (e.g., tap, multi-tap, drag, long touch, or double tap) corresponding to an input detected by the second electronic device (210) or raw data detected to recognize the command information.

According to various embodiments, the second electronic device (e.g., processor (120 or 400)) may perform a function (or operation) corresponding to the input information in operation 2109 when the first input mode is not set (e.g., 'No' in operation 2105). According to one embodiment, when the processor (400) detects a tap input (1120) as in FIG. 11b, the processor (400) may control the display device (440) to enlarge and display (1130) an image object selected by the tap input (1120) as in FIG. 11c.

FIG. 22 is a flowchart for linking a second electronic device with a first electronic device based on control of the first electronic device according to various embodiments. In the embodiments below, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 4. As an example, at least a portion of the configuration of FIG. 22 will be described with reference to FIGS. 23 and 24. FIG. 23 is an example for changing a display method based on linkage with a first electronic device in a second electronic device according to various embodiments. FIG. 24 is an example for changing content displayed on a display device based on linkage with a first electronic device in a second electronic device according to various embodiments.

Referring to FIG. 22, according to various embodiments, a second electronic device (e.g., the processor (120) or the communication module (190) of FIG. 1 or the processor (400) or the communication circuit (410) of FIG. 4) may, in operation 2201, determine whether linkage information is received from a first electronic device (e.g., the first electronic device (200) of FIG. 2) with which a communication link (or communication channel) is established. According to one embodiment, when the first electronic device (200) determines that the augmented reality service is linked with the second electronic device (210), as in operations 501 to 507 of FIG. 5, the first electronic device (200) may transmit linkage information related to the augmented reality service to the second electronic device (210). Accordingly, the processor (400) may determine whether linkage information related to the augmented reality service is received from the first electronic device (200) through the communication circuit (410).

According to various embodiments, when the second electronic device (e.g., the processor (120 or 400)) receives linking information from the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'Yes' in operation 2201), in operation 2203, the second electronic device (e.g., the second electronic device (210) of FIG. 2) may switch the input mode of the second electronic device to the first input mode. For example, the first input mode may include an input mode for sharing the input of the second electronic device (210) with the first electronic device (200) for linking an augmented reality service.

According to various embodiments, the second electronic device (e.g., the processor (120 or 400)) may change the display information and/or display method of the display device (e.g., the display device (440) of FIG. 4) based on the switching to the first input mode in operation 2205. According to one embodiment, when the input mode of the second electronic device (210) is switched to the first input mode, the processor (400) may recognize that the second electronic device (210) is linked with the first electronic device (200) for an augmented reality service. Accordingly, the processor (400) may control the display device (440) to change the display information and/or display method of the display device (440) for the augmented reality service. For example, the display device (440) may display an execution screen (e.g., an image list) (2300) of an application (e.g., a gallery application) running on the second electronic device (210), as shown in FIG. 23 (a). For example, when the input mode of the second electronic device (201) is switched to the first input mode, the processor (400) can control the display device (440) so that the brightness of the display device (440) is changed to be relatively dark, as shown in (b) of FIG. 23. For example, when the input mode of the second electronic device (201) is switched from the first input mode to the second input mode, the processor (400) can control the display device (440) so that the brightness of the display device (440) is restored, as shown in (a) of FIG. 23. For example, the display device (440) can display the standby screen (2400) of the second electronic device (210), as shown in (a) of FIG. 24. For example, when the input mode of the second electronic device (201) is switched to the first input mode, the processor (400) can control the display device (440) to display content (e.g., game information) (2410) corresponding to the augmented reality service, as shown in (b) of FIG. 24. For example, when the input mode of the second electronic device (201) is switched from the first input mode to the second input mode, the processor (400) can control the display device (440) to display a standby screen (2400), as shown in (a) of FIG. 24.

According to various embodiments, when the second electronic device (e.g., the processor (120 or 400)) does not receive linking information from the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'No' in operation 2201), in operation 2207, the second input mode of the second electronic device (e.g., the second electronic device (210) of FIG. 2) may be maintained. In one embodiment, the processor (400) may set the second input mode to a default input mode of the second electronic device (210), and when linking information related to an augmented reality service is received from the first electronic device (200), the input mode of the second electronic device (210) may be switched to the first input mode.

FIG. 25 is a flowchart illustrating a method for determining a link between a second electronic device and a first electronic device according to various embodiments. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 4.

Referring to FIG. 25, according to various embodiments, a second electronic device (e.g., the processor (120) or the communication module (190) of FIG. 1 or the processor (400) or the communication circuit (410) of FIG. 4) may receive information related to an augmented reality service from a first electronic device (e.g., the first electronic device (200) of FIG. 2) with which a communication link (or communication channel) is established, in operation 2501. According to one embodiment, the processor (400) may receive information related to an augmented reality service provided by the first electronic device (200) from the first electronic device (200) through the communication circuit (410). For example, the information related to the augmented reality service may include at least one of object information included in an area corresponding to a field of view of a user wearing the first electronic device (200), gaze information of a user wearing the first electronic device (200), or location information of the first electronic device (200).

According to various embodiments, the second electronic device (e.g., the processor (120 or 400)) may determine whether to link with the first electronic device (e.g., the first electronic device (200) of FIG. 2) an augmented reality service based on information related to the augmented reality service in operation 2503. According to one embodiment, the processor (400) may determine whether to link with the first electronic device (200) based on at least one of: a maintenance time of a user's gaze corresponding to an object included in an area corresponding to a field of view of a user wearing the first electronic device (200), whether an object related to the second electronic device (210) is included in an area corresponding to the field of view of a user wearing the first electronic device (200), or user gaze information corresponding to an object related to the second electronic device (210). According to one embodiment, the processor (400) may determine whether to link the first electronic device (200) with an augmented reality service based on a comparison result of the location information of the first electronic device (200) and the second electronic device (210).

According to various embodiments, when the second electronic device (e.g., the processor (120 or 400)) determines that the first electronic device (e.g., the first electronic device (200) of FIG. 2) is linked to an augmented reality service (e.g., 'Yes' in operation 2503), in operation 2505, the second electronic device (e.g., the second electronic device (210) of FIG. 2)) may switch the input mode of the second electronic device to the first input mode.

According to various embodiments, the second electronic device (e.g., the processor (120 or 400)) may change the display information and/or display method of the display device (e.g., the display device (440) of FIG. 4) based on the transition to the first input mode in operation 2507. According to one embodiment, when the input mode of the second electronic device (210) is switched to the first input mode, the processor (400) may control the display device (440) to be switched to an inactive mode so that the user can focus on information related to augmented reality.

According to various embodiments, if the second electronic device (e.g., the processor (120 or 400)) determines that the first electronic device (e.g., the first electronic device (200) of FIG. 2) is not to be linked with the augmented reality service (e.g., 'No' in operation 2503), the second input mode of the second electronic device (e.g., the second electronic device (210) of FIG. 2) may be maintained in operation 2509.

According to various embodiments, the second electronic device (210) may set the input mode of the second electronic device (210) based on the gaze information of the user wearing the first electronic device (200) at the time when the input information is detected. According to one embodiment, when the input information is detected, the second electronic device (210) may check the gaze information of the user wearing the first electronic device (200). When the second electronic device (210) determines that the gaze of the user wearing the first electronic device (200) is gazing at an active object, the second electronic device (210) may set the input mode of the second electronic device (210) to the first input mode. For example, the second electronic device (210) may check the gaze information of the user wearing the first electronic device (200) based on information related to an augmented reality service periodically received from the first electronic device (200). For example, when input information is detected, the second electronic device (210) can request the user's gaze information from the first electronic device (200) to confirm the gaze information of the user wearing the first electronic device (200).

FIG. 26 is a flowchart for providing IoT services in a first electronic device according to various embodiments. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the first electronic device (200) of FIG. 3. For example, at least a portion of the configuration of FIG. 26 will be described with reference to FIGS. 27A and 27B. FIGS. 27A and 27B illustrate screen configurations of a first electronic device for registering an IoT device according to various embodiments.

Referring to FIG. 26, according to various embodiments, a first electronic device (e.g., a processor (120) or a communication module (190) of FIG. 1 or a processor (300) or a communication circuit (330) of FIG. 3) may connect a communication link (or a communication channel) with a second electronic device (e.g., a second electronic device (210) of FIG. 2) in operation 2601. For example, user identification information of the first electronic device (200) and the second electronic device (210) may be the same. For example, the first electronic device (200) and the second electronic device (210) may successfully receive authentication related to establishing a communication link from a user.

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may output information related to an augmented reality service in operation 2603. According to one embodiment, the processor (300) may control the display device (320) to display at least one virtual object in an area corresponding to a field of view of a user wearing the first electronic device (200). For example, the user wearing the first electronic device (200) may recognize an object (2710) related to the augmented reality service through an area (2700) corresponding to a field of view of the user wearing the first electronic device (200), as illustrated in FIG. 27A.

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may, in operation 2605, determine whether an unregistered device is recognized in an area corresponding to the field of view of a user wearing the first electronic device (e.g., the first electronic device (200) of FIG. 2). According to one embodiment, the processor (300) may determine whether an object related to an Internet of Things device exists in an area corresponding to the field of view of a user wearing the first electronic device (200) through at least one of a camera module (310) or a sensor module (340) (e.g., a spatial sensor). When the processor (300) detects an object related to an Internet of Things device (2710) in an area corresponding to the field of view of a user wearing the first electronic device (200), the processor (300) may determine whether the corresponding Internet of Things device (2710) is registered with the first electronic device (200). For example, the processor (300) can check whether registration information of an Internet of Things device (2710) is stored in a memory (e.g., memory (130) of FIG. 1).

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may provide an augmented reality service if an unregistered device is not recognized (e.g., 'No' in operation 2605).

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) recognizes an unregistered device (e.g., 'yes' in operation 2605), in operation 2607, the first electronic device may transmit information related to the unregistered device to a second electronic device (e.g., the second electronic device (210) of FIG. 2). For example, the information related to the unregistered device may include the type of device (e.g., television) obtained through image analysis and/or location information of the first electronic device (200) (e.g., living room).

According to various embodiments, the first electronic device (e.g., the processor (120 or 300)) may determine whether a registration request signal is received from a second electronic device (e.g., the second electronic device (210) of FIG. 2) at operation 2609. According to one embodiment, the second electronic device (210) may determine an unregistered device recognized by the first electronic device (200) based on information related to the unregistered device. The second electronic device (210) may determine whether the unregistered device (2710) of the first electronic device (200) is registered with the second electronic device (210). If the unregistered device (2710) of the first electronic device (200) is not registered with the second electronic device (210), the second electronic device (210) may transmit a registration request signal to the first electronic device (200). Accordingly, the processor (300) can confirm whether a registration request signal is received from the second electronic device (210) through the communication circuit (410).

According to various embodiments, when a first electronic device (e.g., processor (120 or 300)) receives a registration request signal from a second electronic device (e.g., second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 2609), in operation 2611, the first electronic device may display a registration screen related to an unregistered device. According to one embodiment, when the processor (300) receives a registration request signal from the second electronic device (210), the processor (300) may control the display device (320) to display a registration screen (2720) corresponding to an object (2710) related to an Internet of Things device, as illustrated in FIG. 27b.

According to various embodiments, the first electronic device (e.g., processor (120 or 300)) may determine, at operation 2613, whether to register an unregistered device with the first electronic device (e.g., the first electronic device (200) of FIG. 2) based on an input corresponding to a registration screen associated with the unregistered device.

According to various embodiments, when the first electronic device (e.g., the processor (120 or 300)) determines that an unregistered device is to be registered in the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'YES' in operation 2613), in operation 2615, information related to the unregistered device may be registered in the first electronic device (e.g., the first electronic device (200) of FIG. 2). According to one embodiment, when the processor (300) detects a selection input of a 'YES' menu (2722) in the registration screen (2720) of FIG. 27b, the processor (300) may determine that an Internet of Things device (2710) is to be registered. In this case, the processor (300) may register information related to the Internet of Things device (2710) in the memory of the first electronic device (200) (e.g., the memory (130) of FIG. 1). For example, when an Internet of Things device (2710) is recognized through a camera module (310), the processor (300) can control the display device (320) to display a virtual object (e.g., a control menu or status information) related to the Internet of Things device.

According to various embodiments, when a registration request signal is not received from a second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'No' in operation 2609), the first electronic device (e.g., the processor (120 or 300)) may determine whether registration information related to an unregistered device is received in operation 2617. According to one embodiment, when an unregistered device (2710) of the first electronic device (200) is registered with the second electronic device (210), the second electronic device (210) may transmit registration information related to the unregistered device (2710) to the first electronic device (200). Accordingly, the processor (300) may determine whether registration information related to the unregistered device (2710) is received from the second electronic device (210) through the communication circuit (410).

According to various embodiments, if the first electronic device (e.g., processor (120 or 300)) does not receive registration information related to an unregistered device (e.g., 'No' in operation 2617), it may determine whether a registration request signal related to an unregistered device is received.

According to various embodiments, when the first electronic device (e.g., processor (120 or 300)) receives registration information related to an unregistered device (e.g., 'yes' in operation 2617), the first electronic device may display information related to the unregistered device in operation 2619. According to one embodiment, when the processor (300) receives registration information related to an Internet of Things device (2710) from the second electronic device (210), the processor (300) may register the Internet of Things device (2710) to the first electronic device (200) at a location of the first electronic device (200) (e.g., a living room). For example, when the Internet of Things device (2710) is recognized through the camera module (310), the processor (300) may control the display device (320) to display a virtual object (e.g., a control menu or status information) related to the Internet of Things device.

FIG. 28 is a flowchart illustrating a method for providing an IoT service based on a first electronic device from a second electronic device according to various embodiments. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the first electronic device may be the electronic device (101) of FIG. 1 or the second electronic device (210) of FIG. 4.

Referring to FIG. 28, according to various embodiments, a second electronic device (e.g., the processor (120) or the communication module (190) of FIG. 1 or the processor (400) or the communication circuit (410) of FIG. 4) may connect a communication link (or communication channel) with a first electronic device (e.g., the first electronic device (200) of FIG. 2) in operation 2801. According to one embodiment, the first electronic device (200) and the second electronic device (210) may establish a communication link (or communication channel) based on a short-range communication method such as Bluetooth, BLE, or wireless LAN (e.g., WiFi direct).

According to various embodiments, the second electronic device (e.g., processor (120 or 400)) may, at operation 2803, determine whether information related to an unregistered device is received from the first electronic device (e.g., the first electronic device (200) of FIG. 2) with which a communication link (or communication channel) is established.

According to various embodiments, when a second electronic device (e.g., the processor (120 or 400)) receives information related to an unregistered device from a first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'Yes' in operation 2803), in operation 2805, the processor (400) may determine whether the unregistered device is registered in the second electronic device (e.g., the second electronic device (210) of FIG. 2). According to one embodiment, the processor (400) may determine the type of the unregistered device recognized by the first electronic device (200) and the location information of the first electronic device (200) based on the information related to the unregistered device provided from the first electronic device (200). The processor (400) may determine whether the unregistered device of the first electronic device (200) is included in a list of devices registered at the location of the first electronic device (200).

According to various embodiments, if the unregistered device is not registered with the second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'No' in operation 2805), the second electronic device (e.g., the processor (120 or 400)) may transmit a registration request signal corresponding to the unregistered device to the first electronic device (e.g., the first electronic device (200) of FIG. 2) in operation 2807.

According to various embodiments, the second electronic device (e.g., processor (120 or 400)) may, at operation 2809, determine whether registration information related to an unregistered device is received from the first electronic device (e.g., the first electronic device (200) of FIG. 2).

According to various embodiments, when the second electronic device (e.g., the processor (120 or 400)) receives registration information related to an unregistered device from the first electronic device (e.g., the first electronic device (200) of FIG. 2) (e.g., 'Yes' in operation 2809), the second electronic device may register the unregistered device to the second electronic device (e.g., the second electronic device (210) of FIG. 2)) in operation 2811. According to one embodiment, when the processor (400) receives registration information related to an unregistered Internet of Things device from the first electronic device (200), the processor (400) may register the Internet of Things device (e.g., a television) at the location (e.g., a living room) of the first electronic device (200).

According to various embodiments, if an unregistered device is registered with the second electronic device (e.g., the second electronic device (210) of FIG. 2) (e.g., 'Yes' in operation 2805), the second electronic device (e.g., the processor (120 or 400)) may transmit information related to the unregistered device to the first electronic device (e.g., the first electronic device (200) of FIG. 2) in operation 2813.

According to various embodiments, when the second electronic device (210) receives information related to an unregistered device, it can check the information related to the unregistered device. According to one embodiment, when the processor (400) receives information related to an unregistered device from the first electronic device (200) through the communication circuit (410), it can check the information of the unregistered device through the memory of the second electronic device (210) (e.g., the memory (130) of FIG. 1) or an external server (e.g., the server (108) of FIG. 1). The processor (400) can control the communication circuit (410) to transmit the information of the unregistered device to the first electronic device (200). According to one embodiment, when the first electronic device (200) receives information of the unregistered device from the second electronic device (210), it can control the display device (320) to output the information of the unregistered device.

The embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the various embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical ideas of the various embodiments of the present invention in addition to the embodiments disclosed herein.

Claims (20)

In the first electronic device,
communication circuit,
camera module,
sensor module,
display device,
A processor operatively connected to the communication circuit, the display device, the sensor module and the camera module; and
Contains memory for storing commands,
The above instructions, when executed by the processor, cause the first electronic device to:
Connecting communication with a second electronic device through the above communication circuit,
Obtaining at least one object information included in an area corresponding to the field of view of a user wearing the first electronic device through at least one of the camera module or the sensor module,
Obtain the user's gaze information,
Using the user's gaze information corresponding to at least one object information, determine whether to use the second electronic device as an input device for an augmented reality service,
A first electronic device that transmits linkage information of the augmented reality service to the second electronic device through the communication circuit when the second electronic device is determined to be used as an input device for an augmented reality service.
In paragraph 1,
The above commands, when executed by the processor, cause the first electronic device to obtain gaze information of the user based on at least one of a head gaze or an eye gaze of the first electronic device.
In paragraph 1,
The above commands, when executed by the processor, cause the first electronic device to display, through the display device, at least one virtual object in at least a portion of an area corresponding to a field of view of a user wearing the first electronic device.
In the third paragraph,
The above instructions, when executed by the processor, determine that the second electronic device is used as an input device for an augmented reality service when the user's gaze corresponding to at least one object included in an area corresponding to the field of view of the user wearing the first electronic device or the at least one virtual object is continuously maintained for a reference time,
A first electronic device that determines that the second electronic device is not to be used as an input device for an augmented reality service if the user's gaze corresponding to any of the above is not maintained for the reference time.
In paragraph 1,
The above instructions, when executed by the processor, cause the first electronic device to determine whether the user's gaze is directed at an object related to the second electronic device if an object related to the second electronic device exists in an area corresponding to a field of view of a user wearing the first electronic device;
If the user's gaze is directed at an object different from the object associated with the second electronic device, the second electronic device is determined to be used as an input device for an augmented reality service.
A first electronic device that determines that the second electronic device is not being used as an input device for an augmented reality service when the user's gaze is directed at an object related to the second electronic device.
In paragraph 1,
The above instructions, when executed by the processor, determine whether input information is received from the second electronic device through the communication circuit when the first electronic device determines to use the second electronic device as an input device for an augmented reality service;
A first electronic device that, when receiving input information from the second electronic device, performs an action corresponding to the input information.
In paragraph 6,
A first electronic device, wherein the input information received from the second electronic device includes at least one of a touch input, a physical button input, or movement information of the second electronic device.
In paragraph 6,
The above commands, when executed by the processor, cause the first electronic device to, when receiving input information from the second electronic device, identify an object corresponding to the user's gaze,
A first electronic device that displays a control menu related to an object corresponding to the user's gaze through the display device.
In the second electronic device,
communication circuit,
input devices,
a processor operatively connected to the above communication circuit and the above input device; and
Contains memory for storing commands,
The above instructions, when executed by the processor, cause the second electronic device to:
Connecting communication with the first electronic device through the above communication circuit,
When linking information related to the augmented reality service is received from the first electronic device through the communication circuit, the input mode of the second electronic device is set to the first mode,
If linkage information related to the augmented reality service is not received from the first electronic device through the communication circuit, or if non-linkage information is received, the input mode of the second electronic device is set to the second mode,
When an input is detected through the above input device, the input mode of the second electronic device is checked,
When the input mode is the first mode, information related to the input is transmitted to the first electronic device through the communication circuit,
A second electronic device that performs an operation corresponding to information related to the input when the input mode is the second mode.
delete In paragraph 9,
The display device further includes:
The above commands, when executed by the processor, cause the second electronic device to change the display information and/or display method of the display device when the second electronic device receives linkage information related to the augmented reality service from the first electronic device through the communication circuit.
In paragraph 9,
The input device is a second electronic device including at least one of a touch circuit or a physical button.
In the second electronic device,
communication circuit,
display device,
A processor operatively connected to the above communication circuit and the above display device, and
Contains memory for storing commands,
The above instructions, when executed by the processor, cause the second electronic device to:
Receive at least one object information included in an area corresponding to the field of view of a user wearing the first electronic device from the first electronic device through the communication circuit,
Based on the at least one object information, it is determined whether an object related to the second electronic device exists in an area corresponding to the field of view of the user wearing the first electronic device,
When an object related to the second electronic device exists in an area corresponding to the field of view of a user wearing the first electronic device, it is confirmed that the user's gaze is fixed on the object related to the second electronic device;
A second electronic device that is determined to be used as an input device for an augmented reality service of the first electronic device when the user's gaze gazes at an object different from an object associated with the second electronic device.
In paragraph 13,
The above commands, when executed by the processor, cause the second electronic device to obtain gaze information of the user based on at least one of a head gaze or an eye gaze of the first electronic device.
In paragraph 13,
A second electronic device, wherein the commands, when executed by the processor, control the first electronic device to display at least one virtual object in at least a portion of an area corresponding to a field of view of a user wearing the first electronic device.
In paragraph 13,
The above commands, when executed by the processor, determine that the second electronic device is used as an input device for an augmented reality service of the first electronic device when no object related to the second electronic device exists in an area corresponding to the field of view of a user wearing the first electronic device.
In paragraph 13,
The above commands, when executed by the processor, cause the second electronic device to determine that the second electronic device is not used as an input device for an augmented reality service of the first electronic device when the user's gaze is directed at an object associated with the second electronic device.
In paragraph 13,
The above instructions, when executed by the processor, transmit information related to an operation corresponding to input information acquired by the second electronic device to the first electronic device when the second electronic device is determined to be used as an input device for an augmented reality service of the first electronic device.
In paragraph 18,
A second electronic device, wherein the input information obtained from the second electronic device includes at least one of a touch input, a physical button input, or movement information of the second electronic device.
In paragraph 18,
The above commands, when executed by the processor, cause the second electronic device to, when detecting the input information, identify an object corresponding to the user's gaze based on the at least one object information,
A second electronic device that transmits a control menu related to an object corresponding to the user's gaze to the first electronic device.