CN109641150B - Method for creating virtual objects - Google Patents

Abstract

A computer-implemented method for creating a virtual object; the method comprising: receiving a digital representation of a visual appearance of a real-world object; creating a virtual object having the visual appearance based on the received digital representation; selecting a portion of the virtual object; or multiple properties assigned to the selected part of the virtual object.

Description

Methods for creating dummy objects

technical field

The present invention relates to a computer-implemented method of creating virtual objects based on digital representations of real-world objects, such as toy construction models constructed from toy construction elements of a toy construction system.

Background technique

Toy construction systems have been known for decades. Over the years, simple box-shaped building blocks have been supplemented with other building elements with specific looks or mechanical or electrical functions to enhance play value. These functions include, for example, motors, switches, and lights, and programmable processors that can accept input from sensors and activate functional elements in response to received sensor inputs.

Several attempts have been made to control virtual games through real-world physical toys.

For example, US2011/298922 discloses a system for extracting images of physical objects. The extracted images may be digitally represented on a display device as part of a virtual world or video game, where objects prohibiting the virtual world and/or video game are designed and constructed according to constructs in the real world. However, in many video games or other virtual environments, it is desirable to provide three-dimensional virtual objects that experience rich behaviors.

According to at least one aspect, it is therefore desirable to provide a process for creating three-dimensional virtual objects that exhibit rich functional behavior from real-world objects in a user-friendly manner. In particular, it is desirable to provide a method in which the user is not required to go through numerous steps to configure and modify the resulting virtual object, eg, in which the creation of the virtual object is performed in an at least partially automated manner, requiring little user interaction.

WO2015/185629 discloses a toy construction system comprising a plurality of toy construction elements and an image capture device operable to capture one or more of a toy construction model constructed from one or more of said toy construction elements image. The prior art system also includes a processor configured to determine one or more visual attribute parameters indicative of corresponding values derivable from one or more of color, shape, and size of the toy construction model visual properties. The processor is also configured to create virtual objects in the computer-generated virtual environment, and to control the virtual objects in the computer-generated virtual environment to have behaviors based on the determined one or more visual property parameters.

According to at least one aspect, it is desirable to provide a method and system that allows for the creation of three-dimensional virtual objects from real-world objects (eg, constructing models from real-world toys) in a user-friendly and reliable and accurate manner. In particular, it is desirable to provide methods and systems that are easy to use and provide 3D representations of virtual objects that are highly functional in virtual environments such as video games. It is also desirable to provide virtual objects that accurately represent the 3D shape of the corresponding real world object.

It is generally desirable to provide a toy construction system that enhances the educational and/or play value of the system. It would also be desirable to provide a toy construction system in which a set of construction elements can be easily used in different toy construction models and/or in combination with existing toy construction elements. Furthermore, it would be desirable to provide a toy construction system that allows a user, especially a child, to construct a plurality of toy figures in a user-friendly efficient and flexible and reliable manner. In particular, it would be desirable to provide a toy construction system that allows a user-friendly and flexible way of creating virtual objects in virtual environments, such as gaming systems.

Various aspects of the invention disclosed herein may address at least some of the disadvantages of prior art systems.

SUMMARY OF THE INVENTION

According to a first aspect, disclosed herein is a computer-implemented method for creating a virtual object; the method comprising:

- obtain a digital representation of the visual appearance of real-world objects;

- creation of virtual objects with a visual appearance based on the received digital representation;

- select part of the virtual object; and

- Assign one or more properties to the selected part of the virtual object.

Thus, one or more captured images of a real-world object or another form of digital representation of the visual appearance of the real-world object may be used by a processor executing the method for generating a user-defined appearance and attributes in a virtual environment The base of virtual objects. For example, a user may create a real-world toy construction model or otherwise select or create an asset-like real-world object for use as a virtual object in a computer-generated virtual environment. The method provides the user with a flexible and easy-to-understand and easy-to-use mechanism for influencing the desired appearance of virtual objects in the virtual environment.

In particular, when properties are assigned to various parts of a virtual object, virtual objects with rich functional behavior can be created. Thus, attributes may relate to positions relative to a 3D representation of a virtual object, eg, a position on a surface of the virtual object. For example, the process may create functional elements of a virtual object, such as movable parts, parts that can be controlled by the user, parts that can change appearance, parts that can interact with other virtual objects or the virtual environment, etc. or have other related A widget with functional properties of the form.

Since the user can construct these objects from toy construction elements, the user has a great deal of freedom in how the objects are constructed.

The digital representation of the visual appearance of the real-world object may be a suitable representation indicative of the three-dimensional shape of the real-world object. In particular, digital representations may represent point clouds and/or surface geometries of real-world objects. The digital representation may also include information about the visible properties of the real-world object, such as one or more colors, surface textures, and the like. For example, the digital representation may be a surface representation, such as in the form of a 3D mesh, or a volumetric representation, such as a voxel-based representation of real-world objects.

The process may receive or create a digital representation from suitable input data, such as data received from one or more sensor devices operable to capture radiation from real-world objects. The sensor device may include one or more sensors that detect light or other forms of electromagnetic radiation, such as light or other electromagnetic radiation reflected by surfaces of real-world objects in the sensor device's field of view. The sensor device may include an array of sensors, such as a CCD chip, or a single sensor operable to scan the field of view, or a combination of multiple sensors that scan. Therefore, a real world object can be passive in that it does not need to actively emit any sound, light, radio signals, electrical signals, etc. Furthermore, the sensor device is operable to capture radiation in a contactless manner without any electrical contact, communication interface, etc. being established between the sensor device and the real world object.

In some embodiments, the sensor device includes an image capture device operable to capture both sides of the real-world object when the real-world object is placed within the field of view of the image capture device (eg, on a suitable object support). one or more images, wherein two or more images are taken from different viewpoints relative to the real-world object. Each image may be a photograph or another form of a two-dimensional representation of the field of view of the image capture device that allows the shape and/or color and/or size of objects within the field of view to be determined. The image may include a 2D array of pixels or other array elements, each array element representing sensed information related to a point or direction within the field of view. Sensing information may include the intensity of the received radiation or wave, the frequency/wavelength of the received radiation or wave, a distance map, a polarization map, a surface normal map, and/or other suitable sensing measures.

Thus, the image capture device may include one or more digital cameras responsive to visible light, infrared light, and the like. For example, the image capture device may comprise two digital cameras adapted to be at respective viewpoints relative to the object support, eg at respective heights relative to the object support. The image capture device may include one or more depth cameras operable to also detect distance information for various points within the field of view relative to the positions of the cameras. Some embodiments of sensor devices may include lasers. In some embodiments, the image capture device is configured to capture depth information in addition to light intensity data, such as RGB data. In some embodiments, the image capture device is configured to capture information indicative of surface normals of one or more surfaces within the field of view of the digital camera. For example, the image capture device may be configured to obtain polarization data of the received light. The image capture device and/or processor may be configured to determine local surface normals from the obtained polarization data. The captured surface normals can also be transformed to the world coordinate system based on the detected tilt or other displacement of the turntable relative to the camera. An example of a camera sensor capable of detecting surface normals includes the system disclosed in US Pat. No. 8,023,724. Other examples of techniques for determining surface normals are included in Wan-Cun Ma et al., "Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination", Eurographics Symposium on Rendering (2007), Jan Kautz and Sumanta Pattanaik (Editors ) described in . The sensor device may output an analog or digital signal indicative of the captured radiation (eg, one or more captured images (eg, as a digital image or other data map)) as a video stream or the like.

The process for creating the digital representation may include multiple stages of a multi-stage process for creating a 3D digital representation based on captured radiation (eg, based on captured images). In some embodiments, the process uses a suitable scheme for sensor pose estimation, eg, based on non-repeating colored patterns arranged on a turntable on which real-world objects are placed. The color pattern can then be imaged macroscopically and/or by distance-to-edge measurement. Other embodiments may use markers.

Thus, the process may include creating from light intensity data and/or from other sensed data (such as depth data, eg in the form of one or more depth maps), from polarization data and/or surface normal data, or a combination thereof 3D digital representation. The process may use structures from motion techniques, space sculpting techniques, or other suitable techniques. If the sensors provide a field of detected surface normals, these can be used to detect marker features, such as edges or corners or other features with abrupt changes in the direction of the surface normal, for structures during motion. In other embodiments, the detected surface normals may be used to transform a voxel representation of an object (eg, obtained by a spatial sculpting process) into an accurate surface mesh. In general, the 3D digital representation may include any suitable type of representation, such as a surface mesh of polygons, a voxel representation, etc., or a combination thereof.

The process creates and associates a three-dimensional (3D) digital representation of the virtual object's appearance with the virtual object to allow the virtual object to be rendered within a virtual environment. The three-dimensional representation may be a digital representation of an obtained real-world object or a three-dimensional representation derived therefrom. For example, some embodiments of the methods disclosed herein may process received digital representations to create digital representations of virtual objects. Examples of this processing may include one or more of the following: changes in spatial resolution, scaling, modification of one or more colors, smoothing operations, and the like. The digital representation of the visual appearance of the virtual object may be in the form of a three-dimensional graphical representation.

In some embodiments, the process creates a data structure that includes a surface representation of the virtual object used to draw the virtual object. If the movement of the virtual object is to be animated in the virtual environment, creating a 3D digital representation of the virtual object may also include creating a data structure representing a skeletal skeleton for animating the virtual object. Accordingly, creating may include creating a surface representation to have a shape and/or size and/or color based on the detected shape and/or size and/or color of the toy construction model, and based on the detected shape and/or size and/or color of the toy construction model or dimensions to create a skeleton to have shape and dimension. For example, creating a skeleton may include selecting one of a set of skeleton templates and scaling the skeleton template based on the detected size and shape of the toy construction model; in some embodiments, a single template may be sufficient. For example, a template skeleton can be defined such that a virtual object is animated to resemble some type of figure, such as a human figure with arms and legs and animated to resemble a walking character, or an animal with four legs, or wings and Birds in flight motion, or fish or snake figures animated to swim or slide. Selecting a skeleton template may be performed automatically, eg, based on the detected toy construction model's shape, and/or based on user selection, eg, selecting the type of character to create, such as a fish, snake, quadruped, etc.

In some embodiments, the process creates a 3D digital representation of the virtual object such that the virtual object - or only a portion of the virtual object - appears to be constructed from toy construction elements. To this end, the process may create a virtual construction model created from virtual construction elements that correspond to real-world toy construction elements of the toy construction system. In particular, in some embodiments, the process creates a data structure that includes information about a plurality of construction elements and their relative positions and orientations with respect to one another and/or with respect to a suitable coordinate system. For each construction element of the virtual model, the data structure may also include information about the type of construction element, its color, and/or other characteristics (such as weight, surface texture, etc.). In some embodiments, the data structure representing the virtual construction element also includes information about the type and location of the coupling members of the construction element. Accordingly, the data structure of the virtual model may include connection information indicating which virtual construction elements are interconnected with each other via their respective linking members. An example of such a data structure is described in US7439972.

Alternatively or additionally, the process may create a three-dimensional representation with visible repeating features, such as coupling members, common to all or some of the toy construction elements of the system. In one embodiment, the process detects the positions of the coupling members of the toy construction elements along the edges of the extracted two-dimensional view, and adds the graphical representations of the corresponding coupling members at corresponding positions in the three-dimensional graphical representation, such as in the three-dimensional graphical representation. corresponding positions on the circumferential surface.

3D digital representations can be associated with virtual objects in video games or other forms of virtual environments. Various aspects described herein can be implemented with various gaming systems, such as computer-generated virtual environments. Generally, virtual objects may represent virtual characters, such as human-like characters, animal-like characters, fantasy creatures, and the like. Alternatively, the virtual objects may be inanimate objects, such as buildings, vehicles, plants, weapons, and the like. In some embodiments, virtual objects (whose real-world counterparts are inanimate, such as cars) may be used as animated virtual characters in the virtual environment. Thus, in some embodiments, the virtual object is a virtual character, and in some embodiments, the virtual object is an inanimate object.

Virtual characters may interact with the virtual environment by moving within the virtual environment, by interacting with or generally engaging with other virtual characters and/or interacting with inanimate virtual objects present in the virtual environment and/or interacting with the virtual environment itself and/or The virtual environment evolves (eg, grows, ages, develops or incapacitates, attributes, etc.) to demonstrate behavior. Often, virtual objects have properties, such as abilities, that affect the game or other evolution of the virtual environment. For example, a car can have a specific maximum speed, or an object can have properties that determine if or how a virtual character interacts with a virtual object, etc.

To this end, the process includes assigning virtual attributes, such as behavioral attributes (such as the virtual object's abilities, needs, preferences or other attributes) or other game-related attributes, to the virtual object, for example based on detected visual characteristics of the real-world object, for example by The mechanism disclosed in International Patent Application PCT/EP2015/062381 is used.

Virtual object-related properties may include one or more global properties that indicate properties of the entire virtual object. Examples of global attributes may include one or more of the following: attributes indicative of abilities or skills, attributes indicative of energy levels, attributes indicative of global behavior of an object, and the like. Other examples of global attributes include the ability to move, such as the ability to fly or move on a surface, speed and/or pattern of movement, and the like. Other examples of properties include local properties that relate to parts of a virtual object rather than the entire virtual object. For example, a local attribute may indicate a functional capability of a portion of an object, such as an attribute indicating the ability of a portion of a virtual object to move relative to the rest of the virtual object or relative to another portion of the virtual object. Other examples of local attributes may indicate the ability of a portion of the virtual object to eject one or more virtual ejection elements, such as virtual projectiles, virtual beams, virtual fluids, and/or other virtual effects. Other examples of local properties may include the ability of a portion of a virtual object to interact with other virtual objects, such as establishing connections, exchanging virtual elements, and the like. Typically, local properties include functional/behavioral properties that represent different functionality/behavior than static properties that do not change (eg, color, texture, etc.).

The process may create a computer-generated virtual environment and simulate the evolution of the virtual environment over time, including the behavior of one or more virtual characters and/or the properties of one or more virtual objects within the virtual environment. For the purposes of this specification, a computer-generated virtual environment may be persistent, that is, it may continue to evolve and exist even without a user interacting with it, such as between user sessions. In alternative embodiments, the virtual environment may evolve only when the user interacts with it, eg, only during an active user session. The virtual object may be controlled, at least in part, by a user, ie, the processor may control the behavior of the virtual object based at least in part on received user input. A computer-generated virtual environment can be a single-user environment or a multi-user environment. In a multi-user environment, more than one user may interact with the virtual environment at the same time, eg, by controlling various virtual characters or other virtual objects in the virtual environment. Computer-generated virtual environments, especially persistent multi-user environments, are also sometimes referred to as virtual worlds. Computer-generated virtual environments are often used in gaming systems, where a user can control one or more virtual characters in the virtual environment. The avatar controlled by the user is also sometimes referred to as a "player". It should be understood that at least some embodiments of the aspects described herein may also be used in contexts other than gaming. Examples of computer-generated virtual environments may include, but are not limited to, video games such as skill games, adventure games, action games, real-time strategy games, role-playing games, simulation games, etc., or combinations thereof.

The portion of the virtual object to which the local attribute is assigned may be identified as part of the 3D digital representation of the virtual object's visual appearance. For example, in embodiments where the 3D digital representation of the visual appearance of the virtual object includes a plurality of geometric elements, such as a plurality of surface elements of a mesh representing the surface of the virtual object or a plurality of voxels of a volumetric representation of the virtual object, the selected portion A selected subset of geometric elements can be included.

Selecting a portion of the virtual object may be performed automatically, eg, based on one or more detected features of the virtual object, or user-assisted, eg, at least in part in response to user input.

In some embodiments, selecting a portion of the virtual object includes detecting predetermined features, such as geometric features, of the virtual object based on a 3D digital representation of the virtual object's visual appearance. The geometrical features may be, for example, predetermined geometrical shapes, such as circles, ellipses, polygons, and the like. In some embodiments, the detection of features may include identifying one or more object components. For example, in embodiments where the real world object is a real world toy construction model constructed from individual toy construction elements, the detected feature may be one or more of the identified toy construction elements that are part of the real world toy construction model.

In some embodiments, the user may indicate a portion of the virtual object, and the process may then detect predetermined features within the user-selected portion. The process can then select a portion of the virtual object based on the detected features. For example, the process may select a detected feature as the selected portion, or the process may select a portion of the virtual object in the vicinity of the detected feature as the selected portion. In general, a user can indicate a portion of a virtual object in a number of ways. For example, the user may indicate the outline of an area within the representation of the virtual model on the display, eg, using a pointing device or another pointing mechanism. Alternatively, the user may point to a location on the surface of the virtual surface, and the process may determine a portion of the virtual object as a portion having one or more properties in common with the user-selected location, such as the same color, texture, etc. Alternatively or additionally, the process may use one or more object segmentation techniques to segment the virtual model into segments. The process may then allow the user to select one or more segments.

In some embodiments, the process may detect multiple candidate features. The process can then indicate detected candidate features and allow the user to select one or more detected candidate features. The process may then determine the selected portion based on the candidate features selected by the user.

The assigned attributes may be selected automatically, manually by the user, or semi-automatically (eg, in a user-assisted manner).

The selection of the attribute may be based at least in part on one or more characteristics of a portion of the virtual object to which the attribute is assigned and/or based on one or more characteristics of another portion of the virtual object. For example, the type of attribute or one or more characteristics of the attribute may be selected based on characteristics of a portion of the virtual object and/or another portion of the virtual object. A feature of a portion of the virtual object or another portion of the virtual object may include color, size, shape, or another detectable feature. Characteristics of an attribute may include the strength, level, or other characteristics of the attribute.

Accordingly, in some embodiments, assigning one or more properties to selected portions of the virtual object includes:

- detecting one or more properties of the selected portion and/or one or more other portions of the virtual object that differ from the selected portion; and

- Selecting one or more attributes based on the detected one or more characteristics.

Each of the detected property or properties may be a visually obtainable visual property of the toy construction model, such as properties that may be derived from the color and/or shape and/or size of the toy construction model.

For example, when the selected portion has been assigned the ability to eject an ejection element, the process may select an ejection element from the selected portion to eject based on characteristics of another portion of the virtual object, such as based on the properties of the virtual object in the vicinity of the selected portion. part of the color. For example, red may cause the discharge element to be gunfire, while blue may cause the discharge element to be water. In other embodiments, one or more other properties may be selected based on properties of the selected portion and/or based on properties of another portion of the virtual object. Examples of other attributes may include the degree of functionality, such as the amount of ejection elements to be ejected, ejection speed, repetition rate, and the like.

In some embodiments, the assigned attributes have associated attribute directions, and the method further includes:

- determine the orientation relative to the selected portion of the virtual object; and

- Assign the determined direction as the attribute direction to the assigned attribute.

For example, the attribute can be movement and the attribute direction can be the direction of movement. Alternatively, the attribute may be the rotation of a portion of the virtual object, and the attribute direction may be the direction of the axis of rotation. Alternatively, the attribute may be an ejection capability to eject one or more ejection elements from a portion of the virtual object, and the attribute direction may be the direction in which the ejection elements are ejected.

The direction relative to the selected portion may eg be determined as the surface normal of the surface position of the predetermined portion, eg the surface normal at the center (eg geometric center) of the surface of the selected portion. Alternatively, another mechanism may be used to determine the orientation of a portion, such as by detecting one or more edges of the selected portion and detecting the orientation of the one or more edges. In yet another alternative embodiment, the direction may be detected from one or more detected features. For example, the process may detect an ellipse in the view of the virtual object, and detect the direction from the major and minor axes of the ellipse, such as such that the direction is the normal direction from the center of the disk, when viewed from the viewpoint , which leads to an ellipse.

In some embodiments, the virtual object is a vehicle operable by the virtual character or a creature on which the virtual character can ride; the assigned attribute is a riding position of the virtual character relative to the virtual object; and the method includes:

- determining the surface normal of the surface of the virtual object at the selected riding position; and

- Based on the determined surface normal, the riding posture of the virtual character is determined.

In some embodiments, the process allows the user to indicate the riding position. During the selection process, the process may animate the avatar to climb on top of the virtual object towards the indicated riding position. To this end, the process can use the surface normals of the digital representation of the virtual object to determine the direction of the virtual character while climbing toward the riding position.

In some embodiments, selecting a portion of the virtual object includes:

- identifying a portion of the virtual object as a representation of an identified one of a plurality of predetermined portions, the identified portion corresponding to a subset of the plurality of geometric elements; wherein the numerical representation of the plurality of predetermined portions is stored in the in a library; wherein the one or more numerical representations of the predetermined portions have one or more predetermined attributes associated with them; and

- Replacing the subset of geometric elements with the stored digital representation of the identified predetermined portion.

For example, the real-world object may be a toy construction model constructed from multiple construction elements, and the process may access a library of virtual construction elements corresponding to each real-world construction element. The toy construction elements may include coupling members for removably interconnecting the toy construction elements to each other. The process may identify one or more construction elements as part of a real-world object. The process may then replace one or more corresponding portions of the virtual object corresponding to the identified toy construction element with the digital representation obtained from the repository. Thus, virtual objects can be represented based on more precise digital representations of individual construction elements than can be achieved from a traditional 3D reconstruction pipeline. Furthermore, some virtual construction elements may have predetermined functions associated with them. For example, a wheel can be animated to spin, a door can be animated to be opened, a fire hose can be animated to spray water, a cannonball can be animated to discharge projectiles, etc.

Thus, users can create real-world toy construction models that resemble objects used as virtual objects in a computer-generated virtual environment. Since the user can build these objects from toy construction elements, the user has a great deal of freedom in how the objects are constructed. Furthermore, the system provides users with flexible, well-understood and easy-to-use mechanisms for influencing desired behaviors or other properties of virtual objects in virtual environments, such as behavioral properties (such as capabilities, needs, preferences, or other properties of virtual objects) or virtual Other game-related properties of the object.

The coupling member may releasably connect the construction elements to other construction elements using any suitable mechanism. In some embodiments, the coupling member includes one or more protrusions and one or more cavities, each cavity being adapted to receive at least one protrusion in frictional engagement.

In some embodiments, the toy construction elements may adhere to a set of constraints, eg, with respect to their shape and size and/or with respect to the position and orientation of the coupling members and with regard to the coupling mechanism employed by the coupling members. In some embodiments, at least some of the coupling members are adapted to define a connection direction and allow each construction element to interconnect with another construction element in a discrete number of predetermined relative orientations with respect to the construction elements. Thus, a variety of possible build options can be provided while ensuring the interconnectivity of the build elements. The coupling members may be positioned at grid points of a regular grid, and the dimensions of the toy construction elements may be defined as integer multiples of a unit length defined by the regular grid. It will be appreciated that a three-dimensional grid may be defined by a single unit length, by two unit lengths, eg one unit length for two spatial dimensions and another unit length for a third spatial dimension. Alternatively, the three-dimensional grid may define three unit lengths, one for each spatial dimension. Accordingly, coupling members conforming to the toy construction system obey the connection rules imposed by the toy construction system, including, for example, the type, location and/or orientation of the coupling members, and they are configured to engage one or more of the toy construction elements of the toy construction system. Mating coupling elements.

The present disclosure is directed to various aspects including the methods described above and below, corresponding apparatus, systems, methods and/or articles of manufacture, each yielding one or more of the benefits and advantages described in connection with the first-mentioned aspect, and each There is one or more embodiments which correspond to the embodiments described in connection with the first mentioned aspect and/or disclosed in the appended claims.

In particular, the present disclosure also relates to a computer-implemented method for creating a virtual object; the method includes:

- receiving a digital representation of the real-world object representing the visual appearance of the shape of the real-world object;

- detect one or more geometric features of real-world objects from the received digital representation;

- create virtual objects that represent real-world objects; and

- Assigning a direction of travel to the created virtual object based on the detected one or more geometric features.

The present disclosure also relates to a data processing system configured to perform the steps of an embodiment of one or more of the methods disclosed herein. To this end, a data processing system may include or be connected to a computer-readable medium from which a computer program may be loaded for execution into a processor, such as a CPU. Accordingly, a computer-readable medium may store thereon program code means, which when executed on a data processing system are adapted to cause the data processing system to perform the steps of the methods described herein. The data processing system may include a suitably programmed computer, such as a laptop computer, tablet computer, smartphone, PDA, or another programmable computing device with a graphical user interface. In some embodiments, a data processing system may include a client system, eg, including a camera and a user interface, and a host system that may create and control a virtual environment. The client and host systems may be connected through a suitable communication network, such as the Internet.

Generally, the term processor here and hereinafter is intended to include any circuit and/or device and/or system adapted to perform the functions described herein. In particular, the above term includes general-purpose or special-purpose programmable microprocessors, such as central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic arrays ( PLA), Field Programmable Gate Array (FPGA), dedicated electronic circuits, etc., or a combination thereof. A processor may be implemented as multiple processing units.

Some embodiments of data processing systems include sensor devices, such as image capture devices, such as cameras (eg, video cameras), or any other suitable device for obtaining one or more images of toy construction models or other real-world objects. Other embodiments may be configured to generate digital representations of real-world objects and/or retrieve previously generated digital representations.

The sensor may include a radiation source operable to direct radiation towards the toy construction model. For example, the image capture device may include a flash, one or more LEDs, lasers, and the like. Alternatively, the image sensor device is operable to detect ambient radiation reflected by the object. Here, the term reflection is intended to mean any type of passive emission, including diffuse reflection, refraction, etc., in response to received radiation or waves. Embodiments of the data processing system may include a display or other output device for presenting a virtual environment to a user. Embodiments of the data processing system may also include a scanning station, eg, including an object support, such as a turntable.

The present disclosure also relates to a computer program product comprising program code means adapted to cause the data processing system to perform the steps of one or more of the methods described herein when executed on a data processing system.

The computer program product may be provided as a computer-readable medium, such as a CD-ROM, DVD, optical disk, memory card, flash memory, magnetic storage device, floppy disk, hard disk, and the like. In other embodiments, the computer program product may be provided as a downloadable software package, eg, on a web server over the Internet or other computer or communication network, or as an application program for download from an App store to a mobile device.

The present disclosure also relates to a toy construction system comprising a data processing system as described above and a plurality of toy construction elements as described herein. A data processing system may include a processor; a data processing system may also include an image capture device and a display or other output device.

The present disclosure also relates to a toy construction set comprising a plurality of toy construction elements, and instructions for obtaining computer program code that, when executed by a data processing system, causes the data processing system to perform the performance described herein The steps of one or more embodiments of a method. For example, instructions may be provided in the form of an Internet address, a reference to an App store, and the like. The toy building set may even include a computer-readable medium, such as computer program code, stored thereon. Such a toy building set may even include a camera or other image capture device connectable to a data processing system. Optionally, the toy construction set may also include a scanning station, a sensor device, or both.

Embodiments of the toy construction system allow a user to construct a variety of toy construction models in a uniform and well-structured manner and with a limited set of different toy construction elements. For example, a toy construction system may be provided as a toy construction set including a plurality of toy construction elements. Users can also create various virtual objects that exhibit various behavioral properties in the virtual environment.

Description of drawings

Figure 1 schematically illustrates an embodiment of the system disclosed herein.

Figure 2 shows a flow diagram of an embodiment of the process described herein.

3-5 illustrate examples of user-assisted selection of a portion of a virtual object.

6 illustrates an example of a user-assisted identification of attributes to assign to a selected portion of a virtual object.

7-8 illustrate further examples of user-assisted selection of a portion of a virtual object.

9 shows a flowchart of a process for determining a principal direction, such as the direction of movement of a virtual object created based on a scan of an image or other form of real-world object.

Figure 10 shows an example of a virtual object displayed in a display area of a computer, with detected edges indicated by arrows.

Figure 11 shows an example of orientation local attributes.

FIG. 12 shows another example of a process for user-assisted selection of a portion of a virtual model.

FIG. 13 shows another example of a process for creating a virtual object.

Detailed ways

Various aspects and embodiments of the methods, apparatus, and toy construction systems disclosed herein will now be described with reference to toy construction elements in the form of bricks, such as in the form of toy construction elements provided under the LEGO name. However, the present invention may be applied to other forms of toy construction elements used in toy construction sets.

Figure 1 shows an embodiment of a toy construction system. The system includes a computer 101, an input device 102, a display 103, a sensor device including a camera 104, an object support including a turntable 105, and a toy construction model 106 constructed from one or more toy construction elements.

The computer 101 may be a personal computer, a desktop computer, a laptop computer, a handheld computer such as a tablet computer, a smartphone, etc., a game console, a handheld entertainment device, or any other suitably programmable computer. Computer 101 includes a processor 109 such as a central processing unit (CPU) and one or more storage devices such as memory, hard disk, and the like.

Display 103 is operably coupled to computer 101 , and computer 101 is configured to present a graphical representation of virtual environment 111 on display 103 . Although shown as a separate functional block in FIG. 1, it is to be understood that the display may be integrated into the housing of the computer.

Input device 102 is operably coupled to computer 101 and is configured to receive user input. For example, input devices may include keyboards, game controllers, mice or other pointing devices, and the like. In some embodiments, the system includes more than one input device. In some embodiments, the input device may be integrated into the computer and/or display, for example in the form of a touch screen. It should be understood that the system may include other peripheral devices operably coupled to, such as integrated into, the computer.

Camera 104 is operable to capture images of toy construction model 106 and forward the captured images to computer 101 . To do this, the user may position the toy construction model 106 on the turntable 105 . In some embodiments, a user may build a toy structure model on top of the base plate. The camera may be a digital camera operable to take digital images, eg in the form of a two-dimensional array of pixels. In particular, the camera may be configured to capture the light intensity of each pixel and additional information, such as polarization information and/or surface normal direction for each pixel or group of pixels. Alternatively, other types of sensor devices may be used, such as other types of image capture devices. Also, in an alternate embodiment, the system does not use a carousel.

Display 103, camera 104, and input device 102 may be operably coupled to the computer in various ways. For example, one or more of the above-described devices may be coupled to computer 101 through a suitable wired or wireless input interface of computer 101, such as through a serial or parallel port of the computer, such as a USB port, through Bluetooth, Wifi, or other suitable wireless Communication Interface. Alternatively, one or all of the devices may be integrated into the computer. For example, a computer may include an integrated display and/or input device and/or an integrated camera. In particular, many tablets and smartphones include integrated cameras, integrated touch screens that can be used as display and input devices.

A program, such as an App or other software application, suitable for simulating a virtual environment is stored on the computer 101 to process the captured images and create virtual objects as described herein. For example, the virtual environment may be part of a computer game.

It should be appreciated that in some embodiments, computer 101 may be communicatively connected to a host system, such as through the Internet or other suitable computer network. The host system can then perform at least a portion of the processing described herein. For example, in some embodiments, a host system may generate and simulate a virtual environment, such as a virtual world that may be accessed by multiple users from respective client computers. The user can use a client computer executing an appropriate program to capture the image. The captured images can be processed by the client computer or uploaded to the host system to process and create corresponding virtual objects. The host system can then add virtual objects to the virtual world and control virtual objects within the virtual world, as described herein.

In the example of FIG. 1, the virtual environment 111 is an underwater environment such as a virtual aquarium or other underwater environment. The virtual objects 107, 108 resemble fish or other underwater animals or creatures. It should be understood, however, that other types of virtual environments may be implemented in which virtual objects represent other types of virtual objects, such as other types of virtual characters, animals, creatures, vehicles, accessories, tools, and the like.

In the example of FIG. 1 , the computer has created a virtual object 107 based on the captured image of the toy construction model 106 . The computer has created the virtual object 107 to construct a model like a toy, for example by creating a 3D mesh or other suitable representation. In the example of FIG. 1 , virtual object 107 is similar in shape and color to toy construction model 106 . In this example, the virtual objects are even similar to the individual toy construction elements that build the toy construction model 106 . However, it should be understood that different levels of similarity can be achieved. For example, in some embodiments, virtual objects may be created to resemble only the overall shape of the construction model without simulating the internal structure of its individual toy construction elements. Virtual objects may also be created to have dimensions that correspond to the dimensions of the virtual construction elements, for example by providing a reference length scale on the turntable 105, to allow the computer to determine the actual dimensions of the toy construction model. Alternatively, the computer may use the dimensions of the toy construction elements as a reference length scale. In other embodiments, the user can manually scale the size of the virtual object.

The system shown in Figure 1 can be configured to create 3D representations of real-world objects, which can then be used to create virtual objects or characters, such as described in more detail below.

Figure 2 shows a flow diagram of an embodiment of the process described herein. For example, the process may be performed by the system described in FIG. 1 or by another suitable programmed data processing system.

In an initial step S1, the process obtains data indicative of the visual appearance of a toy construction model or another real world object. For example, the process may capture multiple digital images of the toy construction model at respective angular positions of the turntable on which the toy construction model is located or from various viewpoints. The data may include digital images and/or depth maps and/or another form of data representing the visual appearance of the object, such as surface texture, color, shape, and the like.

In a subsequent step S2, the process builds a 3D digital representation of the toy construction model from the data obtained (eg from digital images). To this end, the process may perform one or more image processing steps known per se in the field of digital image processing. For example, processing may include one or more of the following steps: background detection, edge detection, color calibration, color detection.

The process for generating 3D digital representations of real-world objects from multiple captured images may employ any suitable technique known in the art of object reconstruction. For example, in one embodiment, the captured image is processed to extract:

- information about the real world scene seen through the camera,

- the turntable position or camera viewpoint relative to the object, and

- Object outline.

In a subsequent step, the obtained contours can be projected onto the corresponding sculpted voxelized volumes. Subsequently, the marching cube algorithm is applied to the 3D objects obtained from the sculpting. The final mesh is then obtained and the texture cut out from the camera frame is applied on top of it. This process can result in a mesh representation of the object's surface, such as triangles using other polygons. Alternatively, the process may result in a voxel representation of the object.

In a subsequent step S3, the process creates a virtual object. The virtual object has a visual appearance defined by the 3D representation created in the previous step or the 3D representation derived therefrom. Additionally, virtual objects may include additional global properties, such as properties that define behavioral characteristics of the virtual object. These characteristics can define how the object moves, its capabilities and functions, etc.

In a subsequent step S4, the process selects a part of the 3D representation of the virtual object, eg a part of the 3D shape of the object. This step can be performed completely automatically, or it can be performed assisted by a user. For example, a user can manipulate a representation of a virtual object to identify a portion to select. Alternatively, the process may use object segmentation techniques and/or feature detection techniques and/or other suitable mechanisms to identify a portion of the virtual object. An example of a method for selecting a portion of a virtual object will be described in more detail below. In some embodiments, the process may identify multiple candidate portions from which the user may select one or more portions. The selected portion may thus be identified in a suitable data structure representing the virtual object, eg by identifying a plurality of grid elements or voxels representing the selected portion of the object.

In a subsequent step S5, the process determines the local attributes to be assigned to the selected portion of the virtual object. To do this, the user can select an attribute, eg from a list of available attributes. Alternatively or additionally, the process may automatically determine the attribute—or at least one or more characteristics of the attribute—to assign to the selected portion. For example, the process may select attributes based on properties of the selected portion and/or based on properties of other portions of the virtual object. Alternatively, the process may determine a number of candidate attributes from which the user may select attributes.

In a subsequent step S6, the process assigns the determined attributes to selected parts of the virtual object. To this end, the process may set corresponding property values associated with the selected portion of the object in the data structure representing the virtual object. It should be understood that some types of properties may be represented by a single value, while other types of properties may have more complex representations.

It should be understood that more than one property may be assigned to a selected portion of an object, and that more than one portion of an object may be selected and assigned a virtual property. To this end, steps S4-S6 may be repeated an appropriate number of times.

3-5 and 7-8 illustrate examples of user-assisted selection of a portion of a virtual object.

In particular, Figure 3 illustrates the selection of a portion of an object based on the detection of predetermined features. More specifically, in this example, the process detects ellipses in a representation of a virtual object, eg, in a 2D view of the virtual object. Figure 3 shows a representation of a virtual object 301 created within the display area of a computer display (eg, of the system of Figure 1 or other suitably programmed processing device or system). In this example, the virtual object represents a vehicle that can be used by a virtual character in a computer game. Virtual objects have been created based on multiple images taken from different viewpoints of a real-world vehicle constructed from toy construction elements, eg, as described above.

The data processing system may provide functionality that allows the user to manipulate the view of the created virtual object 301, eg, by rotating the object, by zooming, and the like. The process can be configured to detect predetermined shapes in the current view, such as ellipses, polygons, etc., and highlight the detected shapes. To this end, the process may use any suitable method to detect predetermined features in the image or 3D representation known in the art. In the example of FIG. 3 , the process detects an ellipse at the end of the tubular construction element 302 . The process highlights the detected ellipse by drawing an emphasized ellipse 303 in a predetermined color (eg, in red). It should be understood that the process may initially identify multiple features, such as other ellipses 304 . The process can then select one of the detected features, for example based on other criteria such as color, size, orientation, user input, etc.

The process may then assign attributes to the selected features 303 of the virtual object, eg, automatically or by allowing the user to select the features. In the example of FIG. 3 , the process has selected the water gun function as the attribute to assign to the selected feature 302 . The water gun function simulates the discharge of water from the detected ellipse. This process may allow the user to select one or more additional characteristics, such as the amount of water to be drained, the speed or extent of the water to be drained, the direction of drain, and the like. In Figure 3, this is illustrated by symbols 305 and 306, which show the two available pistol selections and allow the user to select one type of pistol. Alternatively or additionally, the process may automatically determine one or more attribute characteristics of the assigned attribute. In particular, the process may determine such property features based on the detected properties of the selected portion and/or based on properties of other portions of the virtual object (eg, adjacent portions).

For example, in this example of a water gun, the color of the selected portion may determine the color of the water for the water cannon attribute or another attribute of the discharged water, such as the virtual temperature of the discharged water, which in turn may affect how the discharged water affects aspects of the game . The size or shape of the selected portion can determine how the drain element will be drained when the drain property is assigned, such as how water is sprayed if the water cannon property is applied. As yet another example, if a water cannon attribute is applied, a graphical indicia of a real-world object and detected as part of the obtained digital representation may determine that the water is hot.

The water pistol function is an example of a property that has a direction associated with it, ie the direction in which the water is expelled in this case. In FIG. 3 , this direction is represented by arrow 307 . The orientation may be user-selectable, or it may be determined automatically by the process, eg, based on the orientation of the ellipse and based on the orientation and dimensions of the major and minor axes of the ellipse.

In general, Figure 3 shows an example of a user-assisted selection of model features, where the process automatically detects one or more model features, and the user manually selects one or more specific features to assign numerical attributes to. The method for implementing user selection will be described in more detail below. Examples of detectable features include colors, shapes (eg, circles, polygons, ovals, etc.), 3D geometric features (eg, toy construction elements), edges, faces, polygons, materials (eg, rubber), graphic assets/markers, and the like.

FIG. 4 illustrates another example of a user-assisted selection process for selecting a portion of an object based on detection of predetermined features. In the example of Figure 4, the process initially performs an object segmentation process, for example using a mesh segmentation algorithm or by detecting toy construction elements or other predetermined parts of the object. The user can then identify one or more segments, such as by pointing to the segment or by selecting a color or other characteristic of the segment to be selected, or the like. Examples of optional features include colors, shapes (eg, circles, polygons, ovals, etc.), 3D geometric features (eg, toy construction elements), edges, faces, polygons, materials (eg, rubber), graphic assets/markers, and the like. The process then assigns attributes to selected segments of the virtual object, eg, automatically or by allowing the user to select features, eg, as described above.

More specifically, Figure 4 shows a representation of a virtual object 301 created within the display area of a computer display (eg, of the system of Figure 1 or other suitably programmed processing device or system). In this example, the virtual object is represented as being located on the carousel 408 of the scanning station. The process shows a virtual character 409 that can be controlled by the user to walk around the virtual object 301 and use a target reticle 410 (eg, a pointer, crosshair, or other aiming device) to target corresponding portions of the virtual object. In the example of FIG. 4 , the process highlights the segment portion 411 of the virtual object that is currently targeted by the virtual character 409 . The user can now assign user-selected or automatically-selected attributes, eg, as shown in functional feature 412 that can be assigned to segment 411 .

As described above, automatic segmentation may be based on detection of individual toy construction elements or other components in the representation of the real-world object. To this end, the process may access a library of representations of a number of known toy construction elements or other known parts. Known components may be stored as CAD files or representations, eg as described in WO2004/034333 or in other suitable form. The corresponding components of the virtual object may then be detected and identified by suitable object identification processes known in the art. The identified parts may then be user-selectable such that attributes may be assigned to individual parts or groups of parts, such as subassemblies of two or more toy construction elements, including, for example, animals, known characters, characters, etc. or parts thereof, such as torso parts, legs, engine parts, wheels, etc. In other embodiments, one or more portions of the virtual object may be detected and identified based on known graphical assets/markers, etc.

5 illustrates another example of a user-assisted selection process for selecting a portion of an object based on detection of predetermined features. The example of FIG. 4 is similar to the example of FIG. 4 with a representation of virtual object 501 (which is obtained based on a captured image of a real-world object located on a turntable of a scanning station) shown on the display of a computer (eg, of the system of FIG. 1 ). in the display area. In this example, the real world object is a toy construction model constructed from toy construction elements, and the process has identified each toy construction element as part of the virtual object as described above. This process provides functionality that allows the user to control the virtual character 409 to target various parts of the virtual object. When the targeting portion corresponds to an identified toy construction element, the process may automatically assign a predetermined attribute to the selected portion, such as a predetermined attribute associated with known toy construction elements stored in a library of known toy construction elements. Alternatively, the identified object may have a number of properties available associated therewith, from which the user may select one or more properties to assign to the selected portion. In the example of FIG. 5, the virtual character 409 is targeting a portion 511 that has been identified as a toy construction element that has a number of attributes available associated therewith. Thus, in response to the user's selection of section 511, the process may display a selection menu that allows the user to select one of the available attributes.

An example of the selection process is shown in Figure 6, which shows the scene displayed after the user has selected the portion 511 and after the user has selected the booster function by clicking on one of the multiple selectable icons 613, 613, Each icon represents one of the available properties of the selected portion 510. The selected icon 613 is highlighted accordingly.

Figure 7 shows another mechanism for selecting a portion of a virtual object. In the example of FIG. 7, the process receives captured data indicating that the user is pointing to a portion of the corresponding real-world object 701 based on which the virtual object is created, eg, by the process described above.

In particular, Figure 7 shows a scanning station comprising a turntable 705 and an image capture device 704 for capturing images of real world objects 701 located on the turntable. When the user points to the portion 711 of the real world object, the image capture device captures one or more images. The process then detects which part the user is pointing at and identifies the corresponding part of the virtual object, eg based on a pre-segmentation of the virtual object as described above. The user may, for example, point to the selected portion with a finger 714, as shown in Figure 7, or employ a specific pointing device, such as a wand that may have a tip that is easily detectable by the system. For example, the tip of the pointing device may include marking features. Accordingly, detection of the selected portion of the object may be performed by suitable image processing techniques known in the art and based on one or more images of the user pointing to a portion of the real-world object.

Figure 8 shows another mechanism that allows a user to select a portion of a virtual object. In particular, FIG. 8 shows a representation of a virtual object 301 created within the display area of a computer display, eg, as described in connection with FIG. 3 . However, in the example of FIG. 8, the user may use a crosshair 815 or other pointing mechanism to point to the target portion 811 of the displayed representation of the virtual object. The user may also rotate and/or resize the display representation to further facilitate identification of the user-selected portion. The user can then select the attribute to assign to the selected portion, or the attribute can be selected automatically as described above.

Typically, the process may provide one or more mechanisms for user-controlled targeting/selection of various parts of the virtual object, including, for example:

- When the user points to a part of a real-world object, the user's finger and/or pointing device such as a "wand" is tracked by a camera that captures an image of the corresponding real-world object.

- User selects feature/section using scroll cursor.

- The user-controlled virtual character moves around the 3D representation of the virtual object and targets a portion of the virtual object with the target reticle.

- Use a virtual reality interface that allows the user to rotate and interact with 3D representations of virtual objects.

Figure 9 shows a flowchart of a process for determining a principal direction, such as the direction of movement of a virtual object created based on an image or another form of scanning of a real-world object. For example, the process may be performed by the system described in FIG. 1 or by other suitable programmed data processing systems.

In initial steps S1-S3, the process obtains data indicative of the visual appearance of the toy construction model or another real-world object (step S1), constructs a 3D digital representation of the toy construction model from the obtained data (step S2) and creates Virtual objects (step S3 ), all as described in connection with the process of FIG. 2 .

In a subsequent step S904, the process detects the edges of the virtual object, eg using suitable edge detection methods known per se in the art. Figure 10 shows an example of a virtual object 1001 displayed in a display area of a computer, with detected edges indicated by arrows.

Still referring to FIG. 9, in the subsequent step S905, the process determines the main direction of movement of the created virtual object based on the detected edges. To do this, the process can detect one or more properties of the detected edges, such as one or more of the following properties:

- the position of the detected edge in 3D space relative to the virtual object,

- the orientation of the detected edge in 3D space relative to the virtual object,

- The length of the corresponding edge.

Alternatively or additionally, the process may detect other features from which the principal direction can be derived. Examples of such features include: edge vectors of toy construction elements, planes (eg, of wheels), flatness of ellipses, grids defined by the positions of coupling members of toy construction elements, and the like. Based on one or more of the aforementioned characteristics, the process may determine one or more possible principal directions. For example, the process can determine the most likely dominant direction as the direction in which the majority of detected edges point. In another embodiment, the process may determine the most likely dominant direction based not only on the number of edges but also on the sum of the lengths of edges pointing in a given direction. In yet another embodiment, the process may also take into account the location of edges, eg by weighting clusters of edges that are close to each other above edges that are spaced apart from each other.

The process may then select the determined most probable principal direction as the principal direction associated with the virtual object. Alternatively, the process may present a plurality of candidate directions to the user in order to allow the user to select the primary direction among the identified candidate directions. Figure 10 also shows a graphical user interface element 1016 that allows the user to select one of a plurality of candidate directions.

In an alternate embodiment, edge detection can also act as a "manual overlay" selector. For example, the system may display detected edges in a displayed representation of the virtual object and allow the user to select the edge that best represents the global direction of travel or another principal direction relative to the virtual object.

Once a global principal direction (eg, a global travel direction) is assigned to a virtual object, the global principal direction may be used to define a direction associated with one or more local attributes, such as the direction in which an ejected element is to be ejected.

It should be understood that the global orientation axis may also be related to the virtual object in other ways, such as during scanning. For example, directions may be indicated on the turntable, and the corresponding orientation of the real-world object may be defined by the orientation of the real-world object relative to the turntable when positioned on the turntable. This orientation can then be detected during the scanning process and assigned to a corresponding virtual object created based on the scanned real-world object.

As described above, some local attributes assigned to selected portions of virtual objects may have orientations associated with them. Figure 11 shows an example of orientation local attributes. In particular, Figure 11 shows a virtual vehicle 1101 having a first part 1111a to which the laser function has been assigned, a pair of second parts 1111b to which the rocket launcher function has been assigned, and a pair of first parts 1111b to which the exhaust pipe function has been assigned Three parts 1111c. In addition, a virtual character holding a tool 1117 is placed on the driver's seat of the vehicle, where the tool also has an interaction direction associated with it. The laser function, rocket launcher function, and exhaust pipe function may each have an exit direction associated with them, which indicates the direction in which the respective exhaust element (ie, laser beam, rocket, exhaust gas, respectively) exits. These directions are indicated by arrows in FIG. 11 .

The aforementioned functional attributes may be assigned to various parts by any of the techniques described herein, eg, based on user selection of a portion of a virtual object, eg, by selecting colors, shapes, 3D geometric shapes, faces, polygons, points, and the like. The directions associated with the respective functional attributes may be determined based on characteristics of the selected portion, ie based on the detected edges of the selected portion and/or based on the major surface normal or average surface normal of the selected portion. The determined direction can then be stored with the attribute and associated with the corresponding portion to which the direction attribute is assigned.

In some embodiments, the process may map all local attributes to a global principal direction associated with the virtual object, eg, such that all ejection elements (eg, rockets) always fire forward relative to the principal direction of motion of the virtual object.

FIG. 12 shows another example of a process for user-assisted selection of a portion of a virtual model. According to this example, the process selects a portion of the virtual object at which interaction of the virtual object with another virtual object or with the virtual environment may occur. More specifically, in the example of FIG. 12 , the user selects a riding position of a virtual object, in this case a virtual vehicle. Once a portion of the dummy object is selected, the process assigns the selected portion a "riding position" attribute. The attribute can also have an orientation associated with it, such as the avatar's pose orientation when sitting in a riding position. The direction associated with the interaction property can be determined by the principal or average surface normal associated with the selected part. The user can select parts of the virtual object that can interact with by moving the pointer or cursor over the representation of the model. The process may show another virtual object interacting with the virtual model at the portion indicated by the user, and show the change in interaction as the user moves a cursor or pointer over the representation of the virtual object. For example, in the context of the riding position example shown in Figure 12, the process may use surface normals at corresponding locations on the virtual object's surface to make The avatar 1218 "climbs" animation. To this end, when the user moves the cursor or pointer over a virtual object, such as with a game controller, the corresponding local normal value is passed to the character animation routine.

FIG. 13 shows another example of a process for creating a virtual object. In initial steps S1 and S2, the process obtains data indicative of the visual appearance of a toy construction model or another real-world object, and from the obtained data (eg, from the digital images described in connection with steps S1 and S2 of the FIG. 2 embodiment) ) to construct a 3D digital representation of a toy construction model. The 3D representation may be a surface mesh representation or another suitable 3D representation. In the example shown in Figure 13, the real world object is a toy construction model in the form of a figurine 1301 assembled from a number of toy construction elements, eg elements similar to the legs 1319 of the character, the torso portion 1320 of the figurine , the figurine's head 1321, the armored helmet 1322 worn by the figurine, and a number of accessories 1323 and 1324 carried by the figurine. It should be understood that this process may also be performed based on another type of toy construction model constructed from other types of toy construction elements.

In step S1303, the process detects one or more toy construction elements of the toy construction model, eg, based on the library 1325 of known toy construction elements. In particular, it has been recognized that for the purpose of creating a virtual object, it may not be necessary to identify all toy construction elements of a model, but it may be sufficient to identify certain types of toy construction elements. In particular, some parts of the virtual object are more important for the realistic animation of the features of the virtual object than others. For example, proper animation of the figurine's torso and/or legs may be more important to the movement of the virtual figurine than, for example, the helmet or accessories. Similarly, the wheels of the car may be more important to the animation of the car than, for example, the chassis part.

Thus, by providing a library of selected known toy construction elements and their characteristics, such as movement features, allows the creation of virtual objects that exhibit rich behavior in a virtual environment, while reducing and identifying multiple toy construction elements of a toy construction model related computing tasks. The library 1325 of known toy construction elements may store therein 3D representations of known toy construction elements, eg in the form of surface mesh representations, voxel representations, CAD representations, etc., eg as described in WO2004/034333. Known toy construction elements may further have one or more properties associated with them stored, such as a color, a skeleton for facilitating motion animation, one or more animation routines, one or more other functional features.

It should be understood that the identification of the subset of toy construction elements of the toy construction model may be performed using identification processes known in the art, for example by comparing the 3D model of the known toy construction elements with portions of the obtained 3D model of the toy construction model Matching and/or by using neural networks or other adaptive data-driven techniques, eg as described in WO2016/075081.

In the example of Figure 13, the process has detected the figurine's torso part 1320 as a known torso part stored in a known torso part library. It will be appreciated that, alternatively or additionally, the process may identify other known portions, such as legs 1319.

In a subsequent step S1304 , the process replaces a portion of the 3D representation of the toy construction model that has been identified as corresponding to a known toy construction element with the corresponding 3D representation of the known toy construction element retrieved from the library 1325 . This replacement is shown schematically in the lower right of FIG. 13 .

The thus modified 3D representation generated in step S1304 can then be used as the 3D representation of the virtual object in the virtual environment. To this end, predefined properties of known toy construction elements that have replaced part of the 3D representation can be automatically assigned to corresponding parts of the 3D virtual object.

It should be appreciated, however, that this process may further allow for user-assisted assignment of attributes through one of the mechanisms described herein. These attributes may be assigned to a portion of the virtual object produced by the replacement step 1304, since this portion can now be easily selected by the user, eg, as described above with reference to FIGS. 5 and 6 . Alternatively or additionally, attributes may be assigned to other parts of the virtual object selected by the user.

Examples of replacing parts of a 3D representation by known elements include:

- Identify and replace the torso of the figurine. The underlying bone structure that animates known parts is also relative to the parts that have been added to replace the original mesh. The user can then manually select the torso to assign other properties, for example, the "Ninja" or "Strongman" properties, to define how the torso animate moves. The same can be done for the legs, head, etc.

- Identify and replace wheels on cars or other vehicles. The wheel's axis of rotation is associated with a known wheel as a property; the user can define other properties, such as the car's global behavior, which defines how the car drives and how much grip property the tires are assigned.

- Identify and replace toy construction elements with graphic markings by known elements. As mentioned above, the orientation information of the surface of the identification part is also known. The user can then manually assign properties. For example, assigning a water cannon attribute to a component with a graphical flame marking could result in a spray of hot water that could melt ice in a computer game.

In some embodiments, the process automatically detects and replaces color and/or graphic markers, and allows the user to manually control the animation and display of features associated with colors or markers through properties. This type of identification and replacement can be performed independently of any identification of toy construction elements. Unlike the above-described embodiments, where colors or markers are used as selectable areas to apply attributes, this embodiment uses identified markers as automatic targets for animation and/or identified colors as automatic targets for visual effects. Selection of effects or animations can still be applied in response to user selections (global or local). Examples of these embodiments include:

- Recognize graphic markers on creature's eyes and overlay corresponding animation targets on virtual objects. If the user selects the global property "Angry Monster", the angry eye animation is displayed. Alternatively, the user may locally define that the left eye will be an "angry monster" and the right a "cute girly monster".

- Recognize colors on creatures and overlay corresponding animation targets on areas. For example, the user can locally assign the "hair" display attribute to all green areas, and green hair appears to grow outside the green areas. Globally, the user can assign a "bee" to the entire object, and the process assigns the yellow and black "blur" display properties to areas of different colors.

Embodiments of the methods described herein may be implemented by means of hardware comprising several distinct elements, and/or at least in part by means of a suitably programmed microprocessor.

In the claims enumerating several means, several of these means may be embodied by one and the same element, means or item of hardware. The mere fact that certain measures are recited in mutually different dependent claims or described in different embodiments does not indicate that a combination of these measures cannot be used to advantage.

It should be emphasized that, when used in this specification, the term "comprising" is used to designate the presence of stated features, elements, steps or components, but does not exclude one or more other features, elements, steps, components or groups thereof presence or addition.

Claims (16)

1. A computer-implemented method for creating a virtual object; the method comprises the following steps:

-obtaining a digital representation of the visual appearance of the real world object, the digital representation being created from data received from one or more sensor devices operable to capture radiation from the real world object;

-creating a virtual object having a visual appearance based on the received digital representation;

-selecting a portion of the virtual object; and

-assigning one or more local properties to the selected portion of the virtual object, the local properties indicating a functional capability of a portion of the virtual object;

the assigned local attribute is a riding position of the virtual character relative to the virtual object; and wherein the method comprises:

-determining a surface normal of a surface of the virtual object at the selected riding position; and

-determining a ride posture of the virtual character based on the determined surface normal.

2. The method of claim 1; wherein the obtained digital representation represents at least a surface geometry of the real world object.

3. The method according to claim 1 or 2; wherein selecting a portion of the virtual object comprises:

-detecting a predetermined feature of the virtual object based on the obtained digital representation; and

-selecting a portion of the virtual object that is related to the detected feature.

4. The method of claim 3; wherein selecting a portion of the virtual object comprises:

-receiving a user input indicating a user selected portion of the virtual object;

-detecting a predetermined feature of the virtual object based on the user selected portion.

5. The method of claim 4; wherein selecting a portion of the virtual object comprises:

-detecting one or more features of the virtual object based on the obtained digital representation;

-receiving a user input indicative of a user selection of one of the detected one or more features; and

-selecting a portion of the virtual object in response to a user selection.

6. The method according to claim 1 or 2; wherein selecting a portion of the virtual object comprises receiving a user input indicating a user selected portion of the virtual object.

7. The method of claim 6; wherein the user input indicates a user selected location on a surface of the virtual object; wherein, the method comprises the following steps:

-determining visible characteristics of a virtual object associated with the user selected position;

-determining a portion of the virtual object as a portion of the virtual object having a common visual characteristic with the determined visual characteristic associated with the user selected position.

8. The method according to claim 1 or 2; wherein the assigned local attribute has an associated attribute direction, and wherein the method further comprises:

-determining a direction associated with the selected portion of the virtual object; and

-assigning the determined direction as the attribute direction to the assigned local attribute.

9. The method according to claim 1 or 2; wherein the digital representation comprises a plurality of geometric elements, such as surface elements or volume elements, representing a surface model or a volume model of the real world object.

10. The method of claim 9, wherein selecting a portion of a virtual object comprises:

-identifying a portion of the virtual object as a representation of the identified one of the plurality of predetermined portions, the identified portion corresponding to a subset of the plurality of geometric elements;

-retrieving the stored digital representation of the identified predetermined part from a library of stored digital representations of a plurality of predetermined parts; wherein the one or more stored digital representations of the predetermined portion have associated therewith one or more predetermined local attributes; and

-replacing the subset of geometric elements with the retrieved digital representation of the identified predetermined portion.

11. The method according to claim 1 or 2; wherein assigning the one or more local attributes to the selected portion of the virtual object comprises:

-detecting one or more characteristics of the selected portion and/or one or more other portions of the virtual object different from the selected portion; and

-selecting one or more local attributes based on the detected one or more characteristics.

12. The method of claim 11; wherein the one or more other portions are portions having a predetermined spatial relationship with the selected portion.

13. The method according to claim 1 or 2, wherein the local attribute indicates a functional capability selected from:

-the ability of a part of the virtual object to move relative to the rest of the virtual object or relative to another part of the virtual object;

-the ability of a portion of the virtual object to eject one or more virtual ejection elements;

-the ability of a part of a virtual object to interact with other virtual objects.

14. A data processing system configured to perform the steps of the method according to any one of claims 1 to 13.

15. A computer program product comprising program code means adapted to cause a data processing system to perform the steps of the method according to any one of claims 1 to 13 when said program code means are executed on said data processing system.

16. A toy construction system comprising:

-a data processing system according to claim 14; and

-a plurality of toy construction elements, each toy construction element comprising one or more coupling members configured for detachably interconnecting the toy construction elements to each other in order to form a real world object.

Images