KR20260004340A - Frame enhancement using diffusion models - Google Patents

Abstract

Systems and techniques for processing image data are provided. According to some aspects, a computing device can determine an optical flow between a current frame having a first resolution and a first previous frame having the first resolution. The computing device can warp a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having a second resolution, wherein the second resolution is higher than the first resolution. The computing device can process the noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

Description

Frame enhancement using diffusion models

The present disclosure relates generally to video processing. For example, aspects of the present disclosure relate to systems and techniques for performing frame enhancement (e.g., video super-resolution, sharpening, etc.) using a diffusion model.

Many devices and systems enable a scene to be captured by generating images (or frames) of the scene and/or video data (including multiple frames). For example, a camera or a device including a camera may capture a sequence of scene frames (e.g., a video of the scene). In some cases, the sequence of frames may be processed to perform one or more functions, output for display, and/or output for processing and/or consumption by other devices, among other uses.

Artificial neural networks (ANNs) aim to replicate the logical reasoning performed by biological neural networks, such as those found in animal brains, using computer technology. Deep neural networks, such as convolutional neural networks (CNNs), are widely used in numerous applications, including object detection, object classification, object tracking, and big data analysis. For example, CNNs can extract high-level features, such as facial features, from input images and use these high-level features to output, for example, the probability that the input image contains a specific object.

The following presents a brief summary of one or more aspects disclosed herein. Therefore, the following summary should not be considered a comprehensive overview of all contemplated aspects, nor should it be considered to identify key or critical elements of all contemplated aspects, or to delineate categories associated with any particular aspect. Accordingly, the following summary serves the sole purpose of presenting specific concepts of one or more aspects of the mechanisms disclosed herein in a simplified form that precedes the more detailed description presented below.

Systems, methods, devices, and computer-readable media for performing frame enhancement (e.g., video super-resolution, sharpening, etc.) using a diffusion model are disclosed. According to at least one illustrative example, a method of processing image data of a current frame is provided. The method includes: determining an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; warping a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and processing the noisy frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

In another illustrative example, an apparatus for processing image data of a current frame is provided. The apparatus includes at least one memory configured to store the image data, and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: determine an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; warp a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and process the noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

In another illustrative example, a non-transitory computer-readable storage medium is provided comprising instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to: determine an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; warp a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and process the noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

In another exemplary embodiment, an apparatus for processing image data of a current frame is provided. The apparatus includes: means for determining an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; means for warping a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and means for processing the noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

The embodiments generally include methods, apparatus, systems, computer program products, non-transitory computer-readable media, user devices, user equipment, wireless communication devices, and/or processing systems, substantially as described herein with reference to the drawings and the specification, and as exemplified by the drawings and the specification.

The foregoing has outlined rather broadly the features and technical advantages of examples according to the present disclosure so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The concepts and specific examples disclosed may readily be utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The features of the concepts disclosed herein, both their organization and method of operation, together with associated advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of the drawings is provided for purposes of illustration and description and is not intended as a definition of the limitations of the claims. The foregoing, together with other features and aspects, will become more apparent upon reference to the following specification, claims, and accompanying drawings.

This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The claimed subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

The accompanying drawings are provided to aid in describing various aspects of the present disclosure and are provided solely for the purpose of illustration, not limitation, of the aspects. So that the features described above of the present disclosure may be understood in detail, a more specific description, briefly summarized above, may be had by reference to aspects, some of which are illustrated in the accompanying drawings. It should be noted, however, that the appended drawings illustrate only certain typical aspects of the present disclosure and are therefore not to be considered limiting of the scope of the present disclosure, for the description may admit to other equally effective aspects. The same reference numerals in different drawings may identify the same or similar elements.
FIG. 1 illustrates an exemplary implementation of a system-on-a-chip (SOC), according to some embodiments.
Figure 2a illustrates an example of a fully connected neural network according to some embodiments.
Figure 2b illustrates an example of a locally connected neural network according to some embodiments.
FIG. 3 is a diagram illustrating the forward diffusion process and the reverse diffusion process of a diffusion model according to some embodiments.
FIG. 4 is a diagram illustrating an example of a system for performing frame enhancement using a diffusion model, according to some embodiments.
FIG. 5 is a diagram illustrating another example of a system for performing frame enhancement using a diffusion model, according to some embodiments.
FIG. 6 is a diagram illustrating another example of a system for performing frame enhancement using a diffusion model, according to some embodiments.
FIG. 7 is a diagram illustrating another example of a system for performing frame enhancement using a diffusion model, according to some embodiments.
FIG. 8 is a flowchart illustrating an example of a process for processing frame data according to some embodiments.
FIG. 9 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein.

Certain aspects of the present disclosure are provided below for illustrative purposes. Alternative aspects may be devised without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure may not be described in detail or may be omitted so as not to obscure relevant details of the present disclosure. Some of the aspects described herein can be applied independently, and some can be applied in combination, as would be apparent to those skilled in the art. In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of the aspects of the present application. However, it will be apparent that various aspects may be practiced without these specific details. The drawings and description are not intended to be limiting.

The following description provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes in the function and arrangement of elements may be made without departing from the scope of the present application as set forth in the appended claims.

The demand for and consumption of image and video data has increased significantly in both consumer and professional settings. As described above, devices and systems are typically equipped with capabilities for capturing and processing image and video data. For example, a camera or a computing device including a camera (e.g., a smartphone or mobile phone including one or more cameras) may capture video and/or images of scenes, people, objects, etc. The images and/or videos may be captured, processed, and then output (and/or stored) for consumption. The images and/or videos may be further processed using one or more frame enhancement techniques for specific effects or improvements (e.g., improvements in quality, bitrate, etc.), such as super-resolution (also referred to as upscaling or resolution enhancement), compression (also referred to as encoding), frame rate upconversion, sharpening, color space conversion, image enhancement, high dynamic range (HDR), noise reduction, and low-light compensation, among others. Images and/or videos may also be further processed for specific applications such as, among others, computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), and autonomous driving. In some examples, the images and/or videos may be processed using one or more image or video artificial intelligence (AI) models, which may include, but are not limited to, AI quality enhancement and AI augmentation models. As used herein, the terms "image processing" and "video processing" may be used interchangeably to describe, for example, an image processing neural network and a video processing neural network (e.g., based on video data comprising a series of frames (e.g., images) that may be processed sequentially).

Image and video processing operations can be computationally intensive. In some cases, image and video processing operations can become increasingly computationally intensive as the resolution of the input images or frames of video data increases (e.g., as the number of pixels to be processed per input image or frame of video data increases). For example, a frame of video data having a 4K resolution may contain approximately four times as many individual pixels as a frame of video data having a Full HD (e.g., 1080p) resolution.

An example of an image or video processing technique is video super-resolution (VSR), which can be used for spatial super-resolution (to increase resolution) and/or temporal super-resolution (e.g., to increase frame rate). For example, VSR can be used to increase the spatial resolution of a video, such as from 720p to 1080p or from 1080p to 4K. VSR can be used to improve the user experience at low cost while reducing the burden on computationally or bandwidth-intensive upstream tasks, such as video synthesis or video coding (e.g., in gaming). An example of a VSR technique for gaming is to generate very high-quality but expensive frames at a lower resolution (e.g., by performing ray tracing), and then "cheaply" interpolate the frames to a higher resolution to keep the computational requirements within a reasonable amount. For example, ray tracing can be used to generate high-quality low-resolution frames, and VSR techniques can be used to upsample the low-resolution frames to a higher resolution. In some cases, these VSR techniques can also perform temporal super-resolution (e.g., frame interpolation). In one illustrative example using a video coding configuration, bitrate can be saved by encoding a 720p to 1080p video and performing video super-resolution at the receiver-end, trading bitrate for computation (e.g., resulting in a lower bitrate at a higher computational cost). For example, media hosting platforms can use this technique to save storage space and incur computational costs on the receiver device.

VSR can be a complex, high-dimensional, one-to-many mapping. Traditional VSR techniques optimize for distortion metrics, such as peak signal-to-noise ratio (PSNR), which do not align well with human-perceived quality. For example, by optimizing for distortion, the reconstructed frame may be very similar to the original frame in terms of distortion, but may have poor perceptual quality. Some machine learning-based VSR techniques optimize for perceptual quality (e.g., GAN-based VSR models), resulting in high perceptual quality (and thus pleasing appearance from a human perspective). However, these machine learning-based VSR techniques focus on still image super-resolution, not video or frame sequences, and can introduce temporal inconsistencies (e.g., flickering) when applied to video. Some techniques also assume the availability of future frames, making them unsuitable for low-latency applications such as teleconferencing and video streaming. While some techniques can achieve both high quality and temporal consistency, they are limited to certain applications (e.g., gaming) that require additional inputs (e.g., optical flow, depth maps, etc.) or high-latency applications such as offline upscaling of historical videos. A VSR technique that balances perceptual quality, temporal consistency, and latency is needed.

Systems, devices, processes (also referred to as methods), and computer-readable media (collectively referred to as “systems and techniques”) for performing frame enhancement (e.g., video super-resolution, sharpening, etc.) using a diffusion model (referred to herein as a video enhancement diffusion model) are described herein. The video enhancement diffusion model provides a general-purpose, high-quality, low-latency frame enhancement (e.g., super-resolution, sharpening, etc.) algorithm that enables more general applications than existing super-resolution systems. For example, a video enhancement diffusion model can enable low-latency frame enhancement (e.g., super-resolution, sharpening, etc.) with high perceptual quality and high temporal consistency based on at least three aspects: diffusion-based training (which can provide perceptual quality), optical flow for temporal modeling via pixel- or feature-space warping using optical flow estimation and backwarping or deformable convolutions (which can provide temporal consistency), and recurrent upsampling instead of parallel upsampling (which can provide low latency).

Various aspects of the present disclosure will be described with reference to the drawings.

FIG. 1 illustrates an exemplary implementation of a system on a chip (SOC) (100), which may include a central processing unit (CPU) (102) or a multi-core CPU configured to perform one or more of the functions described herein. Among other information, parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., a neural network having weights), delays, frequency bin information, task information may be stored in a memory block associated with a neural processing unit (NPU) (108), in a memory block associated with a CPU (102), in a memory block associated with a graphics processing unit (GPU) (104), in a memory block associated with a digital signal processor (DSP) (106), in a memory block (118), and/or may be distributed across multiple blocks. Instructions executed in the CPU (102) may be loaded from a program memory associated with the CPU (102) or may be loaded from a memory block (118).

The SOC (100) may also include additional processing blocks tailored to specific functions, such as a GPU (104), a DSP (106), a connectivity block (110) that may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and a multimedia processor (112) that may detect and recognize gestures, for example. In one implementation, the NPU is implemented in the CPU (102), the DSP (106), and/or the GPU (104). The SOC (100) may also include a navigation module (120) that may include a sensor processor (114), image signal processors (ISPs) (116), and/or a global positioning system. In some examples, the sensor processor (114) may be associated with or connected to one or more sensors to provide sensor input(s) to the sensor processor (114). For example, the one or more sensors and the sensor processor (114) may be provided on, coupled to, or otherwise associated with the same computing device.

The SOC (100) may be based on the ARM instruction set. In one aspect of the present disclosure, instructions loaded into the CPU (102) may include code for searching a stored multiplication result in a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU (102) may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into the CPU (102) may include code for storing a computed multiplication product of the input value and the filter weight when a lookup table miss of the multiplication product is detected. The SOC (100) and/or components thereof may be configured to perform image processing using machine learning techniques according to aspects of the present disclosure discussed herein. For example, the SOC (100) and/or its components may be configured to perform semantic image segmentation and/or object detection according to aspects of the present disclosure.

Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences, without the use of explicit instructions. An example of an ML system is a neural network (also referred to as an artificial neural network), which can include interconnected groups of artificial neurons (e.g., neuron models). Neural networks can be used for a variety of applications and/or devices, such as image and/or video coding, image analysis, and/or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, and service robots, among others.

Individual nodes within a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of these operations are then optionally passed on to other neurons. Weight values are associated with each vector and node within the network, and these values constrain how the input data relates to the output data. For example, the input data of each node may be multiplied by the corresponding weight value, and the resulting products may be summed. The sum of the products may be adjusted by an optional bias, and an activation function may be applied to the result to produce the node's output signal, or "output activation" (sometimes referred to as an activation map or feature map). The weight values may be initially determined by iteratively passing training data through the network (e.g., established during a training phase in which the network learns to identify specific classes based on typical input data characteristics).

There are different types of neural networks, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multilayer perceptron (MLP) neural networks, transformer neural networks, and diffusion-based neural networks, among others. For example, convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. Convolutional neural networks may comprise a set of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) that collectively tile the input space. RNNs operate on the principle of storing the output of a layer and feeding this output back to the input to help predict the layer's performance. GANs are a form of generative neural network that can learn patterns from input data, allowing the neural network model to generate new synthetic outputs that could have been appropriately derived from the original dataset. A GAN may include two neural networks working together: a generative neural network that generates synthesized outputs and a discriminative neural network that evaluates the outputs for authenticity. In MLP neural networks, data can be fed into an input layer, and one or more hidden layers provide levels of abstraction for the data. Predictions can then be made at the output layer based on the abstracted data.

Deep learning (DL) is an example of a machine learning technique and can be considered a subset of machine learning (ML). Many DL approaches are based on neural networks, such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), and utilize multiple layers. The use of multiple layers in deep neural networks allows for progressively higher-level features to be extracted from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input to the second layer of artificial neurons, the output of the second layer becomes the input to the third layer of artificial neurons, and so on. The layers located between the input and output of the entire deep neural network are often referred to as hidden layers. The hidden layers learn (i.e., are trained) to transform intermediate inputs from preceding layers into slightly more abstract and complex representations that can be fed to subsequent layers, until the final or desired representation is obtained as the final output of the deep neural network.

As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of the input layer, processing is performed by hidden nodes of one or more hidden layers, and output is produced through output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of the neural network may include feature maps or activation maps, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of one or more nodes in the layers. In some cases, a deep learning network may have a series of many hidden layers, where early layers are used to determine simple, low-level input features, and later layers build a hierarchy of more complex, abstract features.

Deep learning architectures can learn a hierarchy of features. For example, when presented with visual data, the first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. The second layer, which takes the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

Deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. For example, classifying electric vehicles can benefit from first-level training that recognizes wheels, windshields, and other features. These features can then be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

Neural networks can be designed with a variety of connectivity patterns. In feedforward networks, information flows from lower layers to higher layers, with each neuron in a given layer communicating with neurons in higher layers. As described above, hierarchical representations can be built across successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also called top-down) connections. In recurrent connections, the output from a neuron within a given layer can be communicated to another neuron within the same layer. Recurrent architectures can be helpful in recognizing patterns that span more than one chunk of input data passed to the neural network in sequence (e.g., examples of recurrent neural network architectures are illustrated in Figures 4-6 and described in more detail below). Connections from neurons within a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of high-level concepts can help identify specific low-level features of the input.

Connections between layers of a neural network can be fully connected or locally connected. Figure 2a illustrates an example of a fully connected neural network (202). In a fully connected neural network (202), a neuron in a first layer can communicate its output to every neuron in a second layer, such that each neuron in the second layer receives input from every neuron in the first layer. Figure 2b illustrates an example of a locally connected neural network (204). In a locally connected neural network (204), a neuron in a first layer can be connected to a limited number of neurons in a second layer. More generally, a locally connected layer of a locally connected neural network (204) can be configured such that each neuron in a layer has the same or similar connectivity pattern, but connection strengths that can have different values (e.g., 210, 212, 214, 216). Locally connected connectivity patterns can give rise to spatially distinct receptive fields in higher layers, because higher-layer neurons within a given region can receive inputs that are tuned through training on properties of a limited subset of the total input to the network.

As mentioned above, one class of machine learning models includes diffusion models (e.g., diffusion-based neural networks), also known as diffusion probabilistic models. Diffusion models are latent variable models. For example, a diffusion model defines a Markov chain of diffusion steps to slowly add random noise (e.g., Gaussian noise) to data, and then learns to construct desired data samples from the noise by reversing the diffusion process. For example, a diffusion model can be trained using a (fixed) forward diffusion process and a (trained) backward diffusion process. A diffusion model can also be trained to perform a generative process (e.g., a denoising process). One exemplary goal of a diffusion model is to remove random noise added to input data (e.g., a video).

Figure 3 provides two sets (300) of images illustrating the (fixed) forward diffusion process and the (learned) backward diffusion process of the diffusion model. As depicted in the forward diffusion process of Figure 3, noise (303) is incrementally added to a first set (302) of images at different time steps for a total of T time steps (e.g., forming a Markov chain) to generate a sequence of noise samples X ₁ through X _T . In one illustrative example, the noise (303) is Gaussian noise. Each time step may correspond to a respective successive image of the first set (302) of images illustrated in Figure 3 . The initial image X ₀ of Figure 3 is of a cat. The addition of noise (303) to each image (corresponding to noise samples X ₁ through X _T ) results in a gradual diffusion of pixels in each image until the final image (corresponding to sample X _T ) essentially matches the noise distribution. For example, by adding noise, each data sample X ₁ to X _T will gradually lose distinguishable features as the time step gets larger, and eventually the final sample X _T will have a target noise distribution, for example, a unit variance zero-centered Gaussian. It becomes equivalent to .

The second set of images (304) illustrates a backward diffusion process where X _T is a starting point with a noisy image (e.g., having Gaussian noise). A diffusion model can be trained to reverse the diffusion process (e.g., by training a model _pθ- (x _t-1 | x _t )) to generate new data. In one illustrative example, the diffusion model can be trained by finding backward Markov transitions that maximize the likelihood of the training data. By traversing backwards along a chain of time steps, the diffusion model can generate new data. For example, as illustrated in FIG. 3, a backward diffusion process is performed to generate X ₀ as an image of a cat.

As mentioned earlier, once captured, images and videos can be further processed for specific effects. For example, video enhancement aims to restore high-quality sequences of frames disturbed by some degradation process. Common problems in this class include super-resolution, compressed video enhancement, and noise removal, for example.

Existing machine learning (e.g., neural networks)-based video enhancement methods can be categorized into unidirectional and bidirectional approaches, which indicate whether the method accesses future frames when enhancing the current frame. Both categories have advantages and disadvantages. Unidirectional approaches access only information from previous frames, or sometimes up to one future (unenhanced) frame. Therefore, these methods can potentially run in real-time, but can suffer from long-term dependencies and temporal coherence. Bidirectional approaches enhance all frames (past and future) simultaneously and freely access both past and future information. Using past and future frames improves temporal coherence and allows for highly parallelized inference, but at the expense of increased memory usage and latency (e.g., preventing real-time operation for applications such as teleconferencing and video streaming).

In some cases, diffusion models (or diffusion probabilistic models (DPMs)) can be used for video enhancement. DPM-based video enhancement approaches typically fall into the bidirectional category, employing architectures that process multiple frames simultaneously using three-dimensional (3D) convolutions or temporal attention layers. Simultaneous processing of multiple frames allows for parallel sampling of adjacent frames, but offers little flexibility (e.g., using the same fixed number of steps for all frames, even if consecutive frames are similar). Furthermore, many DPM-based works omit frame alignment, opting instead to let the model perform the necessary processing without frame alignment. Other DPM-based video enhancement methods provide contextual information (e.g., depth maps) as input to the DPM to improve fidelity in video-to-video conversion. However, even in these cases, frame alignment is typically implicit. One could argue that convergence to a bidirectional approach without explicit alignment is advantageous, since custom flow estimators or warping operators are no longer required. However, since DPM must now perform these operations implicitly, removing these priorities is generally computationally expensive.

As previously mentioned, systems and techniques for performing frame enhancement (e.g., video super-resolution, sharpening, etc.) using a video enhancement diffusion model are described herein, which provides a general-purpose, high-quality, low-latency frame enhancement (e.g., video super-resolution, sharpening, etc.) algorithm that enables more general applications than existing super-resolution systems. For example, the video enhancement diffusion model can be trained based on diffusion-based training, which can provide perceptual quality. In some cases, optical flow can be performed for temporal modeling (e.g., in pixel-space using optical flow estimation and backwarping, or in feature-space using deformable convolutions) that can provide temporal consistency. For example, the model can use explicit motion estimation between input frames and can be conditioned on motion between input frames. Circular upsampling can also be performed instead of parallel upsampling (e.g., by performing the diffusion process sequentially on a frame-by-frame basis). For example, processing frames in a circular manner introduces sequential dependencies, in which case parallel sampling may not be performed. However, performing circular processing of frames (e.g., by performing the diffusion process sequentially frame by frame) may be slower than "parallel" sampling of all or many frames of the video. The inherent delay of circular processing (e.g., due to the lack of parallel sampling) can be compensated for by reusing information from previous time steps, such as by reusing noise and previous diffusion latencies of the sampling scheme, skipping sampling steps, reusing sampling steps, and/or any adaptation thereof. For example, the delay can be compensated for by reusing or skipping sampling steps between adjacent time steps. In some cases, the video enhancement diffusion model described herein can be performed in a unidirectional video enhancement setting.

Examples will be described herein using video super-resolution as an illustrative example of frame enhancement operations that can be performed by a video enhancement diffusion model. However, in some embodiments, the video enhancement diffusion model can be trained to perform other frame enhancement operations.

FIG. 4 is a diagram illustrating an example of a video model (400) that includes a video enhancement diffusion model (408) according to aspects described herein. As illustrated, inputs to the video enhancement diffusion model (408) include an input low-resolution frame (404), input noise (406), and one or more previous upsampled frames (402). The one or more previous upsampled frames (402) may include a frame prior to the current input low-resolution frame (404) (e.g., frame x _t-1 if the input low-resolution frame (404) is frame x _t ). The input noise (406) may include a noise map. The video enhancement diffusion model (408) may generate an output upsampled frame (410) based on the input low-resolution frame (404), the input noise (406), and the one or more previous upsampled frames (402). The output upsampled frame (410) has a higher resolution than the input low-resolution frame (404), but has high perceptual quality and high temporal consistency with previous frames.

As previously mentioned, most existing video super-resolution methods access future low-resolution frames and are therefore unsuitable for low-latency applications. As illustrated in Figure 4, future low-resolution frames are not required to perform video super-resolution. The use of a diffusion model for the video enhancement diffusion model (408) results in higher perceptual quality compared to other machine learning approaches.

FIG. 5 is a diagram illustrating another example of a video model (500) including a video enhancement diffusion model (508) according to aspects described herein. Similar to the video model (400) of FIG. 4, the input to the video enhancement diffusion model (508) of FIG. 5 includes an input low-resolution frame (504) and input noise (506). The input noise (506) may include a noise map.

The video model (500) also includes a flow estimation engine (514). The flow estimation engine (514) can estimate or determine optical flow between a current input low-resolution frame (504) and a previous low-resolution frame (505) (e.g., a frame prior to the current input low-resolution frame (504), such as frame x _{t -1} , when the input low-resolution frame (504) is frame x t). In some aspects, the flow estimation engine (514) is a machine learning-based optical flow estimator, such as a neural network-based optical flow estimator. For example, the flow estimation engine (514) can be a neural network model (e.g., a CNN, an RNN, etc.) trained to estimate optical flow between frames. The neural network model can be trained using any training technique, such as a supervised learning technique, a semi-supervised learning technique, an unsupervised learning technique, a self-supervised learning technique, or another training technique. As an illustrative example, using supervised learning, the training dataset may include one or more videos, and the ground truth for training may include optical flow maps for frames of the videos. A loss (e.g., mean square error (MSE) or another loss) may be determined based on a comparison of the estimated optical flow maps output by the flow estimation engine (514) with the ground truth optical flow maps. The loss may be used to perform backpropagation to tune parameters (e.g., weights, biases, etc.) of the neural network of the flow estimation engine (514).

In some embodiments, the flow estimation engine (514) may perform optical flow motion estimation on a pixel-by-pixel basis. For example, for each pixel in a previous low-resolution frame (505), a motion estimate f defines the location of the corresponding pixel in the input low-resolution frame (504). The motion estimate f for each pixel may include an optical flow vector (e.g., a motion vector) representing the movement of the pixel between frames. In some examples, an optical flow map (also referred to as, e.g., a motion vector map) may be generated based on the computation of the optical flow vectors between frames. In some cases, the optical flow map may include an optical flow vector for each pixel in a frame, where each vector represents the movement of the pixel between frames. For example, a dense optical flow may be computed between adjacent frames to generate an optical flow vector for each pixel in a frame, which may be included in the dense optical flow map. In some cases, the optical flow map may include vectors for fewer than all pixels in the frame. In some examples, the optical flow vector for a pixel may be a displacement vector (e.g., representing horizontal and vertical displacements, such as horizontal (x-) and vertical (y-) displacements) representing the movement of the pixel from a previous low-resolution frame (505) to an input low-resolution frame (504).

The warping engine (516) of the video model (500) can warp a previous upsampled frame (503) (e.g., upsampled by the video model (500)) to generate a warped previous upsampled frame (517) using the determined optical flow between the input low-resolution frame (504) and the previous low-resolution frame (505). For example, to generate the warped previous upsampled frame (517), the warping engine (516) can adjust (e.g., translate) each pixel of the previous upsampled frame (503) by an amount (e.g., in the horizontal and vertical directions) indicated by each optical flow vector (or motion vector) determined for each pixel.

As illustrated in FIG. 5, the warped previous upsampled frame (517) generated by the warping engine (516) based on optical flow can be used as an additional input to the video enhancement diffusion model (508). The video enhancement diffusion model (508) can generate an output upsampled frame (510) based on the input low-resolution frame (504), input noise (506), and the warped previous upsampled frame (517). The output upsampled frame (510) has a higher resolution than the input low-resolution frame (504) with high perceptual quality and high temporal consistency with the previous frames. For example, by estimating optical flow using low-resolution frames (e.g., input low-resolution frame (504) and previous low-resolution frame (505)) and then warping the previous upsampled frame (503) using optical flow, the upsampled prediction (output upsampled frame (510)) will be closer (more temporally consistent) to the previous upsampled frame (503).

In some embodiments, one or more deformable convolutions can perform feature-space warping using optical flow (e.g., features output by one or more layers of a machine learning system of a video model (500)) determined between an input low-resolution frame (504) and a previous low-resolution frame (505) in feature space. For example, a deformable convolution kernel or filter can process features (e.g., feature vectors or other feature representations) output by one or more layers of a neural network constituting the video model along with an optical flow map (500) to perform warping based on optical flow. In some cases, to perform the warping, the deformable convolution can replace a pixel value as a weighted sum of other pixel values in a particular region or neighborhood of pixels, where the optical flow map can specify locations of pixel values in a region or neighborhood of pixels around the pixel value.

As illustrated in Figure 5, no auxiliary inputs (e.g., depth maps, albedo maps, etc.) are required, making the solution applicable to a wide variety of scenarios or use cases. Many other VSR methods, but also applications that utilize flow estimation, use auxiliary inputs (e.g., depth maps, albedo maps, etc.).

In some cases, the input noise provided to one or more of the video enhancement diffusion models described herein (e.g., the input noise (406) provided to the video enhancement diffusion model (408) of FIG. 4 or the input noise (406) provided to the video enhancement diffusion model (508) of FIG. 5) may be resampled from a target distribution of noise (e.g., unit variance, zero-centered Gaussian noise) at specific points or intervals. For example, as noted above, the input noise (e.g., noise (406) and/or noise (506)) may include a noise map (or noise image). In one illustrative example, the input noise may be a map/image of Gaussian noise at a specific resolution (e.g., 4K resolution). Resampling the input noise refers to obtaining a new noise map from the target distribution of noise (e.g., a new map of Gaussian noise that is different from the previous input noise map), for example, by drawing new samples from the Gaussian noise distribution.

In one exemplary example, the input noise may be resampled at every time step (or a subset of time steps) in the input video sequence. In another exemplary example, the input noise may be resampled once per sequence of frames in the input video sequence (e.g., the sequence of frames may include 12 frames, 24 frames, 60 frames, 120 frames, or any other number of frames). In such an example, the noise may be constant for the number of frames in each sequence (e.g., 12 frames, 24 frames, etc.) and may be resampled at the first frame of each sequence. In another exemplary example, the input noise may be resampled when a scene cut is detected (e.g., based on a change in scene from one frame to another). In another exemplary example, the input noise can be resampled at set time intervals (e.g., every 0.5 second, every 1 second, or at other time intervals).

FIG. 6 is a diagram illustrating another example of a video model (600) that includes a video enhancement diffusion model (608) according to aspects described herein. Similar to the video model (400) of FIG. 4, the video model (600) includes a flow estimation engine (614) and a frame warping engine (616). The flow estimation engine (614) may perform the same operations as the flow estimation engine (514) described above with respect to FIG. 5. For example, the flow estimation engine (614) may estimate or determine optical flow between a current input low-resolution frame (604) and a previous low-resolution frame (605).

The frame warping engine (616) may perform the same operations as the warping engine (516) described above with respect to FIG. 5 . For example, the frame warping engine (616) may warp a previous upsampled frame (603) (e.g., upsampled by the video model (600)) to generate a warped previous upsampled frame (617) using the determined optical flow between the input low-resolution frame (604) and the previous low-resolution frame (605). In some cases, to generate the warped previous upsampled frame (617), the warping engine (616) may adjust (e.g., translate) each pixel of the previous upsampled frame (603) (e.g., in the horizontal and vertical directions) by an amount indicated by an individual optical flow vector (or motion vector) determined for each pixel.

The video model (600) further includes a noise warping engine (624). In some aspects, the noise warping engine (624) and the frame warping engine (616) may be the same warping engine (e.g., a shared warping engine may warp the previous noise input (622) and the previous upsampled frame (603)). The noise warping engine (624) may warp the previous noise input (622) using the determined optical flow between the input low-resolution frame (604) and the previous low-resolution frame (605) to generate warped noise (619). For example, the previous noise input (622) may include a frame having pixel values sampled from a noise distribution (e.g., a Gaussian noise distribution). The noise warping engine (624) can adjust (e.g., translate) each pixel of the previous noise input (622) by an amount indicated by an individual optical flow vector (or motion vector) determined for each pixel of the input low-resolution frame (604) (e.g., in the horizontal and vertical directions).

In addition to the input low-resolution frame (604) and input noise (606) (e.g., a noise map or image), the warped previous upsampled frame (617) generated by the frame warping engine (616) and the warped noise (619) generated by the noise warping engine (624) can be used as inputs to the video enhancement diffusion model (608). The video enhancement diffusion model (608) can generate an output upsampled frame (610) based on the input low-resolution frame (604), the input noise (606), the warped previous upsampled frame (617), and the warped noise (619). The output upsampled frame (610) has a higher resolution than the input low-resolution frame (604) with high perceptual quality and high temporal coherence with the previous frames.

In addition to using optical flow to warp the previously upsampled frame (603), the video model (600) of FIG. 6 may also warp the input noise (606) (e.g., a noise map) to generate warped noise (619) that may be used as input to the video enhancement diffusion model (608). In some cases, the warped noise (619) may replace the input noise (606) or may be combined with the input noise (606) (e.g., as a linear combination between the previously sampled noise and the currently resampled noise). As previously mentioned, in some cases, the input noise provided to one or more of the video enhancement diffusion models described herein (e.g., the input noise (406) provided to the video enhancement diffusion model (408) of FIG. 4 or the input noise (406) provided to the video enhancement diffusion model (508) of FIG. 5) may be resampled at specific points or intervals (e.g., at each time step, for each sequence of frames, at predetermined time intervals, when a scene cut or change is detected, any combination thereof, and/or at other points or intervals). As illustrated in FIG. 6, warping the noise map from one or more previous frames with optical flow may help enforce a consistent texture on objects in a scene (e.g., moving objects). In some embodiments, a combination of warped previous noise and newly sampled noise can be used, which can help with occlusions (e.g., where parts of a scene are occluded), scene cuts (e.g., where a new scene is presented from one frame to another), etc.

In some embodiments, the video enhancement diffusion models described herein (e.g., the video enhancement diffusion model (508) of FIG. 5 , the video enhancement diffusion model (608) of FIG. 6 , etc.) may be latent-variable models that learn to perform a backward diffusion process to reverse a forward diffusion process that destroys data in T steps, for example, as illustrated below:

In equation (1), and controls the noise schedule (referring to the progress of the forward diffusion process). Then, the generative model is a Markov chain with Gaussian transitions, with fixed diagonal covariance and user-specified conditioning c according to the schedule σ _t ² :

Sampling from (x) then involves T steps of ancestral sampling. To reduce the computational load of sampling from the diffusion model, fewer sampling steps can be performed at test time, for example, by respacing the sampling schedule (as described below) or through more advanced techniques such as step distillation.

In some cases, video enhancement diffusion models (e.g., video enhancement diffusion model 508 of FIG. 5, video enhancement diffusion model 608 of FIG. 6, etc.) can be trained using v-parameterization. For example, v-parameterization is an alternative objective to diffusion models that essentially stabilizes training by handling low signal-to-noise ratio (SNR) cases. Low SNR is increasingly likely when training for fewer and fewer diffusion steps. This is why the SNR+1 objective (also known as v-parameterization) is used. In some cases, v-parameterization is commonly used. A lower number of steps may be more appropriate than the objective function. For example, the video enhancement system described herein (including the video enhancement diffusion model) can be designed with a lower number of steps to achieve both low complexity and low latency, since multiple steps require more computation and time. The system can perform diffusion in the residual space, reuse noise from previous time steps, and utilize specific scheduling and noise (e.g., the denoising diffusion implicit model (DDIM)) to achieve a lower number of steps.

In some aspects, residuals The space of can be modeled instead of directly modeling the space of the image. go For lower resolutions, naive upsampling (e.g., bicubic interpolation) can be used as a preparatory step. For example, in some examples, at least two options for residuals, including spatial residuals and temporal residuals, can be used. For spatial residuals, and here is a low-resolution frame. For temporal residuals, and here is the previous upsampled frame, and f is the optical flow vector field.

Therefore, the video enhancement diffusion model is can learn to predict, where and controls the noise schedule, and r is the residuals (as previously described). ) is. In one illustrative example, as mentioned above, the SNR+1 training objective function (or v-parameterization) can be used, which is SNR( ) is based on a weighting factor equal to plus 1:

Here can be the output of a video enhancement diffusion model. Note that the output may vary depending on the exact loss formula, and that SNR+1 (v parameterization) is given as an example only.

In some embodiments, the image model may be used to perform super-resolution on single frames of a sequence of frames (e.g., a video) (e.g., a first or initial frame or other individual frames when previous frames are not available). For example, when performing image enhancement (e.g., super-resolution) on frames, the image model may be used on the first or initial frame of the sequence when previous low-resolution frames are not available. The image model may be a conditional diffusion model, where the conditioning is the first low-resolution frame (e.g. the frame to be enhanced), Any super-resolution method can be used. Using a diffusion model with a similar architecture to the video model allows for the reuse of weights. In these aspects, a video model including a video enhancement diffusion model described herein (e.g., a video model (500) including a video enhancement diffusion model (508), a video model (600) including a video enhancement diffusion model (608), etc.) can be applied to the remaining frames of a sequence of frames (e.g., frames occurring after a first or initial frame).

As mentioned above, the video enhancement diffusion model can also consider previously enhanced frames, in which case the conditioning can be as follows: . Figure 7 is a diagram illustrating an exemplary example of a system (700) including a video enhancement diffusion model using this notation. As illustrated in Figure 7, the flow estimation engine (714) takes as input a previous low-resolution frame (705) ( ) and currently low resolution (704) ( ) is received. The previous low-resolution frame (705) may be a frame prior to the current input low-resolution frame (704). The flow estimation engine (714) generates an optical flow frame (for each pixel of the previous low-resolution frame (705) including optical flow values such as displacement values) between the current input low-resolution frame (704) and the previous low-resolution frame (705), similar to that described with respect to FIGS. 5 and 6. The optical flow (725) can be estimated or determined as shown.

The warping engine (716) can warp a previous upsampled frame (703) (e.g., upsampled by the system (700)) using the determined optical flow (725), resulting in the generation of a warped previous upsampled frame (717). For example, the warping engine (716) can adjust (e.g., shift) each pixel of the previous upsampled frame (703) by an amount indicated by an individual optical flow vector (or motion vector) represented for each pixel in the optical flow (725) (e.g., in the horizontal and vertical directions).

A current low-resolution frame (704), a warped previous upsampled frame (717), and input noise (706) can be input to a video enhancement diffusion model (708). The video enhancement diffusion model (708) can generate an output upsampled frame (710) based on the current low-resolution frame (704), the input noise (706), and the warped previous upsampled frame (717). As described herein, by estimating an optical flow (725) using the current low-resolution frame (704) and the previous low-resolution frame (705) and warping the previous upsampled frame (703) using the optical flow (725), the upsampled frame (710) is closer to (e.g., more temporally consistent with) the previous upsampled frame (703). As a result, the output upsampled frame (710) has a higher resolution than the current low-resolution frame (704) and has high perceptual quality and high temporal consistency with respect to previous frames (including the previous upsampled frame (703)).

In some aspects, to improve temporal consistency, previously-enhanced frames (e.g., frame ) before providing the current frame as input to a video enhancement diffusion model, using, for example, optical flow, feature-based warping (e.g., with deformable convolutions), any combination thereof, and/or other temporal modeling. , for example, the input low-resolution frame (504) of FIG. 5, the input low-resolution frame (604) of FIG. 6, etc.). In some cases, the diffusion model may also be conditioned directly on the optical flow field (e.g., the motion vector field) or on previously aligned (previously warped) frames. In some examples, a pre-trained neural network optical flow estimator (e.g., using recurrent all-pair field transforms (RAFT) as the optical flow estimator) may be used (e.g., as the flow estimation engine (514) and/or the flow estimation engine (614)) to estimate or determine motion between non-enhanced frames (e.g., between the input low-resolution frame (504) and the previous low-resolution frame (512)). To ensure that the non-enhanced frames have the correct resolution and avoid retraining the flow estimator, bicubic upsampling of the low-resolution frames can be performed before determining the optical flow (e.g., by or before being input to the video enhancement diffusion model). As previously described, the estimated optical flow is directly applied to the previously enhanced frames using backward warping, and the aligned enhanced frames may result.

In some embodiments, training a diffusion model may include two training stages: 1) training an image model to enhance isolated frames (e.g., the initial or first frame of a video or other individual frames), and 2) training a warm-started video model using the image model weights (e.g., using the trained weights of the image model as a starting point for the video model). In one illustrative example, the first training stage may include approximately 500,000 steps, and the second training stage may include approximately 500,000 steps. In some cases, when training the video model, the output of the fully trained image model may be used as conditioning. This may result in super-resolving all frames of the training dataset, which may be expensive in some instances. In other cases, the ground truth image may be a previous frame that has been enhanced. can be perturbed by fixed downsampling and upsampling operations to generate proxies for .

To sample from the image model, a respacing procedure can be performed, and a denoising diffusion implicit model (DDIM) schedule can be used (e.g., due to its robustness when several sampling steps are used). For example, the respacing can be a different discretization of the same continuous function. In one illustrative example, using the function f(x) = x (linear) can be a schedule in which times are represented by continuous numbers between 0 and 1, where 0 represents the image distribution and 1 represents the noise distribution. If there are T steps in the backward process, continuous times 0 and 1 can map to discrete times 0 and T. The function can be discretized by sampling at a number of points (e.g., 100 points, 1000 points, 2 points, etc.). The number of points corresponds to the number of steps taken in the diffusion process. The fewer the steps, the greater the difference between the input and output of this diffusion step. Too many steps can underutilize and unnecessarily increase computation, while too few steps can prevent the model from representing such large differences and lead to errors. DDIM is a respacing procedure that increases robustness by disparating the steps between training (e.g., 1,000 steps) and evaluation (e.g., 100 steps). In one illustrative example, the default of 75 sampling steps can be used.

In some cases, latencies from previously enhanced frames can be used to increase the sampling rate when sampling from a video model. For example, using previous latencies can be advantageous for reducing sampling steps for videos with similar or identical frames (e.g., in the extreme case of still video where consecutive enhanced frames are exactly the same, zero sampling steps are needed, Reusing latencies from previous frames can also help with temporal consistency. For example, image models For (x), the same latency Two samples starting from , and can be the same when deterministic transitions are used. Assuming a deterministic sampling scheme and architecture is used and the latency is sampled at time step T, ) accurately determines the samples for a given conditioning. Intuitively, this is the motion vector This means that the noise vector provides information about how the noise should be reused. If there is little or no motion, there is likely no need for resampling, but if there is significant motion, a new noise vector must be sampled.

In some cases, the video super-resolution techniques described herein can utilize optimized hardware acceleration. For example, machine learning (ML) accelerators implemented by computing devices, such as mobile computing devices (e.g., smartphones, tablet computers, etc.) and/or other edge computing devices, may be referred to as mobile ML accelerators. ML accelerators may be implemented as specialized microprocessors for accelerating various computations associated with performing inference using ML models running on the computing device. ML accelerators may be provided as general-purpose hardware capable of performing acceleration for various types of ML operations and/or various ML networks and architectures. For example, a mobile ML accelerator incorporated in a smartphone may be used to accelerate image processing, natural language processing (NLP), speech recognition, etc. Different ML operations, networks, and/or architectures may be associated with different quantities of input channels. For example, image processing machine learning networks can utilize three-channel inputs (e.g., one red channel, one blue channel, and one green channel for RGB image input). NLP machine learning networks can utilize inputs with dozens or more channels. For example, an NLP machine learning network may utilize separate channels for different word embeddings, vocabularies, phrases, and so on.

FIG. 8 is a flowchart illustrating an example of a process (800) for processing frame data to perform frame enhancement on one or more frames using the techniques described herein. The process (800) may be performed by a computing device (or apparatus) or a component (e.g., a chipset, a codec, etc.) of a computing device. The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable such as a watch, an extended reality (XR) device (e.g., an extended reality (VR) device or a virtual reality (AR) device), a vehicle or a component or system of a vehicle, or another type of computing device. The operations of the process (800) may be implemented as software components that run and operate on one or more processors (e.g., the CPU (102), GPU (104), DSP (106) and/or NPU (108) of FIG. 1 , the processor (910) of FIG. 9 , or other processor(s)). Additionally, transmission and reception of signals by the computing device in process (800) may be enabled by, for example, one or more antennas, one or more transceivers (e.g., wireless transceiver(s)), and/or other components of the computing device.

At block (802), the computing device (or a component thereof) can determine optical flow between a current frame having a first resolution and a first previous frame having the first resolution. For example, the first previous frame can be the frame immediately preceding the current frame in the video (e.g., frame x _t-1 , if the current frame is frame x _t ). As an illustrative example, referring to FIG. 5, the flow estimation engine (514) can estimate or determine optical flow between a current input low-resolution frame (504) and a previous low-resolution frame (505). In some cases, the optical flow includes individual motion vectors for each pixel of the current frame (e.g., a first motion vector for a first pixel of the current frame, a second motion vector for a second pixel of the current frame, a third motion vector for a third pixel of the current frame, etc.).

At block (804), the computing device (or a component thereof) may warp a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having a second resolution, wherein the second resolution is higher than the first resolution. For example, the second previous frame may be an upsampled version of the first previous frame. As an illustrative example, referring to FIG. 5 , the warping engine (516) may warp a previous upsampled frame (503) to generate a warped previous upsampled frame (517).

At block (806), the computing device (or a component thereof) may process the noisy frame, the current frame, and the warped previous frame using a diffusion machine learning model (e.g., a video enhancement diffusion model described herein, such as video enhancement diffusion model (508), video enhancement diffusion model (608), etc.) to generate an output frame having a second resolution (e.g., an upsampled version of the current frame). As an illustrative example, referring to FIG. 5 , the video enhancement diffusion model (508) may process the warped previous upsampled frame (517), the input low-resolution frame (504), and the input noise (506) to generate an output upsampled frame (510). In some aspects, the noisy frame is sampled from a Gaussian noise distribution. As noted above, in some cases, the optical flow includes individual motion vectors for each pixel of the current frame. In these cases, to warp the second previous frame, the computing device (or a component thereof) may adjust each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow.

In some aspects, the computing device (or a component thereof) may warp a previous noise frame based on the determined optical flow to generate a warped noise frame. As an illustrative example, referring to FIG. 6, the noise warping engine (624) may warp a previous noise input (622) based on the determined optical flow from the optical flow estimation engine (614) to generate warped noise (619). In such aspects, the computing device (or a component thereof) may further generate an output frame based on processing the warped noise frame using a diffusion machine learning model (e.g., as illustrated in FIG. 6). In some cases, the computing device (or a component thereof) may use the warped noise frame instead of the noise frame to generate the output frame. For example, the computing device (or a component thereof) may process the warped noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having a second resolution. In some cases, the computing device (or a component thereof) may use a combination (e.g., a linear combination) of the noise frame and the warped noise frame to generate the output frame. For example, the computing device (or a component thereof) may process the combination of the noise frame and the warped noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate the output frame having the second resolution.

In some cases, a computing device (or a component thereof) may sequentially (or cyclically) process multiple consecutive frames of a video. As previously described, cyclically upsampling consecutive frames (e.g., instead of upsampling multiple frames in parallel) may provide lower latency and thus result in low-latency applications. In some aspects, the computing device (or a component thereof) may reuse previous diffusion latencies of the diffusion machine learning model, reuse at least one sampling step between time steps of the diffusion machine learning model, and/or skip one or more sampling steps between time steps of the diffusion machine learning model. As described above, reusing such information may compensate for the delay in performing the cyclic/sequential processing.

In some instances, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and/or receive data, any combination thereof, and/or other component(s). The network interface may be configured to communicate and/or receive Internet Protocol (IP)-based data or other types of data.

Components of a computing device may be implemented as circuitry. For example, the components may include and/or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or other suitable electronic circuits), and/or may include and/or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

Process (800) is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. With respect to computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be limiting, and any number of the described operations may be combined in any order and/or in parallel to implement the processes.

Additionally, process (800) and/or any other process described herein may be performed under the control of one or more computer systems comprised of executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

FIG. 9 illustrates an exemplary computing device architecture (900) of an exemplary computing device that may implement various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. Components of the computing device architecture (900) are shown as electrically communicating with one another using a connection (905), such as a bus. The exemplary computing device architecture (900) includes a computing device connection (905) that couples various computing device components to the processor (910), including a processing unit (CPU or processor) (910), and computing device memory (915), such as read-only memory (ROM) (920) and random-access memory (RAM) (925).

The computing device architecture (900) may include a cache of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor (910). The computing device architecture (900) may copy data from the memory (915) and/or storage device (930) to the cache (912) for rapid access by the processor (910). In this manner, the cache may provide a performance boost by avoiding processor (910) delays while waiting for data. These and other engines may control, or be configured to control, the processor (910) to perform various actions. Other computing device memories (915) may also be available for use. The memory (915) may include a number of different types of memory with different performance characteristics. The processor (910) may include any general purpose processor and hardware or software services, such as Service 1 (932), Service 2 (934), and Service 3 (936), stored in a storage device (930) configured to control the processor (910), as well as special purpose processors in which software instructions are incorporated into the processor design. The processor (910) may be a self-contained system including multiple cores or processors, a bus, a memory controller, a cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing device architecture (900), the input device (945) may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, and the like. The output device (935) may also be one or more of a number of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, and the like. In some cases, multimodal computing devices may enable a user to provide multiple types of input to communicate with the computing device architecture (900). The communication interface (940) may generally control and manage user input and computing device output. It is not limited to operating with any particular hardware arrangement, and thus, the basic features herein may be readily replaced with improved hardware or firmware arrangements as they are developed.

The storage device (930) may be a non-volatile memory and may be a hard disk or other types of computer-readable media capable of storing data accessible by a computer, such as magnetic cassettes, flash memory cards, solid-state memory devices, digital versatile disks, cartridges, random access memories (RAMs) (925), read-only memories (ROMs) (920), and hybrids thereof. The storage device (930) may include services (932, 934, 936) for controlling the processor (910). Other hardware or software modules or engines are contemplated. The storage device (930) may be coupled to the computing device connection (905). In one aspect, a hardware module that performs a particular function may include a software component stored on a computer-readable medium in association with necessary hardware components, such as the processor (910), the connection (905), the output device (935), etc., to perform the function.

Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) that includes or is coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to a single optical projector, aspects of the present disclosure are applicable to devices having any number of optical projectors and are therefore not limited to specific devices.

The term "device" is not limited to one or a specific number of physical objects (e.g., one smartphone, one controller, one processing system, etc.). As used herein, a device can be any electronic device having one or more parts capable of implementing at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to a number of components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

To provide a thorough understanding of the aspects and examples provided herein, specific details are provided in the above description. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in methods implemented in software, or functional blocks comprising combinations of hardware and software. Additional components other than those depicted in the drawings and/or described herein may be used. For example, circuits, systems, networks, processes, and other components may be depicted as components in block diagram form so as not to obscure the aspects with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be depicted without unnecessary detail so as to avoid obscuring the aspects.

Individual aspects may be described as processes or methods, as depicted above, in the form of a flowchart, flow diagram, data flow diagram, architecture diagram, or block diagram. While a flowchart may depict operations as a sequential process, most operations may be performed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates when the processor's operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function's return to the calling function or the main function.

The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or available from computer-readable media. Such instructions may include, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or processing device to perform, or otherwise configure, a particular function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions, such as assembly language, firmware, source code, etc.

The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions(s) and/or data. Computer-readable media can include non-transitory media on which data can be stored and which do not include carriers and/or transitory electronic signals that are propagated wirelessly or via wired connections. Examples of non-transitory media can include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memory or memory devices, magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, compact discs (CDs) or digital versatile disks (DVDs), or any suitable combination thereof, among others. A computer-readable medium may store code and/or machine-executable instructions that may represent a procedure, function, subprogram, program, routine, subroutine, module, engine, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

In some aspects, computer-readable storage devices, media, and memories may include wireless signals or cables, including bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored on a computer-readable or machine-readable medium. The processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be implemented as peripherals or add-in cards. Such functionality may also, as a further example, be implemented on a circuit board between different processors or different chips executing on a single device.

Commands, media for conveying such commands, computing resources for executing the commands, and other structures for supporting such computing resources are exemplary means for providing the functions described in this disclosure.

In the foregoing description, aspects of the present application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that the present application is not limited thereto. Accordingly, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the concepts of the present invention may be variously implemented and employed, and the appended claims are intended to encompass such variations, except as limited by prior art. The various features and aspects of the application described above may be utilized individually or jointly. Additionally, the aspects may be utilized in any number of environments and applications other than those described herein, without departing from the broader spirit and scope of the present disclosure. Accordingly, the present specification and drawings are to be considered illustrative, rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be recognized that, in alternative embodiments, the methods may be performed in an order different from that described.

Those skilled in the art will understand that the less than (“<”) and greater than (“>”) symbols or terms used herein do not depart from the scope of this detailed description, and are not limited to (“ ") and above(" ") can be replaced with symbols, respectively.

When components are described as being "configured" to perform a particular action, such configuration may be accomplished, for example, by designing electronic circuitry or other hardware to perform the action, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the action, or by any combination thereof.

The phrase "coupled to" refers to any component that is physically connected, directly or indirectly, to another component, and/or any component that is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection, and/or other suitable communication interface).

Claim language or other language reciting "at least one of" and/or "one or more of" a set indicates that one member of that set or multiple members of that set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In other examples, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and/or "one or more of" a set does not limit the set to the items listed in that set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

Claim language or other language that recite "at least one processor configured to," "at least one processor being configured to," "one or more processors configured to," "one or more processors being configured to," etc., indicates that one processor or multiple processors (in any combination) can perform the associated action(s). For example, claim language that states "at least one processor configured to do X, Y, and Z" can mean that a single processor can be used to perform actions X, Y, and Z; or that multiple processors work together to perform X, Y, and Z, each of which is tasked with a particular subset of actions X, Y, and Z; or that a group of multiple processors work together to perform actions X, Y, and Z. In another example, claim language that states "at least one processor configured to do X, Y, and Z" can mean that any single processor can perform only at least a subset of actions X, Y, and Z.

When referring to one or more elements that perform functions (e.g., steps of a method), a single element may perform all of the functions, or more than one element may collectively perform the functions. When more than one element collectively performs functions, each function need not be performed by each of those elements separately (e.g., different functions may be performed by different elements) and/or each function need not be performed entirely by just one element (e.g., different elements may perform different sub-functions of the function). Similarly, when referring to one or more elements that are configured to cause another element (e.g., a device) to perform functions, a single element may be configured to cause the other element to perform all of the functions, or more than one element may be configured to collectively cause the other element to perform the functions.

When referring to an entity (e.g., any entity or device described herein) that performs functions or is configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively) to perform the functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and/or any combination thereof. In reference to an entity that performs functions, the entity may be configured to cause one component to perform all of the functions, or to cause more than one component to collectively perform the functions. When an entity is configured to have more than one component collectively perform functions, each function need not be performed by each of these components separately (e.g., different functions can be performed by different components) and/or each function need not be performed entirely by just one component (e.g., different components can perform different sub-functions of the function).

The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, engines, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices, which have numerous uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together as an integrated logic device or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized, at least in part, by a computer-readable data storage medium comprising program code comprising instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM), synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. Additionally or alternatively, the techniques may be realized, at least in part, by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as radio signals or waves.

The program code may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other comparable integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Accordingly, the term "processor," as used herein, may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

Exemplary aspects of the present disclosure include:

Aspect 1. A method for processing image data of a current frame, comprising: determining an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; warping a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having a second resolution, wherein the second resolution is higher than the first resolution; and using a diffusion machine learning model, processing a noise frame, the current frame, and the warped previous frame to generate an output frame having the second resolution.

Aspect 2. A method according to aspect 1, further comprising the step of warping a previous noise frame based on the determined optical flow to generate a warped noise frame, wherein the output frame is generated further based on processing the warped noise frame using a diffusion machine learning model.

Aspect 3. A method according to either aspect 1 or aspect 2, wherein the optical flow comprises an individual motion vector for each pixel of the current frame.

Aspect 4. The method of aspect 3, wherein the step of warping the second previous frame comprises adjusting each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow.

Aspect 5. A method according to any one of aspects 1 to 4, wherein the output frame is an upsampled version of the current frame.

Aspect 6. A method according to any one of aspects 1 to 5, wherein the second previous frame is an upsampled version of the first previous frame.

Aspect 7. A method according to any one of aspects 1 to 6, wherein the first previous frame is a frame immediately preceding the current frame in the video.

Aspect 8. A method according to aspect 7, further comprising the step of sequentially processing a plurality of consecutive frames of the video.

Aspect 9. A method according to any one of aspects 1 to 8, further comprising at least one of the steps of reusing a previous diffusion latency of the diffusion machine learning model; reusing at least one sampling step between time steps of the diffusion machine learning model; or skipping one or more sampling steps between time steps of the diffusion machine learning model.

Aspect 10. A method according to any one of aspects 1 to 9, wherein the noise frame is sampled from a Gaussian noise distribution.

Aspect 11. A device for processing image data of a current frame, comprising: at least one memory configured to store the image data; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: determine an optical flow between a current frame having a first resolution and a first previous frame having the first resolution; warp a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and process the noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution.

Aspect 12. A device according to aspect 11, wherein the at least one processor is configured to generate an output frame by warping a previous noise frame based on the determined optical flow to generate a warped noise frame; and further processing the warped noise frame using a diffusion machine learning model.

Aspect 13. A device according to any one of aspects 11 or 12, wherein the optical flow comprises an individual motion vector for each pixel of the current frame.

Aspect 14. The device of aspect 13, wherein, to warp the second previous frame, the at least one processor is configured to adjust each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow.

Aspect 15. A device according to any one of aspects 11 to 14, wherein the output frame is an upsampled version of the current frame.

Aspect 16. A device according to any one of aspects 11 to 15, wherein the second previous frame is an upsampled version of the first previous frame.

Aspect 17. A device according to any one of aspects 11 to 16, wherein the first previous frame is a frame immediately preceding the current frame in the video.

Aspect 18. A device according to aspect 17, wherein the at least one processor is configured to sequentially process a plurality of consecutive frames of the video.

Aspect 19. A device according to any one of aspects 11 to 18, wherein the at least one processor is configured to at least one of: reuse a previous diffusion latency of the diffusion machine learning model; reuse at least one sampling step between time steps of the diffusion machine learning model; or skip one or more sampling steps between time steps of the diffusion machine learning model.

Aspect 20. A device according to any one of aspects 11 to 19, wherein the noise frame is sampled from a Gaussian noise distribution.

Aspect 21. A non-transitory computer-readable storage medium comprising stored instructions, the instructions, when executed by at least one processor, causing the at least one processor to perform operations according to any one of aspects 1 to 10.

Aspect 22. A device for processing image data, comprising one or more means for performing operations according to any one of aspects 1 to 10.

Claims (30)

As a device for processing image data of the current frame,
At least one memory configured to store the image data; and
At least one processor coupled to at least one memory, wherein the at least one processor comprises:
Determining the optical flow between the current frame having the first resolution and the first previous frame having the first resolution;
Warping a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and
A device for processing image data of a current frame, the device configured to process a noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution. In the first paragraph, the at least one processor,
Generating a warped noise frame by warping the previous noise frame based on the determined optical flow; and
A device for processing image data of a current frame, configured to generate the output frame further based on processing the warped noise frame using the diffusion machine learning model. A device for processing image data of a current frame, wherein the optical flow comprises an individual motion vector for each pixel of the current frame. A device for processing image data of a current frame, wherein in the third aspect, to warp the second previous frame, the at least one processor is configured to adjust each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow. A device for processing image data of a current frame, wherein the output frame is an upsampled version of the current frame. A device for processing image data of a current frame, wherein the second previous frame is an upsampled version of the first previous frame. A device for processing image data of a current frame, wherein the first previous frame is a frame immediately preceding the current frame in a video. A device for processing image data of a current frame, wherein the at least one processor is configured to sequentially process a plurality of consecutive frames of the video. In the first paragraph,
At least one processor,
Reusing the previous diffusion latency of the above diffusion machine learning model;
Reusing at least one sampling step between time steps of the above diffusion machine learning model; or
A device for processing image data of a current frame, configured for at least one of skipping one or more sampling steps between the time steps of the diffusion machine learning model. A device for processing image data of a current frame, wherein the noise frame is sampled from a Gaussian noise distribution in the first paragraph. As a method of processing image data of the current frame,
A step of determining an optical flow between the current frame having a first resolution and a first previous frame having the first resolution;
A step of generating a warped previous frame having a second resolution by warping a second previous frame having a second resolution based on the determined optical flow, wherein the second resolution is higher than the first resolution; and
A method for processing image data of a current frame, comprising the step of processing a noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution. In Article 11,
Further comprising the step of generating a warped noise frame by warping a previous noise frame based on the determined optical flow,
A method for processing image data of a current frame, wherein the output frame is generated further based on processing the warped noise frame using the diffusion machine learning model. A method for processing image data of a current frame, wherein the optical flow comprises an individual motion vector for each pixel of the current frame. A method for processing image data of a current frame, wherein the step of warping the second previous frame comprises the step of adjusting each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow. A method for processing image data of a current frame, wherein the output frame is an upsampled version of the current frame. A method for processing image data of a current frame, wherein the second previous frame is an upsampled version of the first previous frame. A method for processing image data of a current frame, wherein the first previous frame is a frame immediately preceding the current frame in a video. A method for processing image data of a current frame, further comprising the step of sequentially processing a plurality of consecutive frames of the video in the 17th paragraph. In Article 11,
A step of reusing the previous diffusion latency of the above diffusion machine learning model;
a step of reusing at least one sampling step between time steps of the diffusion machine learning model; or
A method for processing image data of a current frame, further comprising at least one step of skipping one or more sampling steps between the time steps of the diffusion machine learning model. A method for processing image data of a current frame, wherein the noise frame is sampled from a Gaussian noise distribution in claim 11. A non-transitory computer-readable storage medium containing stored instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to:
Determine the optical flow between a current frame having a first resolution and a first previous frame having the first resolution;
Warping a second previous frame having a second resolution based on the determined optical flow to generate a warped previous frame having the second resolution, wherein the second resolution is higher than the first resolution; and
A non-transitory computer-readable storage medium for processing a noise frame, the current frame, and the warped previous frame using a diffusion machine learning model to generate an output frame having the second resolution. In paragraph 21, the instructions, when executed by the at least one processor, cause the at least one processor to:
Warping the previous noise frame based on the determined optical flow to generate a warped noise frame; and
A non-transitory computer-readable storage medium that generates the output frame further based on processing the warped noise frame using the diffusion machine learning model. A non-transitory computer-readable storage medium in claim 21, wherein the optical flow includes individual motion vectors for each pixel of the current frame. A non-transitory computer-readable storage medium, wherein, in order to warp the second previous frame, the instructions, when executed by the at least one processor, cause the at least one processor to adjust each pixel of the second previous frame by an amount indicated by each individual motion vector of the optical flow. A non-transitory computer-readable storage medium in claim 21, wherein the output frame is an upsampled version of the current frame. A non-transitory computer-readable storage medium in claim 21, wherein the second previous frame is an upsampled version of the first previous frame. A non-transitory computer-readable storage medium in claim 21, wherein the first previous frame is a frame immediately preceding the current frame in the video. A non-transitory computer-readable storage medium in claim 27, wherein the instructions, when executed by the at least one processor, cause the at least one processor to sequentially process a plurality of consecutive frames of the video. In paragraph 21, the instructions, when executed by the at least one processor, cause the at least one processor to:
Reusing the previous diffusion latency of the above diffusion machine learning model;
Reusing at least one sampling step between time steps of the above diffusion machine learning model; or
A non-transitory computer-readable storage medium that causes at least one of skipping one or more sampling steps between the time steps of the diffusion machine learning model. A non-transitory computer-readable storage medium in claim 21, wherein the noise frame is sampled from a Gaussian noise distribution.