AMD is exploring a novel approach to real-time rendering by leveraging image generation models to produce global illumination, a technique that simulates how light bounces off surfaces in a scene. The company's researchers have published a paper detailing a method that tackles global illumination as an image generation problem, aiming to create more realistic lighting without the intensive computational cost of traditional methods like path tracing.
The technique, outlined in the paper "Temporally stable generative illumination with a one-step diffusion model," utilizes a modified diffusion model, specifically Stable Diffusion 2.1 Turbo. Unlike standard diffusion models that often require numerous steps to denoise an image, AMD's researchers employ a one-step process. This efficiency is crucial for real-time applications. The model is trained on a synthetic dataset of 31,000 interior scenes rendered in Blender Cycles, exposing it to a wide variety of lighting conditions and object arrangements.
SungYe Kim, one of the paper's authors, explained that global illumination is key to a rendered scene's realism but also the most computationally expensive aspect. AMD's method uses a frame with direct lighting as an initial 'hint' for the diffusion model. Sparse signals related to lighting, radiance, material, and geometry are fed into the model to ensure the generated illumination aligns with the scene's structure and existing light sources. This focused approach aims to prevent the model from hallucinating details not present in the scene or distorting geometry, issues that can arise with other generative techniques.
To ensure visual consistency between frames, the researchers incorporate a temporal VAE (TVAE) that uses motion vectors to align the current frame with previous outputs. This history helps prevent flickering or jarring changes in lighting from frame to frame, a common challenge with generative models operating in real-time.
The resulting generated global illumination is then composited with the original frame that contains only direct lighting. According to the paper, this process is suitable for real-time rendering, with each frame's illumination step taking approximately 0.29 seconds at a 512 x 512 resolution on an RTX 3090. The process requires about 8-9 GB of VRAM. This is for the illumination step alone, not the entire rendering pipeline.
This research aligns with AMD's broader push into neural rendering technologies, as previously announced with its FSR Diamond suite. FSR Diamond is slated to include enhanced AI-based upscaling, multi-frame generation, ray regeneration, and path tracing, with neural rendering being a significant component. While Nvidia has also been investing heavily in AI for graphics, AMD's work suggests a parallel effort within the industry to explore these advanced rendering techniques. The company's strategy for integrating these AI-driven features into consumer products will be closely watched, especially given the mixed reception to some AI announcements in the graphics space.