DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance (2025)

Image generation

Generative Adversarial Networks (GANs)(Goodfellow etal., 2014) have demonstrated remarkable capabilities in producing high-fidelity images. Subsequent research has focused on GAN improvements such as training stability (Kodali etal., 2017; Karras etal., 2018), attribute disentanglement (Karras etal., 2019), conditional controllability (Li etal., 2021; Park etal., 2019), and generation quality (Karras etal., 2020, 2021). GAN can be used in various applications including text-to-image synthesis (Reed etal., 2016b, a; Zhu etal., 2019), image-to-image translation (Isola etal., 2018; Zhu etal., 2020),video generation (Tulyakov etal., 2017), and even 3D shape generation (Li etal., 2019).

Recent advancements in text-to-image generation have been mainly driven by diffusion models (DMs) (Sohl-Dickstein etal., 2015; Ho etal., 2020; Ramesh etal., 2022). Later advancements (Song etal., 2020, 2021; Liu etal., 2022) have explored efficient sampling strategies to significantly reduce the number of required sampling steps, thereby improving image generation performance. Rombach etal. (2022) proposed Latent Diffusion Model (LDM) to perform the denoising process in learned compact latent space, enabling high-resolution image synthesis and efficient image manipulation.

Controllable generation

Integrating multi-modal controllability into a text-to-image diffusion model is crucial for creation applications.Recent research (DenisZavadski and Rother, 2023; Zhang etal., 2023; Ye etal., 2023; Hu etal., 2022c; Mou etal., 2023) has focused on lightweight multi-modal controllability without the requirements of extensive data and high computational power.Hu etal. (2022c) introduces a fine-tuning strategy using low-rank matrices, enabling domain-specific adaptation. Zhang etal. (2023) proposed ControlNet, adding spatial conditioning to diffusion models for precise generation control. Ye etal. (2023) presented a lightweight framework enhancing diffusion models with image prompts using a decoupled cross-attention mechanism.

Material generation

Guo etal. (2020) proposed an unconditional MaterialGAN for synthesizing SVBRDFs from random noise. The learned latent space facilitates efficient material estimation in inverse renderings. Zhou etal. (2022) developed a StyleGAN2-based model, conditioned by spatial structure and material category, for tileable material synthesis. These GAN-based methods show advantages in generating high-resolution and visually compelling materials. However, their diversity is constrained by the training instability of GANs and the limited range of training datasets.In procedural material generation, Guerrero etal. (2022) first introduced a transformer-based autoregressive model. Later work by Hu etal. (2023) proposed a multi-model node graph generation architecture for creating high-quality procedural materials, guided by both text and image inputs.While procedural representations are compact and resolution-independent, they are limited to stationary patterns and cannot create arbitrary styles.

In concurrent work, Vecchio etal. (2023) introduced ControlMat, a diffusion-based material generative model, capable of generating tileable materials using text and a single photograph as input. This model was trained on a synthetic material dataset comprising $126,000$ samples, derived from $8,615$ raw material graphs.While quite large in the material domain, this dataset is relatively small compared to the billions of text-image pairs used in text-to-image diffusion model training. This scale discrepancy leads to constrained diversity. Furthermore, this work only supports guidance of text and single photograph, limiting the scenarios range.

In contrast, our method significantly enhances material generation diversity through the efficient integration of pretrained diffusion models with material priors. We also provide a variety of user-friendly controls for guiding the generation process, expanding the scope and flexibility of applications.

Preliminaries

The goal of our method is to generate spatially-varying materials which are represented by the Cook-Torrance microfacet BRDF model with GGX normal distribution function (Walter etal., 2007). Specifically, we use metallic-based PBR workflow and represent surface reflectance properties as albedo map $\mathcal{P}$, normal map $\mathcal{N}$, roughness map $\mathcal{R}$, and metallic map $\mathcal{M}$.

DreamPBR is a Latent Diffusion Model (LDM)-based generative framework capable of producing diverse, high-quality SVBRDF maps under text and multi-modal guidance, as illustrated in Figure2.

The core generative module of our framework is the material Latent Diffusion Model (material LDM), which takes textual description $T$ as inputs to encode high-dimensional surface reflectance properties into a compact latent representation $z$.This representation effectively compresses complex material data and guides the SVBRDF decoder in reconstructing detailed SVBRDF maps (i.e. albedo, normal, roughness, and metallic) $S=\{\mathcal{P},\mathcal{N},\mathcal{R},\mathcal{M}\}$.Our critical observation is that while pre-trained text-to-image diffusion models can capture a wide range of natural images that fulfill the diversity needs of material generation, their flexibility often leads to less plausible materials due to the absence of material priors.Instead of training material LDM from scratch with limited material data, we opted to fine-tune a pre-trained text-to-image diffusion model with target material data. This strategy effectively tailors the model from a broad image domain to a specific material domain, ensuring both diversity and authenticity of output.

Our text-to-material framework seamlessly integrates three types of control modules to enhance material generation capabilities.First, we introduce the Pixel Control module $G_{P}$ that takes pixel-aligned inputs (e.g. sketches, masks), utilizing the ControlNet architecture (Zhang etal., 2023). It adds conditional controls into diffusion models, providing spatial guidance for material generation.Second, we use Style Control module $G_{I}$ to extract image features from the input image prompt, which are then utilized to adapt material LDM via cross attention.Third, we propose a Shape control module $G_{S}$ to generate SVBRDF maps automatically for a given segmented 3D shape. This module can leverage large language models to generate text prompts corresponding to different parts of the input shape. It also supports taking a 2D photo exemplar as additional input, enabling the generation of material maps for each segmented part, guided by the segmented 2D image.In the rest of the current section, we will dive into the key components of our framework. Section3.2 introduce our core text-to-material module that enables tileable, diverse material generation. Next, Section3.3 describes the SVBRDF decoder, responsible for reconstructing high-resolution SVBRDF maps from a unified latent space. Finally, Section3.4 discusses the Multi-modal control module, providing image and 3D control capabilities to the diffusion model.

Seamless tileable texture synthesis

Creating tileable texture maps is critical in material generation, involving meeting two requirements: a) maintenance of consistent spatial patterns and visual appearance, and b) the ability to tile textures without visible artifacts like seams and blocks.

While zero padding is the standard practice in CNNs, we found that circular padding is particularly effective for seamless content generation. We employ circular padding in all convolutional layers of our generative model for two main reasons:

Our tileable generation algorithm can serve two purposes: firstly, it can inherently produce tileable material maps without additional post-processing. Secondly, it can transform a non-tileable texture into a tileable version through an image-to-image generation pipeline, maintaining visual similarity with the original.

Training of PBR decoder

For the rendering loss, we adopt the sampling scheme proposed by Deschaintre etal. (2018b) to render nine images per material map. The images include three images rendered with independently distant light and view directions, and six images using near-field mirrored view and lighting directions. The rendering loss yields desirable SVBRDF reconstructions, achieved by encouraging the training process to focus on minimizing errors in crucial material parameters rather than treating them with equal importance.

Highlight-aware albedo decoder

As previously mentioned in Section3.2, our material LDM training utilizes the standard VAE decoder to map the latent space to the albedo map.While effective in generating plausible RGB images, this decoder tends to produce images with strong highlights, especially for shiny materials such as leather and metal.

To address this, we introduce a highlight-aware albedo decoder $\mathcal{D}_{P}$, which is finetuned on a synthetic shaded-to-albedo dataset, ensuring robust regularization to effectively minimize highlight artifacts in albedo maps. For each material sample in our SVBRDF dataset, we simulate various lighting conditions and viewpoints by randomly positioning point lights and cameras parallel to the material plane and then rendering SVBRDFs to reference shaded images by a physically-based renderer.

During training, the default VAE image encoder maps the shaded images into latent space, which are then decoded back to image space by our specialized albedo decoder. The training process for this decoder follows the original VAE loss function (Kingma and Welling, 2014).

Material super-resolution

High-resolution material maps are essential for achieving photorealistic renderings. However, due to the memory and performance constraints, current diffusion models typically generate images at a resolution of $512\times 512$, which falls short of high-quality production rendering.

We introduce a material super-resolution module comprising four super-resolution networks $SR$, each following the Real-ESRGAN architecture(Wang etal., 2021). These super-resolution networks, denoted as $SR_{P},SR_{N},SR_{R},SR_{M}$, are designed to augment the resolution of different SVBRDF property maps to $2,048\times 2,048$.

We fine-tune the Real-ESRGAN with material data, which is trained on purely synthetic data, to more effectively capture the high-frequency details of materials. We incorporate a rendering loss (similar to Equation3) into the training of the super-resolution module to ensure that the generated details contribute to high-frequency shading effects rather than visual artifacts. We should note that special care must be taken for normal maps during augmentation involving flipping and rotation. The directions stored in a normal map must be adjusted consistently with the map orientation to ensure consistent knowledge about surface normals.

3.4.1. Pixel Control

Spatial property guidance is widely used in material creation by artists. Our Pixel Control module $G_{P}$ takes spatial control maps $I_{P}$ as input, utilizing the ControlNet architecture (Zhang etal., 2023), to guide the generation of spatially-consistent SVBRDFs. It supports controlled generation under sketch guidance and allows for image-to-image material inpainting with a binary mask.

Our material LDM, as described in Section3.2, is adapted in the material domain, enabling plausible material generation controlled by pre-trained ControlNet checkpoints, which are trained with 2D supervision. However, we found that fine-tuning pre-trained ControlNet with material data significantly improves both the controllability and the quality of generated materials. Specifically, we initialize our ControlNet using the ControlNet 1.1 Scribble checkpoint and fine-tune it on our SVBRDFs dataset. To generate the sketch guidance, we employ Pidinet (Su etal., 2021) for extracting sketches from albedo maps.

3.4.2. Style Control

The Style Control module $G_{I}$ takes image prompt $I_{S}$ as input and extracts the style characteristics to guide material generation. Inspired by Ye etal. (2023), image prompts $I_{s}$ are first encoded into image features by CLIP’s image encoder, and then embedded into material LDM using a decoupled cross-attention adaptation module. Multimodal material generation can be achieved by accompanying the image prompt with a text prompt.

Style Control module can effectively capture the appearance properties and structural information from the input images, to generate realistic and coherent material maps. This functionality is particularly useful in scenarios where materials need to be created based on specific exemplar images, which is a frequent requirement in the material design industry.The interaction of the Style Control module with the Shape Control module will be detailed in Section3.4.3.

3.4.3. Shape Control

The Shape Control module $G_{S}$ takes a segmented 3D model $O_{s}=\{O,s\}$ ($s$ denotes the geometry segmentation) and an optional photo exemplar $I_{o}$ as input and automatically generates high-quality material maps for each segmentation.When provided with only a segmented 3D model and a basic text prompt, we leverage large language models(LLMs) such as ChatGPT (Achiam etal., 2023), to enrich the text descriptions for each segmentation. For instance, given a 3D chair model, the language model can generate diverse text descriptions for each part like seat, leg, and armrest, each featuring varied design styles. Furthermore, integration with existing Pixel Control and Style Control modules supports enhanced SVBRDF generation, ensuring superior quality and detailed material characteristics.

Our model integrates the material transfer pipeline TMT (Hu etal., 2022d) to automatically assign diverse generated materials to 3D shapes based on an image exemplar.The TMT pipeline involves two stages: firstly, translating color from exemplar image to the projection of 3D shape and vice versa for segmentation results; secondly, assigning materials to projected parts using a material classifier network, based on the translated image.Unlike Hu etal. (2022d), we do not rely on predefined material collections in material assignment. Instead, we use predicted material labels of TMT directly as text prompts and translated images as image prompts in the Style Control module, allowing high-quality SVBRDF generation for each part.The proposed algorithm offers two significant advantages over traditional material transfer models: it expands material diversity beyond limited predefined material collections and transfers not only color and category information but also comprehensive material attributes including styles and spatial structures from 2D exemplar to 3D shapes, leveraging the capabilities of our Style Control module.

4.1.1. Dataset Generation

Our dataset comprises a total of 711 PBR materials, each including four 2$k$ texture maps: albedo, normal, metallic, and roughness, along with corresponding textual labels. The data are sourced from PolyHaven ¹¹1https://polyhaven.com/ and freePBR ²²2https://freepbr.com/. We categorized the data into ten types manually: Brick (58), Fabric (60), Ground (99), Leather (45), Metal (130), Organic (45), Plastic (40), Tile (75), Wall (69), and Wood (90).

The input text prompt $\mathcal{T}$ is in the format of “a PBR material of [type], [name], [tags]” during the finetuning of material-LDM, where ‘type’ refers to the type of material, ‘name’ (title name) and ‘tags’ for each material are given by the website. These tags are randomly retained at a ratio of $30\%-100\%$ during training. To address the issue of uneven distribution in the original data, we selected high-quality and representative data within categories of large volumes and randomly duplicated existing data for categories with smaller data volumes, which helps to balance the sample sizes across all categories, ensuring more uniform training data distribution.

For the 2$k$ textures we obtained, we perform horizontal flipping, vertical flipping, random rotation, and multi-scale cropping and adjust the direction of the normal maps accordingly, eventually resizing them to $512\times 512$ pixels as our training data.After augmenting textures, we render each of them with randomly sampled viewpoints and lightings by Laine etal. (2020). The rendering images are also used to train our highlight-aware albedo decoder.

Concerning the paired data for training ControlNet, we utilized Pidinet (Su etal., 2021) to extract sketches from the albedo maps as mentioned in Section3.4.1.

4.1.2. Other Details

DreamPBR was trained on quadruple Nvidia RTX 3090 GPUs. During the training of Material LDM, we employ Adam as our optimizer with a base learning rate of $1.6\times 10^{-3}$ and closed learning rate scaling. Starting with the stable-diffusion-v1-5 checkpoint for 9000 epochs, we finetuned it for approximately 10 days. For the training of the PBR decoder, we set the base learning rate to $4.5\times 10^{-6}$ and enabled scale_lr, taking 4 days total in which the output channels of the decoder were set to 8, with albedo and normal having three channels each, and metallic and roughness being single-channel. For the highlight-aware albedo decoder, we set the base learning rate to $4.5\times 10^{-6}$ and enabled scale_lr, taking 2 days total in which the output channels of the decoder were set to 3. We incorporate rendering loss as detailed in Section Section3.3 during the training process above.

During the training of the Rendering-aware super-resolution module, we initially utilized the preset weights from Real-ESRGAN (Wang etal., 2021) and finetuned four super-resolution modules specifically for albedo, normal, metallic, and roughness textures. These modules were finetuned using the learning rates of $1\times 10^{-4}$ and 10000 total iter. Furthermore, we combined the training of all four modules in a model to render the result of each module during training and incorporated rendering loss.

To enhance image control performance, we set the learning rate to $1\times 10^{-5}$ for training ControlNet, which requires about 2 days to complete. For Style Control, we directly utilize the ip-adapter_sd15 checkpoint along with our finetuned checkpoint, as we have observed satisfactory results.

4.2.1. Tileable texture generation

Although the users would introduce various controls, we can generate seamless tileable textures all the time, which allows users to apply the generated textures in different scales and different scenes. In Figure8, we present several tileable textures from direct and guided generation with their splicing results, showing the effectiveness of circular padding in our method as mentioned in Section3.2.

4.2.2. Results of Pixel Control

By finetuning an additional ControlNet, DreamPBR is able to generate textures according to given patterns. In practice, a designer could decide on a pattern in advance, and then try different materials. It may also be the other way around. For those two situations, DreamPBR ensures reasonable textures for certain patterns or materials as demonstrated in Figure4 and Figure5.

With additional control of binary images, inpainting is also a usual method for users to obtain specified results so we present several inpainting results in Figure9 to replace a region in texture with another region users describe.

4.2.3. Results of Style Control

A styled image expresses more easily for a person than only text like Su etal. (2021) does. To do so, we evaluate the adaptation of Su etal. (2021) for our Style Control. Specifically, we obtain several styled images online and present the generation results under different styles from images as shown in Figure6. Figure10 illustrates the situation in that users would like to combine Style Control with Pixel Control, which enables users to generate the results they want more freely.

4.2.4. Results of Shape Control

With the ability to generate various textures, DreamPBR can be extended to non-planar objects such as chairs. Specifically by giving a segmented object, we can utilize dialogue with a large language model to get different descriptions of each region. For more specified objects, a more direct way is to be in conjunction with cropped areas from exemplar images used with pixel control and style control. Thanks to the tileable features, the results from our pipeline of Shape Control are shown in Figure11.

4.4.1. PBR Decoder

When the PBR Decoder is trained, we introduce $\mathcal{L}_{\text{render}}$ to solve the regression problem from images rendered with random lights and viewpoints, which enforces that the decoded textures are realistic after being rendered. It reduces the search space of output values compared to the one that rendering images is not used.We trained two PBR decoders with and without $\mathcal{L}_{\text{render}}$, and evaluated their effectiveness of them by comparing the outputs with reference textures. Figure14 presents the comparison results, in which our rendering-aware decoder is capable of achieving more realistic results in rendered results and more consistent results in generated textures.

4.4.2. Super-Resolution Module

Although the super-resolution models originally show great results in natural images, we finetune it again with our material data and employ a novel rendering loss $\mathcal{L}_{\text{render}}$ from the level of perception. In practice, we finetune super-resolution modules for each component of textures based on the pre-trained Real-ESRGAN as our baseline. With four single modules(albedo, metallic, normal, and roughness), we jointly finetune them and introduce the $\mathcal{L}_{\text{render}}$ by rendering four textures after super-resolution to image space.The comparison results are shown in Figure15. Similar to the training of PBR Decoder, the finetuning super-resolution modules with $\mathcal{L}_{\text{render}}$ contributes to better results.

4.4.3. Highlight-aware decoder

As mentioned in Section3.3, we introduce a highlight-aware albedo decoder to remove the potential highlights in generated RGB images.For a good de-highlight module, there are two key points to be taken into account: 1) effectively removing the highlights in images, and 2) leaving them unchanged for those without highlights. In practice, only training on rendered images potentially affects the decoded albedo(without highlights), so we finetune the highlight-aware decoder by randomly choosing rendered images from different lights or pure albedo maps. Furthermore, we compare the outputs of the highlight-aware decoder with the ones of the initially pre-trained decoder in Figure16, suggesting that our decoder addresses the issues of two key points above.

4.4.4. Pixel Control

To realize the sketch-guidance control, we embed a pre-trained ControlNet in DreamPBR. However, different from the IP-Adapter for Style Control focuses on incorporating semantics of images in clip space independent of training data, the initial ControlNet leads to domain shift, from the albedo domain back to the image domain, in our experiments. To address this problem, we finetuned the ControlNet with our sketch-albedo pairs as mentioned above. The comparison of ControlNet before and after being finetuned is shown in Figure17.