Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild

Yanhui Guo1, Xinxin Zuo2, Peng Dai2, Juwei Lu2, Xiaolin Wu1, Li Cheng3,
Youliang Yan2, Songcen Xu2, Xiaofei Wu2,
1McMaster University, Canada, 2Noah's Ark Lab, Canada, 3University of Albert, Canada
NeurIPS 2023
Paper Supplementary Video Code

This paper introduces Decorate3D , a flexible and easy-to-use tool to create and edit 3D objects from images. Given multi-view photos of an object of interest, our approach starts with a textured mesh 3D reconstruction of the object, followed by either manual editing, or text-to-3D texture automatic generation of textures that align with the mesh. Decorate3D decomposes the Neural Radiance Field into an explicit mesh representation and a view-dependent texture, allowing for flexible decoration. One key contribution of Decorate3D is paving the way to text-driven 3D high-quality texture generation, which has remained an unresolved problem. To tackle this challenge, we propose a structure-aware score distillation sampling method to optimize a UV neural texture based on user-defined text and empower an image diffusion model with 3D-consistent generation capability. Furthermore, we introduce a few-view resampling training method and utilize a super-resolution model to obtain high-quality UV textures (2048x2048) for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in decorating real-world 3D objects.

Video

Method Overview

To facilitate the modifiability of the 3D representations, we bake the NeRF model into a triangle mesh representation with a view-dependent texture in UV space, which is called a Decomposition phase in this paper.

The decoupled geometry and UV texture representations make the subsequent Decoration phase more manageable. In the decoration phase, Decorate3D allows flexible texture editing and controllable generation of UV textures with the instruction of prompts.

Comparison with Fantasia3D and Magic3D

Magic3D
(Prompt: An ancient Chinese-style house)

Magic3D
(Prompt: Fairy house with a garden)

Fantastic3D
(Prompt: An ancient Chinese-style house)

Fantastic3D
(Prompt: Fairy house with a garden)

Ours
(Prompt: An ancient Chinese-style house)

Ours
(Prompt: Fairy house with a garden)

Decoration Results

A haunted house

Fairy house with a garden

An ancient Chinese-style house

Sneaker pained with NIPS Logo, put on a music box

A cartoon style snow boot on a shoe box made of bricks

Sneaker,put on a bamboo box

A lotion case, printed with Sunflowers

a plush teddy bear doll

A close up face sculpture of Albert Einstein

Ronald McDonald

Captain America stands on the desert

A doll model of the movie of Zootopia, wearing a Mickey Mouse suit

Husky sits on the grassland

A detergent case, printed with Pepsi-Cola packaging

A detergent case, printed with a movie poster of Avatar

A gothic-style house

A zebra running on a blanket

A detergent case, printed with a movie poster of Star Wars

A doll model of the movie of Zootopia, wearing a Spider Man suit

A man wears a cosplay suit of an armored medieval Japanese samurai

Turn him into a Joker

Ironman, with hands clasped behind back, standing up on the grass

Optimus Prime

An astronaut stands up in the milky way

A close up face sculpture of Newton

A doll model of the movie of Zootopia, dressing up like a cheerleader

Woman martin shoes, decorated with diamond, put on a jewelry box

Citation


@inproceedings{guo2023decorate3d,
    title={Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild},
    author={Guo, Yanhui and Zuo, Xinxin and Dai, Peng and Lu, Juwei and Wu, Xiaolin and Cheng, Li and Yan, Youliang and Xu, Songcen and Wu, Xiaofei},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)},
    year={2023},
}