2D, 3D and Surface Texture Analysis and Synthesis

Texture synthesis has been widely recognized as an important research topic. In this talk, I present a series of graphics or vision related projects that focus on texture analysis and synthesis. These projects cover 2D texture synthesis, 3D texture reconstruction, tensor-based BTF compression, static surface texture synthesis and dynamic surface flow simulation. Specifically, in 2D texture synthesis, we have introduced a new technique called feature-based synthesis which can produce superior results on aperiodic structural textures. It is based on the observation that the human visual system is most sensitive to edges, corners, and other high-level features in images. To produce high-fidelity texture results, we detect curvilinear features and explicitly match these features using a generalized distance metric including the Hausdorff distance.

Regarding 3D texture analysis, we have developed methods for recovering the geometric details of 3D textures by exploiting shadows. We introduced a new concept called shadow graphs which give a novel graph-based representation for shadow constraints. Shadow graphs provide a much simpler and more systematic approach to represent and integrate shadow constraints from multiple images. To recover 3D textures from a sparse set of images, we developed an optimization method for integrating shadow and shading constraints. The rendered synthetic images of the recovered 3D textures are comparable to the original photographs.

In terms of BTF compression, we developed an out-of-core tensor approximation algorithm. Traditionally, each multi-dimensional data item is represented as a vector. Such a scheme flattens the data and partially destroys the internal structures established throughout the multiple dimensions. In this work, we retain the original dimensionality of the data items to more effectively exploit existing spatial redundancy and allow more efficient computation. Experimental results indicate that our techniques can not only process out-of-core data, but also achieve higher compression ratios and quality than previous methods.

Following the Poisson-based mesh editing technique jointly developed with MSR Asia, we have further applied the basic ideas to surface flow simulation and surface texture synthesis from multiple sources. I will briefly talk about our techniques and results on these topics. In particular, the surface flow project is for inviscid fluid simulation over triangle meshes. It can enforce incompressibility on closed surfaces by utilizing a discrete formulation of the Poisson equation.

This is joint work with Narendra Ahuja, John Chang, Lin Shi, Hongcheng Wang, and Qing Wu.