Materialistic: Selecting Similar Materials in Images

The Big Picture

Imagine walking into a furniture store and pointing to a chair, asking the salesperson to find everything else made from the same wood. You wouldn’t be confused by different lighting, the sheen of varnish, or a lamp’s shadow. You’d just know that table shares the same grain. Computer vision researchers have been trying to teach machines the same trick for decades, and it turns out to be very hard.

The challenge isn’t recognizing a material category like “wood” or “metal.” It’s recognizing this specific wood, with its grain, color, and reflective character, versus that other wood across the room, while ignoring how dramatically lighting changes a surface’s appearance. Shading, specular highlights, and cast shadows transform how a material looks without changing what it is. Standard color-picking tools and AI labeling systems fall apart under these conditions.

Materialistic, from researchers at MIT and Adobe Research, reframes the problem entirely.

Key Insight: By treating material selection as a similarity problem rather than a classification problem, Materialistic identifies matching materials without being fooled by lighting effects, and generalizes to real photographs despite training entirely on synthetic scenes.

How It Works

The core design decision is almost philosophical: don’t classify materials into fixed buckets. Traditional segmentation systems label pixels as “wood,” “metal,” or “fabric,” but coarse labels can’t distinguish two different woods in the same scene. Materialistic asks a different question: given a user-clicked pixel, which other pixels are made of the same stuff?

To answer this, the system builds on DINO (self-distillation with no labels), a pretrained visual model that recognizes meaningful structure in image patches without human annotation. Raw DINO features mix color, texture, shape, and object type together. Materialistic adds a specialized layer on top to tease them apart.

The pipeline works in three steps:

1. Feature extraction: DINO processes the image at multiple scales, producing compact numerical summaries for each image region.
2. Cross-Similarity Feature Weighting: A novel module takes the query pixel’s summary and reweights features across the entire image, asking which properties are most diagnostic for matching the query. This operates at multiple resolutions simultaneously.
3. MLP scoring head: A lightweight multilayer perceptron takes the modulated, multi-scale summaries and outputs a per-pixel similarity heatmap indicating which regions share the same material.

Training data posed its own problem. Real photographs rarely carry per-pixel material identity labels. The team sidestepped this by building a synthetic dataset: 50,000 rendered images from 100 indoor scenes, drawing on 16,000 physically-based materials. The renderer produced realistic shading, reflections, and shadows while providing exact ground-truth material labels for every pixel.

Despite training only on synthetic indoor scenes, the model generalizes to real-world photographs, including outdoor images with entirely different lighting and materials. That synthetic-to-real transfer suggests the Cross-Similarity module is picking up on genuine material properties rather than overfitting to rendering artifacts.

No ground-truth material selection benchmark previously existed, so the team hand-annotated 50 real photographs to create one. They also ran controlled ablation studies, varying lighting angle, material glossiness, query pixel location, and image resolution to isolate what each component contributes.

Why It Matters

Material selection sounds niche, but it reaches further than you’d think. The authors show material editing (swap all the upholstery in a scene at once), in-video selection (pick a material in one frame and track it through a whole clip), and image retrieval (search product photos for objects made from a specific material). Think e-commerce, architectural visualization, digital content creation, film production.

There’s a deeper payoff too. An image isn’t just a pixel grid; it’s the result of geometry, illumination, and materials interacting through the physics of light. Systems that pull apart these components advance inverse rendering (recovering 3D scene properties from 2D photos), physically accurate image synthesis, and AI that reasons about physical properties rather than pixel statistics.

Open questions remain. The method operates at the patch level, which limits spatial precision at fine material boundaries. Highly transparent or translucent materials, where light behavior gets especially complex, are still difficult. Synthetic-to-real generalization works well here, but AI-generated image augmentation might help close the remaining domain gap on harder cases.

Bottom Line: Materialistic shows that material identity can be extracted from photographs without semantic labels, using a query-driven similarity architecture over self-supervised features. It’s a concrete step toward AI that understands the physical world, not just its appearance.