Rui Yu
University of Louisville, KY·Computer Science
AI systems can build detailed 3D digital copies of the world, yet they often fail at simple physical common sense: will this stack of objects fall, or is this chair stable? This collaborative project with Penn State proposes that machines understand scenes better by learning a "structural skeleton," a simplified map of a scene's core geometric shapes, rather than dense pixel-by-pixel detail.
- Fields
Computer Science
Engineering
- Topics
computer vision
scene understanding
3D reconstruction
- Grants
- 1 known grant, through Jul 2030
What they work on
- Extract a labeled structural skeleton from single noisy images by fitting geometric primitives such as planes and cylinders to monocular depth estimates, using large vision-language models to assign meaning.
- Design a physics-informed reasoning module using graph neural networks on scene skeletons to predict whether object arrangements are stable, precarious, sliding or tipping.
- Build a generative framework that completes partial skeletons and models structural grammars, validating AI predictions against ground-truth physics simulations after scene changes.
More about the research
The aim is AI that is more robust and efficient when acting in the real world.
Why it matters
AI that can reason about physical stability could make robots and autonomous systems safer and more reliable. Examples include industrial robots and assistive devices that help elderly people or people with disabilities at home. The compact skeleton representation may also improve augmented and virtual reality on mobile devices.
Known grants
From NIH and NSF research grants. Labs often have other funding we can't see.
Closest labs
Yuxiong Wang
University of Illinois Urbana-Champaign, IL·Computer Science
Develop a multi-resolution video encoding strategy that adjusts token granularity to balance visual detail against efficiency in transformer-based vision-language models.
Noah Snavely
Cornell University, NY·Computer Science
Develop compositional implicit neural scene representations that pair a network's ability to capture fine detail with physical models of scene structure, such as light fields.
Lap Fai Yu
George Mason University, VA·Computer Science
Investigate how artificial intelligence techniques can be integrated with mixed reality headsets to assist first responders during tasks such as firefighting and search and rescue.
The school
University of Louisville: 15 labs on PI Pages, $225M a year on research, 159 research doctorates a year.
Reach out
A start for your email. Fill in the brackets.