NVIDIA GANverse 3D app brings to life KITT from 'Knight Rider'

MANILA: The latest model of NVIDIA Research, which features generative adversarial networks (GANs) under the hood, converts two-dimensional images into three-dimensional objects for architects, designers, artists, and game developers. 

NVIDIA Research is developing a new deep learning engine that generates 3D object models from standard 2D images and will bring to life legendary vehicles like the Knight Rider's artificial intelligence (AI)-powered KITT via NVIDIA Omniverse.

The GANverse3D application, which was created by the NVIDIA AI Research Lab in Toronto, can inflate flat images into realistic three-dimensional (3D) models that can be regulated and visualized in digital environments. 

This application can assist designers, game developers, creators, and architects in conveniently adding new objects to their respective mock-ups without the need for 3D modeling, or a big budget for rendering.

For instance, an image of a vehicle can be converted into a 3D model that can move around a digital setting, complete with realistic blinkers, taillights, and headlights. 

To create a dataset for training, the researchers used a GAN to produce photos that depict a similar object from numerous perspectives. This is similar to a photographer who goes around a parked vehicle and takes multiple shots of it from various angles.

These multi-view images were fed into a rendering framework for inverse graphics, which is the method of inferring 3D mesh models from 2D images. 

GANverse3D, once trained on multi-view images, requires only one 2D image for the prediction of a 3D mesh model. This model can be utilized with a 3D neural renderer that provides developers ample control to personalize objects and switch backgrounds. 

GANverse3D can also be utilized for the reconstruction of any 2D image into 3D when transported as an extension in the NVIDIA Omniverse platform, such as the cherished crime-fighting vehicle KITT from the well-known 1980s Knight Rider TV show. 

The research behind GANverse3D will be shown at the two upcoming conferences, namely, the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June. 

This is where a trained GANverse3D application can be utilized to transform vehicle images into a 3D figure that can be animated and personalized in Omniverse. 

For the recreation of the KITT, the researchers were able to feed the trained model with a photo of a vehicle, allowing GANverse3D to predict a specific 3D textured mesh, together with various components of the car like the headlights and wheels. 

They also utilized the NVIDIA PhysX and NVIDIA Omniverse Kit to transform the predicted texture into high-grade materials that provide the KITT a more realistic vibe and appearance, and positioned it in a dynamic driving sequence along with other vehicles. 

In the acquisition of multi-view images from real-world information, such as vehicle images that are publicly accessible on the web, the NVIDIA researchers resorted to a GAN model, controlling its neural network layers to transform it into a data generator. 

The team discovered that the opening of the first four layers of the neural network and allowing the remaining 12 to freeze caused the GAN to render images of a similar object from various perspectives. 

Through the manual assignment of standard viewpoints, with vehicles photographed at a certain elevation and camera distance, the researchers could quickly produce a multi-view dataset from single 2D images. 

The final model, which is trained on 55,000 vehicle images that are GAN-produced, surpassed an inverse graphics network that was trained on the well-known Pascal3D dataset. 

Photos from NVIDIA

Also read: Volvo Cars, other automakers use newest NVIDIA Drive innovations for autonomous vehicles