At Intel, Koltun contributed to the development of
virtual reality simulators for urban autonomous driving, robots, and drones, focusing on deep reinforcement learning techniques with neural networks in virtual environments. These networks underwent trial-and-error learning in VR before being transferred to robots or drones for real-world applications. This method was applied to the
ANYmal robot, a quadrupedal machine with proprioceptive feedback in locomotion control. The studies in the domain of urban autonomous driving led Koltun's group to the development of the Car Learning to Act (CARLA) project in 2017. It is an
open-source simulator, powered by
Unreal Engine, that can be used to test self-driving technologies in realistic environments with random dangerous situations. The project was funded by the Intel Labs and
Toyota Research Institute. In 2020, inspired by
Google Cardboard, Koltun developed OpenBot along with a German scientist Matthias Müller. It is a
software stack that transforms
Android smartphones into four-wheeled robots capable of navigation, object tracking, and obstacle avoidance. The robot features a 3D-printable chassis, accommodating a controller, LEDs, a smartphone mount, and a USB cable. Koltun also contributed to further development in the fields of 3D photorealistic view synthesis and rendering. In 2021, using his work with other researchers at Intel,
Enhancing Photorealism Enhancement, a photorealism enhancement system was tested in the
Grand Theft Auto 5. Koltun co-authored a research that developed Swift, an autonomous drone system using onboard sensors that can match the performance of human world champions. The system integrates
deep reinforcement learning with real-world data, enabling the drone to perform effectively in physical environments. ==Selected works==