How do we scale up robot learning? Data is the magic which makes deep learning work, but for robots it is very expensive to collect data in the real world. We combine Gaussian Splatting with simulators to create a pipeline which can generate very realistic looking data for robots.

I came up with the idea of using Gaussian Splatting as a renderer over Simulators and wrote most of the code. Sparsh's contribution was crucial in the early implementations of SplatSim, especially helping in coding the initial scripts for forward kinematics. However, he was away due to an internship for rest of the project. I wrote most of the paper while Sparsh made all the amazing figures for the paper. Francisco was very helpful in day to day discussions and solving system issues. Prof. David provided some very good advice on deploying policies in the real world. Prof. George and Prof. Abhi provided invaluable insights on making the SplatSim framework practical and applicable to real world robots. Their focus, especially on the practicality of solution which can be deployed to real world fields was crucial.

@misc{qureshi2024splatsimzeroshotsim2realtransfer,
        title={SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting}, 
        author={Mohammad Nomaan Qureshi and Sparsh Garg and Francisco Yandun and David Held and George Kantor and Abhishesh Silwal},
        year={2024},
        eprint={2409.10161},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2409.10161}, 
  }

How do we use foundational visiion models for robot tasks? Diffusion models are very good at generating image goals, servoing is very good at reaching them. Combining the two is a natural fit.

I came up with the formulation for this paper. Pranjali did most of the work and Gunjan did the real world robot experiments.

@misc{qureshi2024timeindexing,
  title     = {On Time-Indexing as Inductive Bias in Deep RL for Sequential Manipulation Tasks},
  author    = {M. Nomaan Qureshi and Ben Eisner and David Held},
  year      = {2024},
  eprint    = {2401.01993},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO}
}

Incorporating laws of motions like gravity into learning object manipulation skills from video, can help recover better trajectories.

This was Vladimir's idea. I worked on improving the segmentation which helped the performance of his idea -- and some help in writing the paper (related work and figures).

@misc{petrik2022learning,
  title     = {Learning Object Manipulation Skills from Video via Approximate Differentiable Physics},
  author    = {Vladimir Petrik and Mohammad Nomaan Qureshi and Josef Sivic and Makarand Tapaswi},
  year      = {2022},
  eprint    = {2208.01960},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO}
}

Obstacle avoidance amidst high rises is a challenging problem. We proposed a simple application of visual servoing to solve this problem.

Harshit came up with the problem of obstacle avoidance amidst high-rises. I came up with the core idea and wrote the paper as well as servoing code along with Shankara. Prof. Madhava was the main advisor.

@misc{sankhla2022flow,
  title     = {Flow Synthesis Based Visual Servoing Frameworks for Monocular Obstacle Avoidance Amidst High-Rises},
  author    = {Harshit K. Sankhla and M. Nomaan Qureshi and Shankara Narayanan V. and Vedansh Mittal and Gunjan Gupta and Harit Pandya and K. Madhava Krishna},
  year      = {2022},
  eprint    = {2207.03557},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO}
}

What inductive bias to use for policy learning in manipulation tasks? We found out that simple linear dynamical systems work surprisingly well when you make them differentiable.

I came up with the idea of differentiable linear dynamical systems and how they can be integrated into policy learning. I did most for this paper. Ben introduced me to dynamical systems and manipulation policy learning and was a wonderful mentor. Prof. David was the main advisor and provided invaluable feedback.

@inproceedings{qureshi2022deep,
  title     = {Deep Sequenced Linear Dynamical Systems for Manipulation Policy Learning},
  author    = {Mohammad Nomaan Qureshi and Ben Eisner and David Held},
  booktitle = {ICLR 2022 Workshop on Generalizable Policy Learning in Physical World},
  year      = {2022},
  url       = {https://openreview.net/forum?id=rF-fT4pN1Wc}
}