Teaching and checking of constraints for surgical tray layout

We introduce an automatic agent that, through linguistic communications with a human, can check constraints for object layouts as well as learn new constraints. This is applied to a surgical instrument preparation task, where the system interacts with an expert to learn both general instrument arrangement regulations and special layouts customized to a particular doctor. When supervising a non-expert human preparing the tools for a surgery, the system can provide suggestions by giving verbal feedback about how to improve the setup while simultaneously highlighting corresponding objects through a graphical interface.