Take the EFIdirectory and all its contents, and put it on the USB partition.I'd suggest editing the stick version of EFI/ubuntu/grub.cfg to changethe name of the menu item "Ubuntu" to "Ubuntu on USB" or something tomake it obvious which set of files you are booting with.