It sounds like there is a mismatch between the ps_init.tcl that gets run when you run your software via JTAG and the FSBL in your boot.bin.
How did you create the boot.bin file? What board are you targeting?
I tested USB in bare metal mode with PS-only reference to https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842047/Zynq+Ultrascale+MPSoc+Standalone+USB+device+driver
It's working fine in JTAG mode.
However, I generated a BOOT.BIN file, copy to a SD card and change the switch to SD mode. It doesn't work. Can anyone help me?