I flashed the combined DFU and usbserial HEX file using the AVRISP on the other 6 pin header and it turns into a at90usb82. Then I set it to running in Flip (Start button with reset unchecked).
At this point I installed the bootloader via the AVRISP and I can successfully install a Sketch via the bootloader. Turns out the test app I use was putting out too much over the serial port, so I went back to ISP installs rather than bootloader, but I am getting correct serial comms and I managed to install the devkit.pde