Turns out it was the PCI-e slot my GPU was plugged into. I swapped it from my top PCI-e Gen3 slot to my bottom PCI-e Gen2 slot and the issues are GONE. However, there is a small noticeable decline in performance due to it being a previous gen so this is not ideal.