Unfortunately, there is no workaround for disabling the four-finger swipe when Remote Desktop is active, apart from disabling it entirely in System Preferences > Trackpad, which doesn't seem to be what you are looking for. As I said, that would be understood as a feature request, and I'm not even sure that it is doable on the API level.