The performance on reads is negligible, but on average you are cutting your write performance by half or more. There is some evidence that leaving a free empty partition (i.e. giving the wear leveling algorithms, which factor into performance, room to work with) has a huge positive benefit, but TRIM does leak data and can theoretically be used to compromise an encrypted partition by someone with enough resources.