Manually editing silences out of your videos can be time consuming, but is definitely worth it. The easiest way is to have an editor that shows the audio waves visually so that you can cut out sections where the waves seem lower or non existent. If you do not have an editor that shows audio waves, you can still remove silences from your video by playing the video through and splitting sections of silence out as you watch.