I've been struggling with this problem on many tested Guest OSs like Ubuntu, Windows XP and Windows 7 (so the issue in my case wasn't related to Ubuntu nor Linux at all) and after properly configuring that setting, I was able to choose larger desired resolution in my Guest OS (of course, you should previously install VBox Guest Additions as it is stated in several answers).