It is possible to prevent the Linux system from presenting display modes that it doesn't support.
There are a couple of options you can add to your 'Screen' section in xorg.conf
Option "ModeValidation" "NoXServerModes"
Option "ModeValidation" "DoubleScanPriority"
Either of the above options should fix the problem without messing with modelines. The first disables any display modes not given as supported by the monitor. The second reverses the priority of standard vs doublescan modes for a given res. (Doublescan being the root of the problem in my case).