Go to the application's bundle (e.g. X11.app) and open the bundle contents. Navigate to Info.plist, open it with a text editor or plist editor. Add or change an item with the key "NSUIElement" and set its value to "1" (type = string). On relaunch of the application, its icon will not appear in the dock.
This is usefull for apps like X11 which are servers and should be running in the background.