The best way to get rid of them is to change the design of the printed object to make them unnecessary.
Instead of printing the one part with support material, the piece can be split into two or more parts which can be printed without support material and assembled after the printing.
Given that this is not always fully possible, a convenient way to get rid of additional structures is to use a different fillament for them that can be removed easily. This list of printing materials includes Polyvinyl Acetate (PVA), which is water soluble. You can wash the support material away given that your actual printign material is not water soluble. Here's a quote from the website (emphasize mine):
PVA (Polyvinyl Acetate) filament prints translucent with a slightly yellow tint and is primarily used as a 3D printing support material because it is water-soluble, meaning that it will dissolve when exposed to water (and so MUST be kept dry prior to use). PVA is most often used with 3D printers capable of dual extrusion: one extruder printing a primary material (such as ABS or PLA) and the other printing this dissolvable filament to provide support for overhanging features. PVA 3D printer filament is available in 1.75mm and 3mm.