Special care has to be taken to seal the crankcase; for example, the area around the distributor, and the dipstick tube, are common sources of crankcase vacuum leaks, even though they may not leak oil during normal operation. Also usually a double lip seal or a reversed seal, in the rear main and front seal positions, needs to be used. When running high vacuums on a standard wet sump oiling system the vacuum in the crankcase will cause oil pressure to decrease. I have also heard that engines running a lot of vacuum tend to starve the wristpins for oil, but I don't really know if that is true or not. I've usually limited crankcase vacuum to 10-15 inches when running a wet sump oiling system.
Running 10"-15" of vacuum picked up the 510" FE in my Mach 1 from 10.60 to 10.45, with no other changes. That's probably about 20 HP, a very worthwhile modification. This was also with a standard tension ring package; I had no thoughts of running a vacuum pump when I built that engine. Probably more to be gained with a light tension ring pack, in terms of reduced frictional losses while still maintaining a good ring seal with the crankcase vacuum.