I've built a couple of engines that run with vacuum pumps on the track, and without them on the street. I've also done some experiments with vacuum pumps on engines that were not really built to take advantage of them with special ring packages. In back to back testing, either on the dyno or at the track, the vacuum pump provides a horsepower increase.
The engines that were designed to be run without a vacuum pump typically had 1/16 - 1/16 - 3/16 ring packages that were standard tension, because that is what I typically use for this type of engine. On the 510" FE in my 69 Mach 1, back to back testing at the track with and without the vacuum pump resulted in a 0.15 second lower ET when the vacuum pump was used.
The engines that were designed specifically for use with the vacuum pump usually had .043" -.043" - 3mm ring packages, with medium or low tension oil rings. These engines, when run on the street, would typically burn a quart of oil every 300-400 miles due to the thin, low tension ring package. Note that on the street, I never ran these engines with the vacuum pump connected; the oil breather can that the vacuum pump blows into would get filled too quickly to make this practical, so I only ran the vacuum pump at the track, even though the engines were designed to use it. On the dyno my big SOHC engine picked up 30 HP with 15" of crankcase vacuum, as compared to without the vacuum pump installed. The low tension rings themselves probably also picked up power, by reducing the frictional losses in the engine, but there was no way to really get any data on that since the engine never went together with a standard ring package.
Hope that helps - Jay