Turns out it was the booster. I had assumed that because it was rebuilt that it was good. Apparently the front seal was leaking allowing vacuum to counteract...