Just a technique, but I always go in from the front, I just take a length out of my tool to make it shorter and put the cone on the bench
However, what I typically do to make sure I am lining both holes up is mark a line where "perfect" will be on the block, then I mark my two lines on the OD of the bearing at the center of the slots. Then just line up the lines. In your case, one of those lines could be an arrow too
Then drive it in from the front, they go straight easily if you are careful, and by using the lines, even if it turns a little on you, you are still within the range of the oil slots and you know the first time that you have both oil holes lined up.
BTW, it took longer for me to type this than make the 4 marks with a sharpie!
Also, I have driven them in and out before, no issues, a little bit of Green Loctite may make you sleep better as Brent said, and I agree, but don't go crazy, it does expand, so you don't need a lot