Straight up doesn't mean dot-to-dot, it means LSA=ICL=ECL, however, I know that isn't what you were asking.
It matters when it matters. I have seen some be 4 degrees off, with good parts, and some early non FE Fords far worse when someone slips a retarded timing set in. So the first thing degreeing does for you is make sure you have the right parts, and the small tolerance window on every part involved, doesn't put you where you shouldn't be
As far as installing 4 degrees advanced versus straight up, depending on the combo, you can have a significant difference in idle vacuum, throttle response, drivability. In fact, back in 2000 I bought a new Dodge Ram to replace my 1995 Dodge Ram. Engines were about the same, but the new one was a pig., turns out they retarded the cam to change how it behaved for emissions. Upon the advice of RJP, a member of this forum, I advanced the cam and the torque came back
On the first version of my Mustang engine, I had low vacuum, 300/250 @ .050 108 LSA on 108 (straight up). The car ran good, but was very soft down low, I advanced the cam 4 degrees to 105 (it actually came in a degree late as originally installed) and vacuum came up from 7 to 11 inches and throttle response was better. Now that was a young man's 427, P-sonic, 850 Holley, TRW pistons in the hole, and too big of a cam, but it did help
Hope this helps