I always loved baseball and liked golf. I played baseball until the differences in my ability to play the two sports (the summer between my soph and junior years in HS) became pronounced. While that doesn't make me an expert, it does allow me to disagree with your assessment. I assume we are talking about good players capable of a good result and, in the baseball case, good pitching.
First and foremost, there is little to no fear factor in golf. Getting hit with a baseball hurts, a lot. Getting hit in the head does more than hurt. Standing in and hitting the ball hard after being "flipped" takes skill and an extra portion of courage. There is nothing like it in golf.
Golf is more forgiving. Again, assuming good players, an off-center hit in baseball is more than likely to end badly. Experience tells me that hitting a golf ball a little thin or a little fat or with a swing that's a little inside/outside the line of flight can, and often does, lead to something close to the hoped for result.
Hitting a round ball, thrown by a good pitcher, with a spherical bat is is the single most difficult athletic skill to master; especially when the good pitcher might decide to bounce one off your coconut.