Even if you're one of the 'lucky ones' - a Premier League footballer or an England (men's) cricket player on a big-money central contract - it must niggle at the back of your mind.
And for those who pursue a career in sport but don't get so lucky, either because they aren't quite good enough or because their sport doesn't pay the big bucks, that voice must be louder-still:
"What the hell am I gonna do when I reach 'That Age' where I can't kick/throw/hit the ball quite so well any more?"
As everyone who reads this blog knows by now, England spinner Holly Colvin has decided she isn't hanging around to find out - she's taking a break from the game right now (aged 24) to "establish [herself] in a career outside cricket."
And who can blame her? Not me! Holly Colvin is a super-intelligent person (4 A grades at A-level) who deserves the opportunity to build a career which will sustain her and her loved ones for life - and she'll make a success of it too, be in no doubt about that!
It would be easy to argue that this is a direct result of the failure of the ECB to sort out 'proper' central contracts for our women cricketers; but to be fair, I think in this case it is a bit more complicated than that.
This is partly because Holly has already had a longer international career (9 years) than most are able to sustain; and secondly, because she is perhaps the one player from the existing group of established team-members who has the capacity to make a lot more money doing something else - so even if she were on a contract earning (say) £50,000 (roughly what the top Australian players are earning, I believe) she would have realistic options that others probably wouldn't.
I'm sad right now. Very sad. But Holly: you gotta do what you gotta do.