THPS2 didn't have caveman and you setup the controller in the game.
Only PS2 players seem to have this issue no one else seems to care. It's the PC version 1 button caveman.
In my opinion I find one button caveman superior to the 2 button method found on PS2, its much more responsive.
There isn't a fix and to be honest I don't think the PC modding community has the urge to do this.
We tried a few methods with either no or simply ill effects and since we don't see it as a problem, well at least this is how I feel about it,
we don't care to strip down an entire game looking for code that may or may not exist.
Short answer, no, there isn't a fix but CWalker used xpadder and came close to getting the controls to be more like the PS2.
Read about it here