Actually the consonant isn’t the hard part. はひへほ and even ふ all use the same H sound (production method) as English. Okay maybe English has a dozen sounds for any given letter but I mean the basic horrible hat hating Huns. The difference is that う doesn’t quite exist in any native English dialect. う keeps the lips relatively close together throughout the entire mora, for every mora that uses that vowel. So producing that H sound with that vowel naturally pushes air through that narrowish opening, creating that harsh friction sound which results in a sound that’s perceptibly closer to the English ‘f’ than ‘h’ even though the production method is much more like ‘hう’.
Basically where English fails to replicate Japanese, ‘fu’ is the alphabet attempting to mimic the sound, but ‘hu’ is the alphabet attempting to hint more faithfully at the the production method. Both are officially accepted romanizations and neither are fully accurate. Actually all of the mora that are grouped together by consonant, (たちてとつ, or さしそせす, etc.) are grouped like that because the entire set shares a consonant production method. The apparent exceptions like tsu (tu), chi (ti) and shi (si) are a natural result of combining their respective consonant with a particular vowel into one sound as a mora. (Mora really are one indivisible sound, not one proceeding another like in English.) They don’t really feel like exceptions unless you’re listening for an approximation based in another language or, God forbid, attempting to transcribe them into an incompatible writing system.