You can use B mode at any time without worry. As others point out, the only caveat is that cruise is disabled in B.
As for brake wear - don't worry too much about it, because the car employs a brake blending strategy. When you press the brake pedal, the car is actually engaging stronger regen...