The simple answer is that, most of the time the VAT rules require you to because they're part of your service - just part of the rich tapestry of value-added tax economics.
In specific circumstances though, expenses can be treated as 'disbursements' and rebilled with no added VAT. This is where you pay money to some third party on behalf of your client and your client actually received the goods or services.
Normally, travelling and accommodation and admin expenses don't count as disbursements, since you made use of them rather than your client. Of course, if your client is another business they would just reclaim the VAT they paid on your services anyway, so neither of you lose out.
For the full rules, see the HMRC guidance.