Is it against the rules to take a weekend trip while on business or attending an academic conference? Why

This should depend on the relevant regulations of your department. Generally, after traveling for business or attending a meeting, if you are still on vacation, you should still be able to travel.

I previously participated in an academic conference on the conference 123 platform. On the last day of the conference, participants were taken to visit scenic spots for free, so it also depends on the different arrangements of different conferences