I think you points you make are very valid Ballen, I choose not to based on personal principle.
I've seen trust broken and abuse of privilege on both sides before. But tracking is just another facet.
I also think your role in society has different expectations and requirements. All I can imagine is the benefits and safety aspects would out weight the negatives.
But the 95% of the rest of the population don't have to deal with anything remotely confronting as you would.
If an employers purpose to track is to performance manage its employees, I see ethical issues with allowing this, as it's give an inch, take a mile. There are plenty of ways to do this without knowing someone's longitude and latitude. And without good reason, is just another element to squeeze more out of people and put them over the barrel.