No, thats not going to work. You need to report each incident to the police as it happens. They might not do anything about it, but it creates a profile of the relationship between you and this person. If anything significant happens later, all the previous times he has assaulted/intimidated you and your brother are on file, demonstrating a clear pattern of behaviour.
Simply keeping the notes yourself isn't sufficient, since those notes can't be taken at face value. As far as evidence goes, they have no value.
Can I ask what disability you and your brother share?
Personally, I don't think you should have to change your behaviour to accomodate this bully. I don't see why you should have to avoid the places and people you enjoy while he doesn't. Simply be assertive, stand your ground, and clearly define your boundaries.