Personally, I would let this go. I have been out shooting numerous times at the back of my house and aircraft have flown over but it doesn't mean I want to have a pop at them. If you report this, the chap can simply say he was out shooting vermin. He was probably enjoying his hobby of shooting as you were enjoying yours of flying. And from 500ft I wouldn't think it was possible to say if a gun was pointing at you or not.
However, if you are adamant it was pointing directly at you, then you will decide what is best.
I am a farmer, gun holder and paramotorist, so I can understand all angles of this.