When I grab it it cocks the bars and wheel to the right slightly. Feels like a slight twist
Yeah, I have had that on another bike we worked on, this is a triple tree clamping problem, plain and simple.
What is happening is when you grab the brake the caliper is trying to shove the left fork ahead and twisting the forks in the trees just a bit. (you have the brake on the left side). This has been on my mind because I have the same thing, the buell brake set up I have has way more grab than the front end can handle, driving me nuts.
These bikes could benefit from an inverted front end soooo much, that's what I am going to do on my next one.
I would check and make sure the top tube is thoroughly tightened to start with, (stock trees??). The bike isn't actually pulling, that's in your head, but it can feel like it, I know. Sounds like it's bugging you as much as it bugs me...lol
Also, Zach has some valid points too.