Except of course he didn’t really turn his back on the place*. I’m pretty sure what he means was trying to get him to pay any attention at all in color theory class (or whatever) was pretty much a lost cause once he discovered the excitement of being a noisy young art student with an earring in a loud loud LOUD band.
*The place of course being the amazing
And does it even need to be said what I wouldn’t give to see what’s in Peter’s mind’s eye when he stands on those steps and remembers 1976?