Well first, it was 7 years, a term no one has done before or since. He was way past the line of being forever associated with DW. Second, a new production crew came in that he didn't particularly get along with. By his 4-6 years, he had established his position in the production and had as much input as he wanted. He supposedly became notoriously difficult to work with, ignoring direction and writers if he felt he wasn't getting his way. In Season 18 (Leisure Hive through Logopolis), new producer John Nathan-Turner and script editor Chris Bidmead kind of brought that to a halt, raising tensions. I think by that time, Baker felt he just wasn't having fun any more (and you can see in Season 18 that it started to affect his health somewhat). So it was time to move on.
He returned to theater work, did the occasional TV series or film, and in many ways became Shatneresque as a personality and brand of his own (like his VO in Little Britain). He's really picky about what work he does accept. I don't think he ever felt DW was below him, he just had his own standards for it. I think he did do a Children in Need thing in the early 90s as the Doctor, and he recently did his first Big Finish audio as the Doctor. But The Five Doctors? It was too soon, and as I said, he wouldn't be front and center.
Robert - Cleveland, OH