I am not sure about the incentive effects of websites such as ratemyprof.com, as Professors with tenure may or may not care about undergrad students. However, they may be more serious about graduate students (supervision, co-authorship etc).
Does a ratemyadvisor type website exist?
This is important (I think) because relationships between supervisor-supervisee are probably more profound (given the time investment) and have arguably more ramifications on the future (reference letters bear more weight).
Any thoughts are welcome!
Yes, there are such websites.
I know of the following two:
However, at least at this moment, these websites remain useless and mostly abandoned. Note (i) is free and (ii) is paid. This is because to, in addition to a number of issues related by other respondents, academics are remarkably overreactive to criticism. This works directly against open criticism, as public relations play a central role in defining academics careers, and petty individuals are frequently empowered enough to take revenge on others. I am sure a number of PIs would exploit (public) institutional resources just to try and tear such websites down should they take root.