we also have tested the POB-Client on a Windows Terminal Server with relatively poor performance. But I can state that the performance of the 1.4 Release has been increased. Also one can adjust the window style by disabling some features like window previews. They cost performance as well. Have you tried a mass maintenance of CI? I explore performance problems also here.
Another thing to regard is the increasing memory consumption of the client when working as admin (working on different windows at a time) and the much worser performance when working with VPN-Connection to the POB-Server.
We are now hoping that the PTW (Web-Client) will be improved in performance dramatically (is not done yet on my opinion)