That was my initial plan when developing displayserver, write a VNC-server which serves the display. There are some examples of a simple server on the VNC-site. I cannot remember exactly why I switched to the java approach. I will look through some old code why it didn't work... I was probably just lazy...

Frank van Gestel