My only experience with X11 on the Mac worked fine. Opened the X11 app, ran the xhost command, and then set the display variable on the server. After that, windows started appearing like I expected. The firewall would be the only thing I can think of as well.