I think it has a lot to do with net connections.
I moved my maxclientrate to 10000 like Tysoft said, and it seems to have fixed it a bit.
Also, with my previous server settings, the CPU usage was always at 100%. When it doesn't constantly idle at max load, it also helps out. So the server has a lot to do with it me thinks.
Also, two people I know said that changing from LAN to CABLE/DSL also helped.