Networking

Bonding

Traffic is distributed between interfaces using LACP protocol. It is recommended to use LACP on switch side too. It is also recommended to enable layer2+3 hashing for better traffic distribution.

ip link set bond0 type bond xmit_hash_policy layer3+4

collectd

Forwarding collectd metrics to another collectd instance

You need to add on destination collectd instance:

LoadPlugin network

<Plugin network>
        <Listen "LISTENIP" "25826">
                SecurityLevel Encrypt
                AuthFile "/etc/collectd/passwd"
        </Listen>
</Plugin>

In /etc/colllectd/passwd you need to add user and password for source collectd instance:

user:password
user2:password2

Source collectd instance:

LoadPlugin network

<Plugin network>
        <Server "DESTINATIONIP" "25826">
                SecurityLevel Encrypt
                Username "user"
                Password "password"
        </Server>
</Plugin>

Selecting right CPU, memory, disk, network card for your GlobalOS

CPU

Find CPU available on cpubenchmark.net website and check "Average CPU mark".

For PPPoE: You can estimate at 7000 points it might take about 4000 users. More points - more users, but growth is not linear. For example "Intel Xeon E3-1230 v3 @ 3.30GHz" has 6795 points.

For NAT: You can estimate at 7000 points it might take about 10Gbps.

Note, if you buy dual CPU server, you will not get 2x performance, it will be still significant boost, and you need to take into consideration, that each PCI-E is "local" for one CPU only, so you might need to install 2 network cards, on each CPU "side". Read about NUMA architecture and how it works.

Memory

For PPPoE it is recommended to have at least 8GB of RAM at least. Recommended to have 16-32GB, but not more. You might take into consideration some servers with too much memory might have slower memory speed. Do not use DDR3 memory/servers, it is too slow, unless you are ok with lower than expected number of users per server.

Disk

For our system it is totally fine to use USB drive, but internal SSD is preferable. System use very little of storage resources, not I/O bound, we know some USB flashes that lasted more than 6 years under 24/7 operation. Always keep nearby USB storage for backup/restore purposes.

Network card

We highly recommend Intel network cards. Several generations exist:

Well tested, but a bit old - X520, 82599 chipset (10Gbps), might have 1 or 2 SFP+ ports. Might accept any SFP+ modules.
X7** series - 10Gbps/40Gbps, might have 1 or 2 SFP+ ports. Need to check compatibility with SFP+ modules, need Intel branded or modified firmware modules. Very high performance, but not tested extensively for PPPoE task.
X9** series - 10Gbps/40Gbps, might have 1 or 2 SFP+ ports. Need to check compatibility with SFP+ modules, need Intel branded or modified firmware modules. Not tested yet.

Not recommended: Broadcom, Chelsio cards, they are known to start packetloss way before 10Gbps.

Boot options

EFI vs BIOS

We support both EFI and BIOS boot modes. But we highly recommend to use EFI boot mode, as BIOS boot known to fail from time to time. Please disable "Secure boot" in BIOS settings.

BIOS settings

We recommend to do following tuning before deploying server to production:

Disable "Secure boot" in BIOS settings.
Change CPU power(powersaving) mode to "OS control" if available, and to "Performance" if not.
TBD, i need to check BIOS of servers i have :)

Other sources

https://github.com/leandromoreira/linux-network-performance-parameters

3.5 KiB Raw Blame History