forked from spinesystemspublic/documentation
219 lines
5.8 KiB
Markdown
219 lines
5.8 KiB
Markdown
# Networking
|
|
|
|
## Bonding
|
|
|
|
Traffic is distributed between interfaces using LACP protocol. It is recommended to use LACP on switch side too.
|
|
It is also recommended to enable layer2+3 hashing for better traffic distribution.
|
|
|
|
```
|
|
ip link set bond0 type bond xmit_hash_policy layer3+4
|
|
```
|
|
|
|
|
|
## collectd
|
|
|
|
### Forwarding collectd metrics to another collectd instance
|
|
|
|
You need to add on destination collectd instance:
|
|
|
|
```
|
|
LoadPlugin network
|
|
|
|
<Plugin network>
|
|
<Listen "LISTENIP" "25826">
|
|
SecurityLevel Encrypt
|
|
AuthFile "/etc/collectd/passwd"
|
|
</Listen>
|
|
</Plugin>
|
|
```
|
|
In /etc/colllectd/passwd you need to add user and password for source collectd instance:
|
|
|
|
```
|
|
user:password
|
|
user2:password2
|
|
```
|
|
|
|
Source collectd instance:
|
|
|
|
```
|
|
LoadPlugin network
|
|
|
|
<Plugin network>
|
|
<Server "DESTINATIONIP" "25826">
|
|
SecurityLevel Encrypt
|
|
Username "user"
|
|
Password "password"
|
|
</Server>
|
|
</Plugin>
|
|
```
|
|
|
|
## Selecting right CPU, memory, disk, network card for your GlobalOS
|
|
|
|
### CPU
|
|
Find CPU available on cpubenchmark.net website and check "Average CPU mark".
|
|
|
|
For PPPoE: You can estimate at 7000 points it might take about 4000 users. More points - more users, but growth is not linear.
|
|
For example "Intel Xeon E3-1230 v3 @ 3.30GHz" has 6795 points.
|
|
|
|
For NAT: You can estimate at 7000 points it might take about 10Gbps.
|
|
|
|
Note, if you buy dual CPU server, you will not get 2x performance, it will be still significant boost, and you need to take into consideration, that each PCI-E is "local" for one CPU only, so you might need to install 2 network cards, on each CPU "side". Read about NUMA architecture and how it works.
|
|
|
|
### Memory
|
|
For PPPoE it is recommended to have at least 8GB of RAM at least. Recommended to have 16-32GB, but not more. You might take into consideration some servers with too much memory might have slower memory speed.
|
|
Do not use DDR3 memory/servers, it is too slow, unless you are ok with lower than expected number of users per server.
|
|
|
|
### Disk
|
|
For our system it is totally fine to use USB drive, but internal SSD is preferable. System use very little of storage resources, not I/O bound, we know some USB flashes that lasted more than 6 years under 24/7 operation. Always keep nearby USB storage for backup/restore purposes.
|
|
|
|
### Network card
|
|
We highly recommend Intel network cards. Several generations exist:
|
|
|
|
1) Well tested, but a bit old - X520, 82599 chipset (10Gbps), might have 1 or 2 SFP+ ports. Might accept any SFP+ modules.
|
|
2) X7** series - 10Gbps/40Gbps, might have 1 or 2 SFP+ ports. Need to check compatibility with SFP+ modules, need Intel branded or modified firmware modules. Very high performance, but not tested extensively for PPPoE task.
|
|
3) X9** series - 10Gbps/40Gbps, might have 1 or 2 SFP+ ports. Need to check compatibility with SFP+ modules, need Intel branded or modified firmware modules. Not tested yet.
|
|
|
|
Not recommended: Broadcom, Chelsio cards, they are known to start packetloss way before 10Gbps.
|
|
|
|
## Boot options
|
|
|
|
### EFI vs BIOS
|
|
We support both EFI and BIOS boot modes. But we highly recommend to use EFI boot mode, as BIOS boot known to fail from time to time. Please disable "Secure boot" in BIOS settings.
|
|
|
|
## BIOS settings
|
|
We recommend to do following tuning before deploying server to production:
|
|
|
|
* Disable "Secure boot" in BIOS settings.
|
|
* Change CPU power(powersaving) mode to "OS control" if available, and to "Performance" if not.
|
|
* TBD, i need to check BIOS of servers i have :)
|
|
|
|
## Other sources
|
|
|
|
* https://github.com/leandromoreira/linux-network-performance-parameters
|
|
|
|
|
|
# IPMI
|
|
|
|
## IPMI setup
|
|
|
|
- Verify if your server has IPMI.
|
|
|
|
```
|
|
modprobe ipmi_devintf
|
|
modprobe ipmi_si
|
|
```
|
|
If second command fails, you might not have IPMI.
|
|
Check if `ipmitool` is installed, if not, install it.
|
|
Then verify if it works:
|
|
```
|
|
ipmitool sdr
|
|
```
|
|
|
|
### IPMI network setup
|
|
|
|
First verify current settings:
|
|
```
|
|
ipmitool lan print
|
|
```
|
|
|
|
Assign IP to IPMI interface:
|
|
```
|
|
ipmitool lan set 1 ipsrc static 1.2.3.4/24
|
|
ipmitool lan set 1 defgw ipaddr 1.2.3.1
|
|
ipmitool lan set 1 access on
|
|
```
|
|
|
|
Set username and password. First you need to find channel with users.
|
|
Try:
|
|
```
|
|
ipmitool user list
|
|
ipmitool user list 1
|
|
ipmitool user list 2
|
|
etc to 8
|
|
```
|
|
When it will show you list of users, you can set username and password for this channel:
|
|
|
|
```
|
|
ipmitool user set name 1 admin
|
|
ipmitool user set password 1 password
|
|
```
|
|
TODO: Some boards doesnt allow to change username, and some have limits on password length.
|
|
|
|
|
|
|
|
# Monitoring more than basic metrics
|
|
|
|
## Temperatures
|
|
|
|
### IPMI
|
|
|
|
If IPMI present, you can use ipmitool to get temperatures:
|
|
```
|
|
ipmitool sensors
|
|
```
|
|
|
|
If you don't have ipmitool please contact to obtain latest version of GlobalOS.
|
|
|
|
### lm-sensors
|
|
|
|
```
|
|
modprobe i2c-dev
|
|
modprobe coretemp
|
|
modprobe k10temp
|
|
modprobe pinctrl_lewisburg
|
|
```
|
|
|
|
In `/sys/class/hwmon/` you will find entries for your sensors, search for one with name "coretemp" or "k10temp". for example
|
|
|
|
```
|
|
grep "" /sys/class/hwmon/hwmon*/name
|
|
/sys/class/hwmon/hwmon1/name:coretemp
|
|
/sys/class/hwmon/hwmon2/name:pch_lewisburg
|
|
|
|
grep "" /sys/class/hwmon/hwmon1/temp*
|
|
/sys/class/hwmon/hwmon1/temp10_crit:98000
|
|
/sys/class/hwmon/hwmon1/temp10_crit_alarm:0
|
|
/sys/class/hwmon/hwmon1/temp10_input:30000
|
|
/sys/class/hwmon/hwmon1/temp10_label:Core 8
|
|
/sys/class/hwmon/hwmon1/temp10_max:88000
|
|
/sys/class/hwmon/hwmon1/temp11_crit:98000
|
|
....
|
|
|
|
grep "" /sys/class/hwmon/hwmon2/temp*
|
|
/sys/class/hwmon/hwmon2/device/temp:37000
|
|
|
|
```
|
|
|
|
`pch_lewisburg` is chipset temperature
|
|
|
|
|
|
### Power consumption monitoring
|
|
|
|
- Power meter (ACPI interface)
|
|
```
|
|
Check in /sys/class/hwmon/ entries where device/name contains "power" or "energy".
|
|
```
|
|
|
|
- Power meter (IPMI)
|
|
```
|
|
ipmitool sensor
|
|
```
|
|
|
|
### Disk temperature
|
|
|
|
Note: Smartctl is available in latest GlobalOS versions.
|
|
|
|
```
|
|
smartctl -a /dev/sda | grep Temperature
|
|
```
|
|
|
|
## SFP/SFP+ modules
|
|
|
|
```
|
|
ethtool -m ethX
|
|
```
|
|
|
|
|
|
|
|
|