documentation/introduction.md

GlobalOS introduction

GlobalOS is custom Linux distribution build from scratch. It is based on Linux kernel and uses BusyBox+glibc as userland. It is designed to be small, fast and secure, to be used in network appliances, embedded systems and virtual machines.

Major differences from other distributions

• No package manager. GlobalOS is built from scratch and does not use any package manager. It is built as a single image (but contains multiple files) that contains all the necessary software. You can compile and add software to the running system, and install in /usr/local, but it will not be managed by any package manager.
• No init system. GlobalOS uses a custom init system that is built into the kernel. It is a simple init system that starts all the necessary services and then waits for the system to be shut down. It does not support runlevels.
• No bash/fish. GlobalOS does not have traditional shell like bash or fish. It uses a custom shell that is built into the busybox - ash. It is a POSIX compliant shell that is very small and fast.
• No systemd. GlobalOS does not use systemd. It uses a custom init system that is built into the kernel.
• No X11, desktop environment. GlobalOS does not have X11 or any other graphical environment. It is designed to be used in headless systems. It does provide web interface for configuration for some services.
• Custom persistence. GlobalOS uses a custom persistence mechanism. It means in most of cases if you edit files in /etc, you need to issue gs save command to save the changes to the disk. This is done to prevent data loss in case of power failure or other unexpected shutdowns.
• Custom configuration. GlobalOS uses a custom configuration system. It uses a multiple files, different from traditional distributions. Please check documentation for more information.

Introduction to GlobalOS

Let's assume you have a fresh GlobalOS installation. You can login to the system using ssh with key authentication, using the default user root. Note: your ssh key might be installed by installer or certified engineer.

ssh root@globalos.ip

After you login, you will see the following prompt (will vary depending on the version and hostname):

System RC (Firmware rev9.2)
t-pppoe1 ~ # 

This is the GlobalOS shell prompt. You can issue commands here. You can use exit or Ctrl+D to exit the shell.

View current network configuration

You can view the current network configuration using following commands:

• ip link show
• ip addr show
• ip route show

ip link show will show you the network interfaces available on the system. ip addr show will show you the IP addresses assigned to the interfaces. ip route show will show you the routing table.

Additional tools available for advanced network configuration are:

• ip neigh (ARP table)
• ip rule (routing rules, policy routing, similar to PBR in Cisco or FBF in Juniper)
• ethtool (NIC configuration, link status, etc.)

You can get familiar with these tools by reading their documentation:

• iproute2: https://wiki.linuxfoundation.org/networking/iproute2 https://www.man7.org/linux/man-pages/man8/ip-route.8.html (follow also links to other iproute2 man subpages)
• ethtool: https://man7.org/linux/man-pages/man8/ethtool.8.html

Edit configuration files

GlobalOS uses a custom configuration system. It uses multiple files for configuration, instead of a single file like in traditional distributions. The configuration files are stored in /etc directory. You can edit the configuration files using the nano editor. You can also use cat, echo, sed, awk, etc. to edit the files. Network configuration file is /etc/network.cfg, this is shell script executed only once at boot time. Firewall (legacy iptables) configuration is in /etc/firewall.cfg, this is shell script executed every time firewall is reloaded. Your can run it multiple times, but make sure rules are idempotent, e.g. you clean first all chains, and then add rules. resolv.conf is in /etc/resolv.conf, specify DNS servers here. You can also use /etc/hosts to specify static DNS entries.

So how for example network.cfg looks like? Here is an example:

#!/bin/sh
modprobe i40e
modprobe ixgbe
modprobe mlx4_core
modprobe mlx5_core

ip link set dev eth0 up
ip link set dev eth1 up
ip link set dev eth2 up
ip addr add 1.2.3.2/30 dev eth0

ip route replace default via 1.2.3.1 dev eth0

As you can see, it is a shell script that is executed at boot time. It loads the necessary kernel modules (some advanced NICs require kernel modules to be loaded), brings up the interfaces, assigns IP addresses and sets the default route. But as file is not idempotent, if you execute it multiple times, you will get errors. So you need to make sure that you clean up the configuration first, before applying it again. If you need to change configuration at runtime without reboot, you can use ip command to change the configuration, verify it, then edit /etc/network.cfg and save it with gs save command.

For example, to change IP address of eth0 to 2.2.2.2/30, you can use following commands:

ip addr del dev eth0 1.2.3.2/30; ip addr add dev eth0 2.2.2.2/30; ip route replace default via 2.2.2.1 dev eth0

Here you have certain risk, that if you make a mistake, you will lose connection to the system, it is better to ensure having either console access or out-of-band management. Then if you are sure that configuration is correct, you can edit /etc/network.cfg and save it with gs save command. Change lines: ip addr add 1.2.3.2/30 dev eth0 to ip addr add 2.2.2.2/30 dev eth0 ip route replace default via 1.2.3.1 dev eth0 to ip route replace default via 2.2.2.1 dev eth0

Then save the configuration with gs save command.

Save configuration

GlobalOS uses a custom persistence mechanism. It means in most of cases if you edit files in /etc, you need to issue gs save command to save the changes to the disk. This is done to prevent data loss in case of power failure or other unexpected shutdowns. Configuration will be saved to the flash disk, /mnt/flash/secure2/config.tgz file.

Apply configuration

• /etc/network.cfg - require full reboot
• /etc/accel-ppp.conf - restart accel-pppd daemon, in some cases accel-cmd reload is enough
• /etc/nft.cfg - nft -f /etc/nft.cfg
• /etc/config.json - killall burst2d
• /etc/burst2d.cfg - killall burst2d
• /etc/firewall.cfg - sh /etc/firewall.cfg

For restarting accel you can use following snippet:

killall accel-pppd; while true; do
    pid=$(pgrep accel-pppd)
    if [ -z "$pid" ]; then
        accel-pppd -c /etc/accel-ppp.conf -d
        break
    fi
    echo "Waiting for accel-pppd to finish"
    sleep 1
done

Network diagnostics and troubleshooting

If you have problems with network configuration, you can use following tools to diagnose the problem:

• ping

• iperf

• tcpdump

• traceroute

• trafw

• iptop

• curl

• dig

• pppoe-discovery

• conntrack

• ping is used to check if the host is reachable, and can be used to verify link reliability. For example ping -q -s1472 -i0.0001 -c10000 1.2.3.1 will send full sized packets (MTU 1500) to the gateway, with interval of 0.0001ms, and count of 10000 packets. It is useful to check if there is any packet loss on the link. It is similar to Cisco rapid ping.

• iperf is used to check the bandwidth between two hosts. You need to run iperf server on one host, and iperf client on the other host. For example iperf -s on one host, and iperf -c IP.of.server on the other host.

• tcpdump is used to capture packets on the interface. You can use it to diagnose network problems, like missing packets, wrong packets, etc. For example tcpdump -i eth0 -n -s65000 not port 22 will capture packets on eth0 interface, without resolving IP addresses, with full packet size, and not capturing ssh traffic (which is likely how you are connected to the system).

• traceroute is used to trace the route to the destination. It is useful to diagnose routing problems. For example traceroute 8.8.8.8 will show you the route to Google DNS server.

• trafw is used to show current load on the interfaces, filtered using pcap filter. TODO: add example

• iptop is used to show current load on the interfaces per dst/src subnets, filtered using iptables filter. TODO: add example

• curl is used to download files from the internet. It is useful to check if the internet connection is working. For example curl -I http://www.google.com will show you the headers of the google.com website.

• dig is used to query DNS servers. It is useful to check if the DNS servers are working. For example dig www.google.com will show you the IP address of the google.com website.

• pppoe-discovery is used to discover PPPoE servers. It is useful to check if the PPPoE server is reachable in network, or if there is malicious PPPoE server in the network. For example pppoe-discovery -I eth0 will show you the PPPoE servers on the eth0 interface. (it will not show pppoe server running on same server)

• conntrack is used to show the connection tracking table. It is useful to check if the connections are being tracked correctly. For example conntrack -L will show you the connection tracking table. Also you can flush the table with conntrack -F.

Directory structure

GlobalOS has a custom directory structure. It is different from traditional distributions. Here is the directory structure:

• /bin,/sbin, other first level directories, except /usr, /mnt/flash - special directories from kernel initramfs. You need to be carefuly, as changes and new files will use RAM, so you can run out of memory. Changes will not be saved.
• /mnt/flash - contains the flash disk, where the configuration is saved and where the software is stored. You can write to this directory, and changes will be saved.
• /usr/* - except /usr/local, contains the rest of the binaries, like ping, iperf, tcpdump, etc., this directory might be overwritten during runtime, but changes will not be saved

BRAS(PPPoE/L2TP/IPoE/SSTP/PPTP) configuration

Some specific files are related to BRAS configuration:

• /etc/accel-ppp.conf - configuration file for accel-ppp, which implements PPPoE, L2TP, IPoE, SSTP, PPTP, etc. Mostly you can read the documentation of accel-ppp to configure it, but our version has some customizations, so here i will provide key differences.

interface=eth6.3197,{"services":["SERVICENAME1","SERVICENAME2","CSP"],"service-blank":1,"acname":"HALBA-CSP"}

We support json format for services, so you can specify multiple services in one line, specific to each vlan. You can also specify AC-Name, which is used in PPPoE discovery. Also few other options are supported, like service-blank, which will allow to connect to the BRAS without specifying service name, service-any which will allow to connect to the BRAS with any service name. If you set service-blank to 1, this means PPPoE clients without service-name can connect to the BRAS. If you set it to 1 - configuring service name on client become mandatory. This might be useful if you are entering network of reseller or other ISP, and you want to give them trial, without disrupting existing clients, so you can set specific service name for them, and set service-blank to 0, so users without service name will not be able to connect.

TBD: Need to test live update of pppoe config without restard by "update" command i added to accel-ppp.

• /etc/burst2d.cfg - configuration file for burst2d, which implements shaper daemon system configuration.

[global]
static_config=/etc/config.json
#tc=/usr/local/sbin/tc
htb_extra=burst 4000000 cburst 4000000
upload_shaper=0
filter_swap=1
#rx_ratio=1.0

• static_config - path to the configuration file, which is in json format, and contains the configuration of the shaper itself.

• htb_extra - additional options for htb qdisc, like burst and cburst, which are used to allow burst of traffic, and then limit it to the configured rate.

• upload_shaper - if set to 1, will enable upload shaper, which is used to limit the upload speed of the clients. It might be disabled to increase speed of shaper updates and reduce CPU usage.

• filter_swap - if set to 1, will swap the filters, so the filters will be applied in reverse order. Sometimes billing specify Upload/Download filters in reverse order, so this option is useful to apply them in correct order.

• rx_ratio - ratio of download to download speed, if you want to apply some ratio to billing configured speed. For example if user have 2Mbps and rx_ratio is set to 0.5, then user will have 1Mbps download speed.

• /etc/config.json - configuration file for burst2d, which implements shaper daemon system configuration. Please check PPPOE_shaper.md for more information.