# Shaping system on PPPoE ## Packet flow and traversing thru rules from internet to user ### Packet marking in firewall.cfg First packet will be processed thru iptables, /etc/firewall.cfg

In case we want to apply mark to packet we will use ipset + skbinfo module
This is optional step and used only if we want some bypass class to match many ips or ports.

For example:
``` ipset add bypass 213.204.127.171 skbmark 0x12 #iptv ipset add bypass 185.99.35.252/30 skbmark 0x12 #iptv ipset add bypass 185.118.26.192/26 skbmark 0x13 # ggc-aoun ipset add bypass 185.118.26.128/26 skbmark 0x13 # fna-aoun ``` This means if packet with source ip will go thru iptables it will be marked with 0x12 or 0x13.

Condition of iptables that apply this mark is:
``` iptables -t mangle -A PREROUTING -i bond0 -j SET --map-set bypass src --map-mark ``` ### Packet evaluation in filters of ppp interface to user Next step is to evaluate packet in filters of ppp interface to user.
Filters are described in /etc/config.json, section "traffic_filters".
For example: ``` "bypass4": { "prio": 14, "classid": 25, "criteria": "handle 0x14 fw" }, "ping": { "prio": 5, "classid": 10, "criteria": "u32 match ip protocol 1 0xff" }, "dns": { "prio": 5, "classid": 10, "criteria": "u32 match ip protocol 17 0xff match ip sport 53 0xffff" }, ``` Take attention on "prio" field, this is order how packet evaluated, from small to big number.
If prio is same, then packet will be evaluated in order of appearance in config file (but better to not rely on that).
So packet will be evaluated first in "ping" filter, then in "dns" filter, then in "bypass4" filter.
ping and dns filters are self-explanatory, they will match icmp and dns packets and send them to classid 10.
bypass4 will verify if packet is marked with 0x14, if yes, then it will be sent to classid 25.

There is usually default filter rule
``` "default": { "prio": 200, "classid": 2, "criteria": "u32 match u32 0 0" } ``` This rule will match all packets that were not matched by previous rules and send them to classid 2.
Classid is special and match account "default" speed set in radius, it is usually "direct traffic".
### Packet classification in traffic shaper Now after we have assigned classid to packet, we will go to traffic shaper and apply shaping rules.
Classes are defined in traffic_classes section of /etc/config.json.
Take attention that "bypass2" of filter and class in traffic_classes are not the same.
``` "traffic_classes": { "bypass1": { "classid": 10, "parent": 0, "qdisc": "pfifo limit 100" }, ``` This is classid 10, parent 0, qdisc pfifo limit 100. Qdisc allows us to set different disciplines for class, some (not pfifo) might be more friendly for gaming, of we can increase queue size of bulk traffic to reduce packetloss if user are full.
Parent is special, it allows to set tree like in Mikrotik traffic shaping by trees.
In our case parent: 0 means that this class is root class, it is not child of any other class.
If we set parent class 1 it means burst traffic will be limited by parent class 1 (which is equal to account speed).
### How traffic shaper speeds are set Speeds are set in "services" of /etc/config.json. ``` "services": { "#default": { "activate_classes": [ "bypass1", "bypass2", "bypass3", "bypass4", "default" ], "cir_classes": [ "%300", "%300", "%200", "%100", "%100" ], "burst_classes": [ "%300", "%300", "%200", "%100", "%100" ], "modifiers_classes": [ "", "", "", "", "" ], "activate_filters": [ "bypass1", "bypass2", "bypass3", "bypass4", "ping", "dns", "default" ] } ``` Interesting part here is "cir_classes" and "burst_classes". They should match by sequence number to "activate_classes".
So for example "default" class will be set to cir 100% and burst 100% of account speed. (last value)
Possible values:
* %XXX - percentage of account speed. So 5Mbit account with value %200 will have 10Mbit cir and burst. * #N - bypass N set by radius attribute Login-LAT-Node, for example "speed:1000/3000/6000" means #0 is 1000, #1 is 3000, #2 is 6000. * NNNN - fixed speed, for example 1000 is 1Mbit So basically if you want to change bypass2 to bypass from radius, you need to change in cir_classes and burst_classes value to #, which by previous example will be 3000. ### How to verify radius attributes Find user ppp interface name, for example pppX, by running: ``` accel-cmd show sessions|grep username ``` Then execute: ``` cat /var/run/radattr.pppX ``` This will show you all radius attributes for this user. ### How to verify traffic classes of user ``` tc -s -d class show dev pppX class htb 1:1 root rate 3072Kbit ceil 3072Kbit linklayer ethernet burst 1999999b/1mpu 0b cburst 1999999b/1mpu 0b level 7 Sent 76855085 bytes 74998 pkt (dropped 0, overlimits 2 requeues 0) backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 81273324 ctokens: 81273324 class htb 1:10 root leaf 10prio 0 quantum 30000 rate 9216Kbit ceil 9216Kbit linklayer ethernet burst 1999998b/1mpu 0b cburst 1999998b/1mpu 0b level 0 Sent 153094 bytes 838 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 lended: 838 borrowed: 0 giants: 0 tokens: 27125025 ctokens: 27125025 class htb 1:2 parent 1:1 leaf 2prio 0 quantum 30000 rate 3072Kbit ceil 3072Kbit linklayer ethernet burst 1999999b/1mpu 0b cburst 1999999b/1mpu 0b level 0 Sent 76855085 bytes 74998 pkt (dropped 114, overlimits 470 requeues 0) backlog 0b 0p requeues 0 lended: 71295 borrowed: 0 giants: 0 tokens: 81273324 ctokens: 81273324 class htb 1:20 root leaf 20prio 0 quantum 30000 rate 6144Kbit ceil 6144Kbit linklayer ethernet burst 1999998b/1mpu 0b cburst 1999998b/1mpu 0b level 0 Sent 185321931 bytes 147757 pkt (dropped 67, overlimits 458 requeues 0) backlog 0b 0p requeues 0 lended: 145722 borrowed: 0 giants: 0 tokens: 40660958 ctokens: 40660958 class htb 1:15 root leaf 15prio 0 quantum 30000 rate 9216Kbit ceil 9216Kbit linklayer ethernet burst 1999998b/1mpu 0b cburst 1999998b/1mpu 0b level 0 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 27126734 ctokens: 27126734 class htb 1:25 root leaf 25prio 0 quantum 30000 rate 3072Kbit ceil 3072Kbit linklayer ethernet burst 1999999b/1mpu 0b cburst 1999999b/1mpu 0b level 0 Sent 48626547 bytes 37274 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 lended: 37019 borrowed: 0 giants: 0 tokens: 81372390 ctokens: 81372390 ``` Classid match definition of classid in config.json.
If you execute command several times, take attention on counters, for example if you see value in backlog, it means that
traffic is being "buffered", so user use all speed for this class.
If counter in dropped is not zero, it means that user is sending more traffic than allowed by class and packets are dropped.
And sure you can match rate and ceil, which match cir and burst values.
# burst2d /etc/burst2d.cfg This is burst daemon system settings file. Available options: ``` [global] static_config=/etc/config.json #tc=/usr/local/sbin/tc htb_extra=burst 4000000 cburst 4000000 upload_shaper=0 filter_swap=0 ``` - static_config - path to config.json, where shaper definitions are done. - tc - optional, used in case we need to use different version of tc - htb_extra - extra options for HTB, for example to set microburst values to reduce CPU load - upload_shaper if set to 0 - upload shaper is disabled, 1 - enabled - filter_swap - swap how Radius speed attributes identified. Values 0 or 1. If you see download speed incorrectly set as upload (and upload as download), change variable to opposite # burst2d config.json options ## Example config.json ```json { "defaultservice": "#default", "traffic_classes": { "fna" : { "classid": 10, "parent": 1, "qdisc": "pie limit 100 target 10ms" }, "ggc" : { "classid": 11, "parent": 1, "qdisc": "pfifo limit 100" }, "bypass" : { "classid": 12, "parent": 0, "qdisc": "pie limit 100 target 10ms" }, "default" : { "classid": 2, "parent": 1, "qdisc": "pie limit 100 target 100ms" } }, "traffic_filters": { "fna" : { "prio": 10, "classid": 10, "criteria": "u32 match ip src 185.22.34.0/24" }, "ggc" : { "prio": 10, "classid": 11, "criteria": "u32 match ip src 91.240.80.224/27" }, "ping" : { "prio": 5, "classid": 12, "criteria": "u32 match ip protocol 1 0xff" }, "dns" : { "prio": 5, "classid": 12, "criteria": "u32 match ip protocol 17 0xff match ip sport 53 0xffff" }, "proxy" : { "prio": 6, "classid": 12, "criteria": "handle 1 fw" }, "default" : { "prio": 200, "classid": 2, "criteria": "u32 match u32 0 0" } }, "classifiers": { "fuped": { "speed": "<512", "setservice": "#fuped", "decision": "final" }, "bigvip": { "speed": ">100000", "tag":"vip", "setservice": "#plainflat" }, "big": { "speed": ">100000", "setservice": "#plainflat", "decision": "final" }, "eveningshapedfna": { "timerange":"17-24", "setservice": "#default" }, "noshapedfna": { "timerange":"0-17", "setservice": "#defaultnofnashape" } }, "services": { "#default" : { "activate_classes": ["fna", "ggc", "bypass", "default"], "cir_classes": ["%25", "10000", "10000", "%1"], "burst_classes": ["%50", "10000", "10000", "%100"], "modifiers_classes": ["", "pfifo limit 1000", "", ""], "activate_filters": ["fna", "ggc", "ping", "dns", "proxy", "default"] }, "#defaultnofnashape" : { "activate_classes": ["ggc", "bypass", "default"], "cir_classes": ["10000", "10000", "%1"], "burst_classes": ["10000", "10000", "%100"], "modifiers_classes": ["pfifo limit 1000", "", ""], "activate_filters": ["ggc", "ping", "dns", "proxy", "default"] }, "#fuped" : { "activate_classes": ["bypass", "ggc", "default"], "cir_classes": ["2000","1000", "%1"], "burst_classes": ["2000","1000", "%100"], "modifiers_classes": ["", "", ""], "activate_filters": ["ping", "proxy", "dns", "ggc", "default"] }, "#plainflat" : { "activate_classes": ["default"], "cir_classes": ["%100"], "burst_classes": ["%100"], "modifiers_classes": [""], "activate_filters": ["default"] } } } ``` ## Radius attributes * Login-LAT-Service tag1,tag2,tag3 - set tags for user * Login-LAT-Node bypass:N1/N2/N3... - set bypass speed for user, for example bypass:1000/3000/6000 means bypass0 is 1000, bypass1 is 3000, bypass2 is 6000, which is set as #0, #1, #2 in cir_classes and burst_classes ## Config.json structure * defaultservice - default service to use if no classifier matched * traffic_classes - classes to use in traffic shaper * classid - classid to use, this classid will be used in filters, so when filter match it will route traffic to this class * parent - parent classid, this means burst will be limited by parent class, or it can be set as 0 if this is root class and no burst expected (cir should be equal to burst) * qdisc - qdisc to use, this is queue discipline, it can be pfifo, pie, fq_codel, etc. * traffic_filters - filters to match traffic and decide which class will handle it * prio - priority of filter, lower number means higher priority * classid - classid to route traffic if filter matched * criteria - criteria to match, it can be u32 match, handle, etc. * classifiers - classifiers to match user and set service. WARNING: if you have complex classifiers, you should set them in order of priority using prio field. Prio evaluated from 0 to last classifier, so if you have classifier with prio 0, it will be evaluated first, then 1, etc. * speed - speed to match, it can be <, >, =, <=, >=, <>, etc. * setservice - service to set if classifier matched * decision - decision to make if classifier matched, it can be final, continue, etc. * timerange - timerange to match, it can be 0-24, 17-24, etc, so you can set different service for different time range * tag VALUE - check if tag is present (Login-LAT-Service tag1,tag2,tag3) * notag VALUE - check if tag is not present (Login-LAT-Service tag1,tag2,tag3) * isreal 0/1 - check if realip is 0 or not * isfup 0/1 - check if bypass_speed[0] is equal to speed * brasname VALUE - check if brasname is equal to VALUE (set in /etc/burst2d.conf brasname=xxx, default value is "default") * services - services to use in traffic shaper * activate_classes - classes to activate in this service * cir_classes - cir values for classes, it can be %XXX, #N, NNNN, where %XXX is percentage of account speed, #N is bypass N, NNNN is fixed speed. For example 10000 is 10Mbit, %200 is 2x account speed, etc. * burst_classes - burst values for classes, it can be %XXX, #N, NNNN * modifiers_classes - modifiers for classes, it can be "", "pfifo limit 1000", etc. * activate_filters - filters to activate in this service ## Example of changing config.json Let's say we want to introduce new service for users where all CDN bypass(fna and ggc) is grouped together and set to 10Mbit, and we want to set it to users with tag "onebypass". New config will look as following: ```json { "defaultservice": "#default", "traffic_classes": { "fna" : { "classid": 10, "parent": 1, "qdisc": "pie limit 100 target 10ms" }, "ggc" : { "classid": 11, "parent": 1, "qdisc": "pfifo limit 100" }, "bypass" : { "classid": 12, "parent": 0, "qdisc": "pie limit 100 target 10ms" }, "default" : { "classid": 2, "parent": 1, "qdisc": "pie limit 100 target 100ms" } }, "traffic_filters": { "fna" : { "prio": 10, "classid": 10, "criteria": "u32 match ip src 185.22.34.0/24" }, "ggc" : { "prio": 10, "classid": 11, "criteria": "u32 match ip src 91.240.80.224/27" }, "ping" : { "prio": 5, "classid": 12, "criteria": "u32 match ip protocol 1 0xff" }, "dns" : { "prio": 5, "classid": 12, "criteria": "u32 match ip protocol 17 0xff match ip sport 53 0xffff" }, "proxy" : { "prio": 6, "classid": 12, "criteria": "handle 1 fw" }, "fnaonebypass" : { "prio": 10, "classid": 12, "criteria": "u32 match ip src 185.22.34.0/24" }, "ggconebypass" : { "prio": 10, "classid": 12, "criteria": "u32 match ip src 91.240.80.224/27" }, "default" : { "prio": 200, "classid": 2, "criteria": "u32 match u32 0 0" } }, "classifiers": { "fuped": { "speed": "<512", "setservice": "#fuped", "decision": "final", "prio": 0 }, "onebypass": { "tag":"onebypass", "setservice": "#onebypass", "decision": "final", "prio": 1 }, "bigvip": { "speed": ">100000", "tag":"vip", "setservice": "#plainflat", "prio": 2 }, "big": { "speed": ">100000", "setservice": "#plainflat", "decision": "final", "prio": 3 }, "eveningshapedfna": { "timerange":"17-24", "setservice": "#default", "prio": 4 }, "noshapedfna": { "timerange":"0-17", "setservice": "#defaultnofnashape", "prio": 5 }, }, "services": { "#default" : { "activate_classes": ["fna", "ggc", "bypass", "default"], "cir_classes": ["%25", "10000", "10000", "%1"], "burst_classes": ["%50", "10000", "10000", "%100"], "modifiers_classes": ["", "pfifo limit 1000", "", ""], "activate_filters": ["fna", "ggc", "ping", "dns", "proxy", "default"] }, "#defaultnofnashape" : { "activate_classes": ["ggc", "bypass", "default"], "cir_classes": ["10000", "10000", "%1"], "burst_classes": ["10000", "10000", "%100"], "modifiers_classes": ["pfifo limit 1000", "", ""], "activate_filters": ["ggc", "ping", "dns", "proxy", "default"] }, "#fuped" : { "activate_classes": ["bypass", "ggc", "default"], "cir_classes": ["2000","1000", "%1"], "burst_classes": ["2000","1000", "%100"], "modifiers_classes": ["", "", ""], "activate_filters": ["ping", "proxy", "dns", "ggc", "default"] }, "#plainflat" : { "activate_classes": ["default"], "cir_classes": ["%100"], "burst_classes": ["%100"], "modifiers_classes": [""], "activate_filters": ["default"] }, "#onebypass" : { "activate_classes": ["bypass", "default"], "cir_classes": ["10000", "%100"], "burst_classes": ["10000", "%100"], "modifiers_classes": ["", ""], "activate_filters": ["fnaonebypass", "ggconebypass", "default"] } } } ``` So what are changed? * Added new filters fnaonebypass and ggconebypass to match fna and ggc traffic to class 12, which is bypass class * Added new classifier onebypass to match users with tag "onebypass" and set service to #onebypass * Added new service #onebypass to activate only 2 classes, bypass and default, and set speed of bypass to 10Mbit, and "default" (which is direct traffic) to 100% of account speed ## How this rules evaluated when shaper find such user? * First it will read radius attributes and find all tags * Then it will start evaluating classifiers in order of prio. First one is fuped, it will check if user speed is less than 512, if yes, it will set service to #fuped and stop evaluating classifiers. But we are not, so it will continue. Next rule is onebypass, it will check if user has tag "onebypass", if yes, it will set service to #onebypass and stop evaluating classifiers. This is our case, so it will set service to #onebypass and stop evaluating classifiers. * Then it will read service #onebypass and activate classes bypass and default, set cir of bypass to 10Mbit and default to 100% of account speed. * Also it will activate filters fnaonebypass and ggconebypass, so all fna and ggc traffic will be matched to bypass class. To verify if user is matched to correct service, you can use tc command to show classes of user, for example: ``` tc -s -d class show dev pppX ``` Where pppX is user ppp interface name of user identified by accel-cmd show sessions|grep username for example. And you can see two classes, one for fna + ggc traffic, and one for all other traffic. ## I have large list of ips to match, how to do it? For example such situation might be with iptv traffic, where you have large list of ips to match, or speedtest servers, etc. You need to add following changes to your firewall first, note that you need to place each line in proper place, not as is. E.g. ipset create added after lines where we flush and destroy old ipset rules and etc. ``` ipset create bypass hash:net skbinfo for i in $(cat /etc/speedtest.txt | awk '{ print $1; }') ; do ipset -A bypass $i skbmark 0x15 ; done iptables -t mangle -A PREROUTING -i bond0 -j SET --map-set bypass src --map-mark ``` in config.json you can add filter speedtest: ``` "speedtest": { "prio": 5, "classid": 10, "criteria": "handle 0x15 fw" }, ``` First line will create ipset named bypass with hash:net type, which is used to store list of ips and together with skbinfo module to store skbmark value which can be matched by "handle 0x15 fw" in filters. Second line will read file /etc/speedtest.txt and add each ip to ipset bypass with skbmark 0x15. You can change 0x15 to any other value. This file should contain list of ips, one per line. Third line will add iptables rule to match all packets with source ip in ipset bypass and set skbmark to 0x15. ## Upload shaper Settings in burst2d.cfg: - `upload_shaper=0` unlimited upload - `upload_shaper=1` upload set as in radius ### Verifying upload shaper `tc -s -d filter show dev pppX ingress` ``` filter parent ffff: protocol all pref 49152 u32 chain 0 filter parent ffff: protocol all pref 49152 u32 chain 0 fh 800: ht divisor 1 filter parent ffff: protocol all pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid not_in_hw (rule hit 29 success 29) match 00000000/00000000 at 0 (success 29 ) police 0x3 rate 20Mbit burst 10000Kb mtu 65000b action drop overhead 0b linklayer ethernet ref 1 bind 1 installed 867 sec used 0 sec firstused 840 sec Sent 4640 bytes 29 pkts (dropped 0, overlimits 0) ``` As you can see it is set to 20Mbit with burstability in bucket to 10Mbyte. Upload shaper, when enabled, is single for all traffic and set as upload speed in billing(radius). Bypass classes and such stuff is not relevant to upload shaper.