# PPPoE management guide
## Adding new vlan interface
```
interface=bond0.2529,{"services":["zone1","Flexnet","ferrari2","fi","rp","pr","dotnet"],"service-blank":1}
```
Possible options:
* services - list of PPPoE services to be added to the interface
* service-blank - if set to 1, then user that didnt specify service can connect to this pppoe
* service-any - if set to 1, then user that specified any service can connect to this pppoe
After updating accel-ppp config, you need to run script that will create vlans and add vlans to accel without restart:
```
sh /etc/add_accel_vlans.sh
```
## Monitoring vlan utilization
```
~ # accel-cmd pppoe interface show
interface: connections: state: service-name policy:
-----------------------------------------------------------
bond0.2622 0 active 3
bond0.2529 2251 active 3
bond0.368 530 active 3
bond0.923 0 active 1
```
You can also check macs on /proc/net/pppoe, and exclude only those you want for specific vlan using grep:
```
~ # cat /proc/net/pppoe|grep bond0.368
```
Also handy command:
```
accel-cmd show sessions username,inbound-if,service-name
```
### Removing vlan interface from accel-ppp instance live
```
accel-cmd pppoe interface del bond0.2529
```
After that you need to monitor using "interface show" command, pppoe will slowly kill all sessions and then remove interface from accel-ppp instance.
### Modifying vlan interface on accel-ppp instance live
This is not easy operation and will cause extended downtime due need to kill all sessions on interface and then add it back. You need to use "interface del" and "interface add" commands.
You can use raw interface del, wait until all users gone, then update accel-ppp.conf, and use shell script to add interface back.
**TODO** I have in development new version of pppoe that will allow to modify interface without downtime.
### Monitoring CPU usage
```
mpstat -P ALL 1
```
If you notice significant disabalance - please run
```
cpubalancerd --rebalance
```
**TODO** I have cpubalancerd in development that will do this automatically in daemon mode, still it is under testing
### How PPPoE session is established
0. Preliminary: you can find out MAC of server using ifconfig INTERFACENAME, for example:
```
~ # ifconfig bond0.2529
bond0.2529 Link encap:Ethernet HWaddr 90:E2:BA:AE:XX:XX
inet addr:
```
Where HWaddr is MAC of server.
1. Client sends PADI packet to server, for example:
```
tcpdump -ni bond0.2529 -vvv -e pppoed
21:32:27.763589 6c:3b:6b:68:XX:XX (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 64: PPPoE PADI [Host-Uniq 0x0031004F] [Service-Name "TR"]
21:32:27.779473 bc:e0:01:25:XX:XX (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 60: PPPoE PADI [Service-Name "id"] [Host-Uniq 0x00001020]
```
First packet is from client that wants to connect to TR service, second packet is from client that wants to connect to id service.
It might be also [Service-Name] - empty service name, which is called service-blank in accel-ppp config.
2. Server answers with PADO packet, for example:
```
21:34:10.705827 90:e2:ba:ae:XX:XX (oui Unknown) > 08:55:31:82:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 72: PPPoE PADO [AC-Name "servername"] [Service-Name "pr"] [AC-Cookie 0xC8D61ACB51BB81F6A75F2097CF120F1DA1AC93E4C97ED3BA] [Host-Uniq 0x00000F30]
```
Server answers with PADO packet, that contains AC-Name, Service-Name, AC-Cookie and Host-Uniq. Most of this entries usually not important in troubleshooting.
As you can see here - client sends PADI with Host-Uniq 0x500FF5C517B8000005020600, and server answers with PADO with same Host-Uniq. This is how PPPoE session is established.
3. Client sends PADR packet to server, for example:
```
21:35:47.143621 6c:3b:6b:81:XX:XX (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 80: PPPoE PADR [Host-Uniq 0x500FF5C517B8000005020600] [AC-Cookie 0x6F71CC054D13DCE911CC1341F1FBDCE7A16830B204F2BEF7] [AC-Name "united"] [Service-Name "pr"]
```
Client sends PADR packet with Host-Uniq and AC-Cookie that he received in PADO packet. If server receives PADR with correct Host-Uniq and AC-Cookie - it will establish PPPoE session.
4. Server sends PADS packet to client, for example:
```
21:35:47.143936 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 52: PPPoE PADS [ses 0x8a48] [AC-Name "united"] [Service-Name "pr"] [Host-Uniq 0x500FF5C517B8000005020600]
```
5. PPPoE session is established, and client can send PPP packets to server. It is more complicated, so i will post whole handshake here of user "VeryGoodUsername".
*Note: i removed pppoed filter, if you want to see LCP and PPP packets - you need to remove it.*
```
tcpdump -ni bond0.2529 -vvv -e ether host 6c:3b:6b:81:XX:XX
21:42:26.848209 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 41: PPPoE [ses 0x1807] LCP (0xc021), length 21: LCP, Conf-Request (0x01), id 107, length 21
encoded length 19 (=Option(s) length 15)
0x0000: c021 016b 0013
Auth-Prot Option (0x03), length 5: CHAP, MD5
0x0000: c223 05
MRU Option (0x01), length 4: 1440
0x0000: 05a0
Magic-Num Option (0x05), length 6: 0x193a92d5
0x0000: 193a 92d5
21:42:26.868270 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] LCP (0xc021), length 16: LCP, Conf-Request (0x01), id 1, length 16
encoded length 14 (=Option(s) length 10)
0x0000: c021 0101 000e
MRU Option (0x01), length 4: 1480
0x0000: 05c8
Magic-Num Option (0x05), length 6: 0x194d41e9
0x0000: 194d 41e9
21:42:26.868295 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 36: PPPoE [ses 0x1807] LCP (0xc021), length 16: LCP, Conf-Ack (0x02), id 1, length 16
encoded length 14 (=Option(s) length 10)
0x0000: c021 0201 000e
MRU Option (0x01), length 4: 1480
0x0000: 05c8
Magic-Num Option (0x05), length 6: 0x194d41e9
0x0000: 194d 41e9
21:42:29.848211 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 41: PPPoE [ses 0x1807] LCP (0xc021), length 21: LCP, Conf-Request (0x01), id 107, length 21
encoded length 19 (=Option(s) length 15)
0x0000: c021 016b 0013
Auth-Prot Option (0x03), length 5: CHAP, MD5
0x0000: c223 05
MRU Option (0x01), length 4: 1440
0x0000: 05a0
Magic-Num Option (0x05), length 6: 0x193a92d5
0x0000: 193a 92d5
21:42:29.905959 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] LCP (0xc021), length 16: LCP, Conf-Request (0x01), id 1, length 16
encoded length 14 (=Option(s) length 10)
0x0000: c021 0101 000e
MRU Option (0x01), length 4: 1480
0x0000: 05c8
Magic-Num Option (0x05), length 6: 0x194d41e9
0x0000: 194d 41e9
21:42:29.905980 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 36: PPPoE [ses 0x1807] LCP (0xc021), length 16: LCP, Conf-Ack (0x02), id 1, length 16
encoded length 14 (=Option(s) length 10)
0x0000: c021 0201 000e
MRU Option (0x01), length 4: 1480
0x0000: 05c8
Magic-Num Option (0x05), length 6: 0x194d41e9
0x0000: 194d 41e9
21:42:29.908534 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] LCP (0xc021), length 21: LCP, Conf-Ack (0x02), id 107, length 21
encoded length 19 (=Option(s) length 15)
0x0000: c021 026b 0013
Auth-Prot Option (0x03), length 5: CHAP, MD5
0x0000: c223 05
MRU Option (0x01), length 4: 1440
0x0000: 05a0
Magic-Num Option (0x05), length 6: 0x193a92d5
0x0000: 193a 92d5
21:42:29.908557 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 52: PPPoE [ses 0x1807] CHAP (0xc223), length 32: CHAP, Challenge (0x01), id 1, Value c879a7f4e8b3272656f5e2faa4e1d3ad, Name accel-ppp
21:42:29.965063 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] LCP (0xc021), length 10: LCP, Echo-Request (0x09), id 0, length 10
encoded length 8 (=Option(s) length 4)
0x0000: c021 0900 0008
Magic-Num 0x194d41e9
21:42:29.965065 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] CHAP (0xc223), length 37: CHAP, Response (0x02), id 1, Value a29cc35e5fc2d30d1770f7a6a099f019, Name VeryGoodUsername
21:42:29.965080 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 30: PPPoE [ses 0x1807] LCP (0xc021), length 10: LCP, Echo-Reply (0x0a), id 0, length 10
encoded length 8 (=Option(s) length 4)
0x0000: c021 0a00 0008
Magic-Num 0x193a92d5
21:42:30.966257 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 47: PPPoE [ses 0x1807] CHAP (0xc223), length 27: CHAP, Fail (0x04), id 1, Msg Authentication failed
21:42:30.966267 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 26: PPPoE [ses 0x1807] LCP (0xc021), length 6: LCP, Term-Request (0x05), id 109, length 6
encoded length 4 (=Option(s) length 0)
21:42:31.004958 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 66: PPPoE [ses 0x1807] LCP (0xc021), length 46: LCP, Term-Request (0x05), id 2, length 46
encoded length 44 (=Option(s) length 40)
0x0000: c021 0502 002c
21:42:31.004961 6c:3b:6b:81:xx:xx (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE S (0x8864), length 60: PPPoE [ses 0x1807] LCP (0xc021), length 6: LCP, Term-Ack (0x06), id 109, length 6
encoded length 4 (=Option(s) length 0)
21:42:31.004977 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 26: PPPoE [ses 0x1807] LCP (0xc021), length 6: LCP, Term-Ack (0x06), id 2, length 6
encoded length 4 (=Option(s) length 0)
```
Most important message here is:
```
21:42:30.966257 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:xx:xx (oui Unknown), ethertype PPPoE S (0x8864), length 47: PPPoE [ses 0x1807] CHAP (0xc223), length 27: CHAP, Fail (0x04), id 1, Msg Authentication failed
```
As you can see, CHAP authentication failed. This is because of wrong password, MAC mismatch, or duplicate session (check billing logs).
### Troubleshooting radius packets by tcpdump
Usually when user try to authenticate, radius server receives Access-Request packet from pppoe. If user is authenticated, radius server sends Access-Accept packet. If user is not authenticated, radius server sends Access-Reject packet. If radius server is not responding, it means that there is no connection between radius server and router or secret is invalid.
To check if PPPoE send requests radius server is responding, you can use tcpdump on PPPoE:
```
tcpdump -ni bond0 port 1812 and host 10.0.0.1 -vvv
tcpdump: listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:03:44.241046 IP (tos 0x0, ttl 64, id 29938, offset 0, flags [DF], proto UDP (17), length 145)
10.168.100.11.40469 > 10.0.0.1.1812: [bad udp cksum 0xdeeb -> 0xd6b4!] RADIUS, length: 117
Access-Request (1), id: 0x01, Authenticator: 1bcd5d43815f3215390182adc83dab1b
User-Name Attribute (1), length: 7, Value: username
0x0000: 7377 3130 31
NAS-Identifier Attribute (32), length: 8, Value: bras01
0x0000: 6272 6173 3031
NAS-IP-Address Attribute (4), length: 6, Value: 10.1.100.11
0x0000: 0aa8 640b
NAS-Port-Type Attribute (61), length: 6, Value: Virtual
0x0000: 0000 0005
Service-Type Attribute (6), length: 6, Value: Framed
0x0000: 0000 0002
Framed-Protocol Attribute (7), length: 6, Value: PPP
0x0000: 0000 0001
Calling-Station-Id Attribute (31), length: 19, Value: 04:5e:a4:f5:xx:xx
0x0000: << hidden >>
0x0010: XX
Called-Station-Id Attribute (30), length: 2, Value:
CHAP-Challenge Attribute (60), length: 18, Value: ..]C._2.9....=..
0x0000: 1bcd 5d43 815f 3215 3901 82ad c83d ab1b
CHAP-Password Attribute (3), length: 19, Value:
0x0000: 0170 22f4 3826 6e16 54b6 0f4c 901e 1e00
0x0010: c5
22:03:44.243758 IP (tos 0x0, ttl 64, id 34019, offset 0, flags [DF], proto UDP (17), length 187)
10.168.100.11.53632 > 10.168.101.2.1812: [bad udp cksum 0xdf15 -> 0x89b4!] RADIUS, length: 159
Access-Request (1), id: 0x01, Authenticator: 2b7da77c13ff40bff0073397f6973096
User-Name Attribute (1), length: 11, Value: jadnet392
0x0000: 6a61 646e 6574 3339 32
NAS-Identifier Attribute (32), length: 8, Value: bras01
0x0000: 6272 6173 3031
NAS-IP-Address Attribute (4), length: 6, Value: 10.1.100.11
0x0000: 0aa8 640b
NAS-Port-Type Attribute (61), length: 6, Value: Virtual
0x0000: 0000 0005
Service-Type Attribute (6), length: 6, Value: Framed
0x0000: 0000 0002
Framed-Protocol Attribute (7), length: 6, Value: PPP
0x0000: 0000 0001
Calling-Station-Id Attribute (31), length: 19, Value: c4:ad:34:5f:xx:xx
0x0000: << hidden >>
0x0010: XX
Called-Station-Id Attribute (30), length: 2, Value:
Vendor-Specific Attribute (26), length: 24, Value: Vendor: The Broadband Forum (3561)
Vendor Attribute: 1, Length: 16, Value: CSS-ABC eth 0/19
0x0000: 0000 0de9 0112 4353 532d 4a61 6420 6574
0x0010: 6820 302f 3139
Vendor-Specific Attribute (26), length: 14, Value: Vendor: The Broadband Forum (3561)
Vendor Attribute: 2, Length: 6, Value: Port20
0x0000: 0000 0de9 0208 506f 7274 3230
CHAP-Challenge Attribute (60), length: 18, Value: +}.|..@...3...0.
0x0000: 2b7d a77c 13ff 40bf f007 3397 f697 3096
CHAP-Password Attribute (3), length: 19, Value:
0x0000: 011c f55c 5141 d61d f602 f854 f32b 6455
0x0010: da
```
### Troubleshooting PPPoE session example
Verify if PPPoE handshake is working:
The only fact is important is to trace sequence for specific user MAC if he sends correct PADI and receives correct PADO, and then sends PADR. For example:
```
tcpdump -ni bond0.2529 -vvv -e pppoed and ether host 6c:3b:6b:81:XX:XX
21:35:47.131362 6c:3b:6b:81:XX:XX (oui Unknown) > Broadcast, ethertype PPPoE D (0x8863), length 60: PPPoE PADI [Host-Uniq 0x500FF5C517B8000005020600] [Service-Name "pr"]
21:35:47.131408 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 80: PPPoE PADO [AC-Name "united"] [Service-Name "pr"] [AC-Cookie 0x6F71CC054D13DCE911CC1341F1FBDCE7A16830B204F2BEF7] [Host-Uniq 0x500FF5C517B8000005020600]
21:35:47.143621 6c:3b:6b:81:XX:XX (oui Unknown) > 90:e2:ba:ae:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 80: PPPoE PADR [Host-Uniq 0x500FF5C517B8000005020600] [AC-Cookie 0x6F71CC054D13DCE911CC1341F1FBDCE7A16830B204F2BEF7] [AC-Name "united"] [Service-Name "pr"]
21:35:47.143936 90:e2:ba:ae:XX:XX (oui Unknown) > 6c:3b:6b:81:XX:XX (oui Unknown), ethertype PPPoE D (0x8863), length 52: PPPoE PADS [ses 0x8a48] [AC-Name "united"] [Service-Name "pr"] [Host-Uniq 0x500FF5C517B8000005020600]
```
Here we see complete handshake, PADI, PADO, PADR, PADS. If you see this sequence - PPPoE handshake is working.
### Decoding values in accel-cmd pppoe interface show
```
interface: connections: state: service-name policy:
-----------------------------------------------------------
bond0.2622 0 active 3
bond0.2529 2251 active 3
bond0.368 530 active 3
bond0.923 0 active 1
```
* service-name policy is bitmask value - 1 means service-blank, 3 means service-blank+service-name-by-list,2 means service-name-by-list (only)
```
/* Answer requests with empty (0 size) service name */
#define PPPOE_FLAG_ANSWER_BLANK (1 << 0)
/* Answer to assigned service names */
#define PPPOE_FLAG_ANSWER_LIST (1 << 1)
/* Answer to ANY service name */
#define PPPOE_FLAG_ANSWER_ANY (1 << 2)
/* Answer to global service name list */
#define PPPOE_FLAG_ANSWER_GLOBAL (1 << 3)
```
So for example 3 means bit 0 and bit 1 is set - so service-name policy is service-blank+service-name-by-list.
### Verify for other PPPoE servers in the network
```
pppoe01 ~ # pppoe-discovery -I bond0.2529
Detected PPPoE server at interface bond0.2529
Detected PPPoE server at interface bond0.2529
Access-Concentrator: MikroTik
Detected PPPoE server at interface bond0.2529
Service-Name: AL
--------------------------------------------------
AC-Ethernet-Address: 74:4d:28:c2:XX:XX
Detected PPPoE server at interface bond0.2529
Detected PPPoE server at interface bond0.2529
Access-Concentrator: ahZZZ ZZZ
Detected PPPoE server at interface bond0.2529
Service-Name: ahm
--------------------------------------------------
AC-Ethernet-Address: 6c:3b:6b:67:XX:XX
Detected PPPoE server at interface bond0.2529
Detected PPPoE server at interface bond0.2529
Access-Concentrator: diab-AHHDJFDF
Detected PPPoE server at interface bond0.2529
Service-Name: TR
...
```
As you can see there are 3 other PPPoE servers in the network, and they are all detected by pppoe-discovery. They might cause conflicts during PPPoE session establishment, so it is recommended to find and remove them.
### References
* [TCPDUMP examples ] (https://hackertarget.com/tcpdump-examples/)
* [More tcpdump docs] (https://danielmiessler.com/study/tcpdump/)