= Manual test plan for rlm_radius

[NOTE]
====
This test plan has not been executed.  There may be errors or false assumptions
Please report any issues with the test plan itself.
Feel free to add any other useful resources such as network diagrams or commands
to introduce packet loss or setup tests hosts.
====

[NOTE]
====
Networking/socket APIs produce faults under different conditions.

sendmsg on MacOS for example, produces no indication of reachability issues, and a real
sendmmsg is only available on Linux, FreeBSD.
====

For load tests, or sending multiple packets in quick succession proto_radius_load should be used to generate
input packets and produce packet rate information.

Do not use radperf as it introduces additional latency and multiple instances are required to achieve high
packet rates (meaning accurate rate information is not available).

For sending small numbers of test packets radclient should be used.

[TIP]
====
Start with a default rlm_radius configuration for each test.  Don't apply configuration changes additively.

====

== Test packets - Sent to Home Server
=== Access-Request
==== PAP Access-Request

```
User-Name = "testuser"
User-Password = "supersecret"
Service-Type = Framed-User
Tunnel-Password = "supersecret"
Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
```

==== CHAP Access-Request - unlang
```
User-Name = "testuser"
Chap-Password = %(chap.password:supersecret)
Service-Type = Framed-User
NAS-IP-Address = 127.0.0.1
Tunnel-Password = "supersecret"
Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
```

==== CHAP Access-Request - radclient
```
User-Name = "testuser"
Chap-Password = "supersecret"
Service-Type = Framed-User
NAS-IP-Address = 127.0.0.1
Tunnel-Password = "supersecret"
Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
```

=== Accounting-Request
```
User-Name = "testuser"
Acct-Session-ID = "0123456789"
Service-Type = Framed-User
NAS-IP-Address = 127.0.0.1
NAS-Port-ID = 0
Message-Authenticator = 0x
```

=== CoA-Request
```
User-Name = "testuser"
Acct-Session-ID = "0123456789"
Service-Type = Authorize-Only
NAS-IP-Address = 127.0.0.1
NAS-Port-ID = 0
Message-Authenticator = 0x
```

=== Disconnect-Request
```
User-Name = "testuser"
Acct-Session-ID = "0123456789"
NAS-IP-Address = 127.0.0.1
NAS-Port-ID = 0
Message-Authenticator = 0x
```
== Topology

The same topology is used for all tests, and consists of two devices.

```
  NAS + Proxy               Home Server
 _____________             ____________
|             |           |            |
| FreeRADIUS  |           |            |
| + radclient | <-------> | FreeRADIUS |
| + wireshark |           |            |
|_____________|           |____________|
```

Packets will be generated on the NAS + Proxy server using either `proto_radius_load` or `radclient`.
`radperf` should not be used as it's deprecated, and cannot produce the load levels required.

The home server should be configured to respond to all packets from the `NAS + Proxy` with a success
response (`Access-Accept` / `Accounting-Response`).

Packet loss should be introduced on the home server.  There are various facilities in IP tables to do this,
or `%{rand:}` and `&control.Response-Packet-Type = none` may be used on the home server to randomly prevent
replies to requests.

If delay needs to be added at a network level, a script is available https://gist.github.com/arr2036/6598137[here].

[NOTE]
====
Restart the `NAS + Proxy` instance after each test.
====

== 1. Replicate mode

Set `replicate = yes` to enable replicate mode.

=== 1.1. Spurious ICMP responses

Platforms:: MacOS, Linux, FreeBSD

Rationale:: We try hard to discard responses as efficiently as possible.  We do however need to ensure that
the method we use to discard replies do not result in spurious ICMP responses sent by the OS when it discards
incoming RADIUS packets.

---

* Send 50 <<PAP Access-Request>>s
** Verify that no ICMP messages are generated by the operating system when responses are received.

=== 1.2. Base TX packet rate

Platforms:: MacOS, Linux, FreeBSD

Rationale:: As there are no ACKs required this tests mesures the rate at which rlm_radius + transport can send
packets. On Linux and FreeBSD we'd expect an exceptionally high packet rate in this mode as we coalesce multiple
outbound packets and send them using single system call
(`sendmmsg` for datagrams, and aggregated buffers + `write` for streams).

When sending datagrams on macOS no native `sendmmsg` function exists, so we expect the rate to be significantly lower.

---

* Ensure the server is running in multi-threaded mode, a non-debug build is being used, and debug messages are set
  to the minimum level.
* Configure proto_radius_load `parallel = 1024` (this matches the default coalesce size in rlm_radius).
* Configure proto_radius_load `start_pps = 10000`
* Configure proto_radius_load `duration = 5`
* Configure proto_radius_load `max_pps = 500000`
* Configure proto_radius_load `max_backlog = 4096` - This controls how many packets are "in flight"
  within the server for any given period.
* Set `<transport>.max_packet_size = 128` to reduce memory consumption - This has a direct effect on the amount of
  memory pre-allocated in trunk requests.
** Use the PAP test packet.  Verify a packet rate of at least 25,000PPS on MacOS, and 50,000PPS on Linux and FreeBSD.

== 2. Proxy mode
=== 2.1. Entering and exiting zombie state (no status checks)

Platforms:: Any

Note:: You may need to increase `max_request_time` in radius.conf for this test.

This test verifies behaviour when no status_check is configured.

In this mode when no responses are received for `zombie_period` seconds, the connection is marked as a zombie.
Whilst in the zombie state no additional requests will be enqueued on the connection. Any requests
in the connection's backlog will be shifted onto other connections or into the trunk's backlog.

A connection in zombie state is set to be `INACTIVE`.  This removes the connection from the set of `ACTIVE`
connections which can be used to send packets.

If there are no `ACTIVE` connections, and the the last trunk event was a connection failure (i.e. a reconnection),
new requests will immediately fail and will not enter the trunk's backlog.
If there are no `ACTIVE` connections, and the last trunk event was a connection opening successfully,
new requests will be enqueued on the trunk's backlog, and will be transferred to the next connection to enter the
`ACTIVE` state.

As there's no method to determine if the upstream server is alive, we need to periodically "revive" the connection.
Reviving in this context means reconnecting the connection.  Reviving the connection is done after `revive_interval`.
When a connection is revived it enters the `ACTIVE` state, and any requests in the trunk's backlog are transferred
to it.

---

* Ensure `status_check.type` is not set.
* Set `zombie_period = 10`.
* Send a <<PAP Access-Request>>
** Verify a response is received.
* Set packet loss rate on network link to be 100%
* Send a <<PAP Access-Request>>
** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
   or `<transport>.Access-Request.max_rtx_duration` is reached.
** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message indicates
  the connection is a zombie.
*** Verify outstanding request enters the backlog _(no active connections, but no failure)_.
* Send a <<PAP Access-Request>>
** Verify this request enters the backlog _(no active connections, but no failure)_.
* Set packet loss rate on network link to be 0%
* Wait `revive_interval` (default 10s).  The connection should enter the `FAILED` then `CLOSED` states.
** Verify both outstanding requests now fail _(no active connections, last event was failure)_.
* Wait `pool.reconnect_delay` (default 5s). The connection should enter the `INIT` then `CONNECTING` states.
* Send a <<PAP Access-Request>>
** Verify this request is sent on the freshly revived connection
   _(active connections, last event was connection connected)_.

=== 2.2. Entering and exiting zombie state (status check - full failure)

Platforms:: Any

Note:: You may need to increase `max_request_time` in radius.conf for this test.

This test verifies the behaviour when a `Status-Server` packet is configured for status checks.

In this mode when no responses are received for `zombie_period` seconds the connection is marked as a zombie.
A connection marked as a zombie enters the `INACTIVE` state.  This removes the connection from the set of
`ACTIVE` connections which can be used to send packets, but does not cancel pending requests already sent
on the connection.

When a connection enters zombie state it will start sending status-check messages.
One of the below conditions will cause the connection to exit zombie state:

- `status_check.num_answers_to_alive` contiguous responses are received to status checks -
  The connection is re-enlivened (enters the `ACTIVE` state).
- `<transport>.Status-Server.max_rtx_duration` is reached - The connection is dead and enters the `FAILED`/`CLOSED` state.
  All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or
  onto another connection.
- `<transport>.Status-Server.max_rtx_count` is hit - The connection is dead and enters the `FAILED`/`CLOSED` state.
  All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or
  onto another connection.

Once `pool.reconnect_delay` seconds have passed, the connection enters the `INIT`/`CONNECTING` state.
Once in the `CONNECTING` state, the connection will begin sending `Status-Server` requests.
`status_check.num_answers_to_alive` contiguous responses are required for the connection to be considered `CONNECTED`
and to receive new requests.
If `max_rtx_duration` or `max_rtx_count` are hit before `status_check.num_answers_to_alive` contiguous responses are
received, the connection will be reconnected (again) after `pool.reconnect_delay`.

[NOTE]
====
Immediate entry into zombie state is likely incorrect and will probably change.
====

---

* Set `status_check.type = Status-Server`
* Set `zombie_period = 10`.
* Send a <<PAP Access-Request>>
** Verify a response is received.
* Set packet loss rate on network link to be 100%
* Send a <<PAP Access-Request>>
** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
   or `<transport>.Access-Request.max_rtx_duration` is reached after which the request should fail.
** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message
   is displayed indicating the connection is now a zombie.
** Verify connection begins sending `Status-Server` requests.
** Verify that after `max_rtx_count` (default 5) packets are sent, or after `max_rtx_duration` (default 30s) passes
   the connection enters the `FAILED`/`CLOSED` state.
* Send a <<PAP Access-Request>>
** Verify the request immediately fails _(no active connections, and previous failure)_.
* Set packet loss rate on network link to be 0%.
* Wait `reconnect_delay` (default 5s).  The connection should enter the `INIT`/`CONNECTING` state
** Verify connection beings sending `Status-Server` requests.
** Verify that `status_check.num_answers_to_alive` (default 3) status checks are sent with a constant delay
   between each check.
** Verify that after `status_check.num_answers_to_alive` responses the connection enters the `CONNECTED` + `ACTIVE`
   state.
* Send a <<PAP Access-Request>>
** Verify this request is sent immediately.

=== 2.3. Entering and exiting zombie state (status check - partial failure and revival)

Platforms:: Any

Note:: You may need to increase `max_request_time` in radius.conf for this test.

See description for previous test.

[NOTE]
====
Immediate entry into zombie state is likely incorrect and will probably change.
====

---
* Set `status_check.type = Status-Server`
* Set `zombie_period = 10`.
* Send a <<PAP Access-Request>>
** Verify a response is received.
* Set packet loss rate on network link to be 100%
* Send a <<PAP Access-Request>>
** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
   or `<transport>.Access-Request.max_rtx_duration` is reached.
** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message
   is displayed indicating the connection is now a zombie.
** Verify connection begins sending `Status-Server` requests.
* Immediately send a <<PAP Access-Request>>
** Verify request enters the trunk's backlog _(no active connections, no previous failure)_.
* After one `Status-Server` packet, set packet loss rate on network link to be 0%.
** Verify that `status_check.num_answers_to_alive` (default 3) additional `Status-Server` packets are sent with a
   constant delay between each.
** Verify that on the last response to a `Status-Server` packet the connection immediately enters the `ACTIVE` state.
*** Verify previously backlogged request is now sent.
* Send a <<PAP Access-Request>>
** Verify this request is sent immediately.

=== 2.4. Synchronous retransmissions

Platforms:: Any

rlm_radius can operate in synchronous and asynchronous retransmission modes.  In asynchronous modes the rlm_radius
module will send retransmissions itself, in synchronous mode the rlm_radius module uses retransmissions by the NAS
to drive retransmissions to the upstream server.

This test verifies synchronous retransmissions operate correctly.

---

* Set `status_check.type = none`.
* Set `zombie_period = 120`.
* Set `synchronous = yes`.
* Set packet loss rate on network link to be 100%
* Send a <<PAP Access-Request>> with 5 retransmissions, 5 seconds apart (use radclient).
** Verify that as radiusd receives each retransmission, rlm_radius sends a new request.
* After 10 seconds set packet loss rate on network link to be 0%.
** Verify response is passed back from rlm_radius, that the response is sent to radclient, and that
  no more retransmissions are sent.

=== 2.5 Memory usage under adverse conditions

Platforms:: Any

Repeat with:

* `status_check.type = Status-Server`, `zombie_period = 1`.
* no `status_check.type`, `zombie_period = 1`, `revive_interval = 10`.

---

* Configure a packet loss rate of 35%.
* Send <<PAP Access-Request>>s at a high rate for 30 minutes.
** Ensure memory usage stabilises within 15 minutes and does not continue to increase.

== 3. Both replicate and proxy modes

i.e. repeat these tests with:

* `replicate = yes`.
* `replicate = no`.

=== 3.1. Well formedness

Platforms:: Any

Notes:: Use radsniff or wireshark to capture sent packets for validation.
You will need to use different subrequest types to produce the different packet types.

---

* Send a <<PAP Access-Request>>
** Verify packet is well formed.
** Verify Message-Authenticator attribute is present and correct.
* Send a <<CHAP Access-Request - radclient>> - Verify that:
** Verify packet is well formed.
** Verify `CHAP-Challenge` attribute is present and matches the Authentication Vector of the input packet.
** Verify Message-Authenticator attribute is present and correct.
* Send an <<Accounting-Request>> - Verify that:
** Verify packet is well formed.
** Verify No Message-Authenticator attribute is present
* Send a <<CoA-Request>>
** Verify packet is well formed.
** Verify Message-Authenticator attribue is present.
* Send a <<Disconnect-Request>>
** Verify packet is well formed.
** Verify Message-Authenticator attribue is present.

=== 3.2. Outbound packet buffer overrun

Platforms:: Any

Rationale:: Verify that internal logic deals correctly with packet buffer overruns.

---

* Set `<transport>.max_packet_size = 64`
* Send a <<PAP Access-Request>>
** Verify an error is produced explaining why the packet can't be encoded (out of buffer space).
** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`).
** Verify the connection is *NOT* reconnected and stays in the `ACTIVE` state.
** Verify the request is not re-queued and rlm_radius returns `fail`.

=== 3.3. SNDBUF exhaustion - single packet

Platforms:: MacOS, Linux, FreeBSD

Rationale:: Verify correct behaviour when the packet size exceeds the `SO_SNDBUF` value
for the socket. `SO_SNDBUF` on the socket should be configured to be small enough that any
outbound requests immediately fail,

---

* set `<transport>.send_buff = 64`
* Send a <<PAP Access-Request>>
** Verify an error is produced explaining why the packet can't be sent (`EMSGSIZE`). If a different
   error is produced, count this as a test fail and record the error.
** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`).
** Verify the connection is *NOT* re-established and stays in the `ACTIVE` state
   _(this is a temporary condition, not a fatal one)_.
** The request is not re-queued and rlm_radius returns `fail`.

=== 3.4. SNDBUF exhaustion - multiple packets

Platforms:: MacOS, Linux, FreeBSD

Rationale:: Verify correct behaviour when no mbuffs are available in the kernel to
accept new packets from userland. `SO_SNDBUF` on the socket should be configured to be large
enough to allow at least one packet to be sent but small enough to cause subsequent packets to
fail.

---

* set `<transport>.send_buff = 128`
* Send multiple <<PAP Access-Request>>s at a high rate
** Verify the first request is sent successfully.
** Verify a subsequent request fails.  You'll likely see two failure conditions, one with an explicit
   error, and one where coalesced packets are silently re-queued.  It depends on whether the
   error occurs on the first packet being processed by `sendmmsg` or a subsequent one.
** Verify that for the explicit failure the error message is (`ENOBUFS`, `EWOULDBLOCK` or `EAGAIN`).
** The connection is *NOT* re-established.  This is a temporary condition, not a fatal one.
** Requests do not explicitly fail (absence of trunk request `PENDING -> FAILED` transitions).

=== 3.5. EHOSTUNREACH - Unreachable host

Platforms:: MacOS, Linux, FreeBSD

---

* Configure a destination host on the same subnet as one of the NAS + Proxy's interfaces.
  This host should not exist.
* Send 50 <<PAP Access-Request>>s 0.5 seconds apart (set parallel = 1 and use the delay module).
* ARP resolution failures should result in sendmmsg returning an error code `EHOSTUNREACH`
  though this may very depending on platform.
** Verify that writing fails with `EHOSTUNREACH` or other appropriate error code.
** Verify that this error results in the connection being re-established.
** Verify that requests are re-queued onto another connection or immediately fail.

=== 3.6. ENETUNREACH - Unreachable network

Platforms:: MacOS, Linux, FreeBSD

---

* Configure a destination host on the a different, unreachable subnet as one of the NAS + Proxy's
  interfaces.  You may need to remove the default route for this host, or ensure that the
  upstream router sends ICMP Unreachable messages.
* Send 50 <<PAP Access-Request>>s 0.5 seconds apart (set parallel = 1 and use the delay module).
* Local routing or ICMP Unreachable messages should result in `ENETUNREACH` being returned.
** Verify that writing fails with `ENETUNREACH` or other appropriate error code.
** Verify that this error results in the connection being re-established.
** Verify that requests are re-queued onto another connection or immediately fail.

=== 3.7. ENETDOWN - Network interface down

Platforms:: MacOS, Linux, FreeBSD

---

* Configure a destination host on the same subnet as one of the NAS + Proxy's interfaces.  You may
  need to remove the default route if there are multiple interfaces.
* Send 50 <<PAP Access-Request>>s 0.5 seconds apart (set parallel = 1 and use the delay module).
* As messages are being sent disable the interface the packets are being set out of.
** Verify that writing fails with `ENETDOWN` or other appropriate error code.
** Verify that this error results in the connection being re-established.
** Verify that requests are re-queued onto another connection or immediately fail.

=== 3.8. Memory usage

Platforms:: Any

---

* Send <<PAP Access-Request>>s at a high rate for 30 minutes.
** Record memory usage every 5 minutes.
** Ensure memory usage stabilises within 15 minutes and does not continue to increase.

