Add notes on how to do cipher suite and key management testing

This describes how various wpa_supplicant/hostapd extensions can be used
to test IEEE 802.11 functionality in other devices.

Signed-off-by: Jouni Malinen <j@w1.fi>

tests/cipher-and-key-mgmt-testing.txt

@@ -0,0 +1,404 @@
+Cipher suite (CCMP, TKIP, GCMP, ..) and key management testing
+==============================================================
+
+wpa_supplicant and hostapd include number of extensions that allow
+special test builds to be used for testing functionality related to
+correct implementation of IEEE 802.11. These extensions allow behavior
+to be modified and invalid operations to be performed to verify behavior
+of other devices in unexpected situations. While most of the testing
+extensions are focused on the fully automated testing framework with
+mac80211_hwsim (see tests/hwsim subdirectory), many of these can be used
+for over-the-air testing of the protocol as well.
+
+Since some of the testing extensions can result in exposing key
+information or allowing non-compliant behavior, these changes are
+disabled in default wpa_supplicant and hostapd builds for production
+purposes. Testing functionality can be enabled by adding
+CONFIG_TESTING_OPTIONS=y into build configuration (hostapd/.config and
+wpa_supplicant/.config).
+
+
+Testing setup
+-------------
+
+These tests can be run as black-box testing without having to modify the
+tested device at all or without knowing details of its
+functionality. The test commands in wpa_supplicant/hostapd control
+interfaces are used to perform unexpected operations and normal data
+traffic is used to verify reaction of the tested device to such
+operations.
+
+In theory, the test functionality is available with most drivers
+supported by wpa_supplicant/hostapd, but the most reliable results are
+likely available through ath9k-based devices. If you are using something
+else, it is strongly recommended that you'll run the first tests with
+sniffer captures and verify that the test tools are behaving correctly.
+
+wpa_supplicant is used to control a test device in station mode to test
+an AP and hostapd is similarly used to control a test device in AP mode
+to test a station.
+
+Various data traffic generators could be used to test the behavior, but
+this document focuses on using ping to test unicast traffic and arping
+to test broadcast traffic. To keep things simple and to reduce
+interference from unrelated traffic, the steps here assume static IPv4
+addresses are used and IPv6 is disabled.
+
+The tests here use WPA2-Personal for simplicity. WPA2-Enterprise and
+other cipher suites can also be tested for more complete coverage.
+
+Example hostapd.conf for the test tool in AP mode:
+
+driver=nl80211
+hw_mode=g
+channel=1
+ieee80211n=1
+interface=wlan0
+ctrl_interface=/var/run/hostapd
+ctrl_interface_group=adm
+ssid=test-psk
+wpa=2
+wpa_key_mgmt=WPA-PSK
+wpa_pairwise=CCMP
+wpa_passphrase=12345678
+
+Example wpa_supplicant.conf for the test tool in station mode:
+
+ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=adm
+
+network={
+    ssid="test-psk"
+    key_mgmt=WPA-PSK
+    psk="12345678"
+}
+
+The examples in this document assume following IPv4 address
+configuration:
+
+Test tool (either AP or station mode): 192.168.1.1/24
+Device under test: 192.168.1.2/24
+
+
+Data traffic tests
+------------------
+
+ping is used to test whether unicast frames go through on the data
+link. It should be noted that ping may need to use broadcast ARP at the
+beginning if the other device is not yet in the ARP table, so working
+broadcast and unicast connectivity may be needed to get this started.
+
+Example:
+
+$ ping -n -c 5 192.168.1.2
+PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
+64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=43.7 ms
+64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=67.9 ms
+64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=900 ms
+64 bytes from 192.168.1.2: icmp_seq=4 ttl=64 time=5.81 ms
+64 bytes from 192.168.1.2: icmp_seq=5 ttl=64 time=135 ms
+
+--- 192.168.1.2 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4004ms
+rtt min/avg/max/mdev = 5.811/230.605/900.223/337.451 ms
+
+This shows working unicast data connectivity.
+
+$ ping -n -c 5 192.168.1.2
+PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
+
+--- 192.168.1.2 ping statistics ---
+5 packets transmitted, 0 received, 100% packet loss, time 4033ms
+
+This shows not working unicast data connectivity.
+
+
+arping is used to test broadcast connectivity.
+
+Example:
+
+$ arping -b -I wlan0 192.168.1.2 -c 5
+ARPING 192.168.1.2 from 192.168.1.1 wlan0
+Unicast reply from 192.168.1.2 [<DUT MAC address>]  119.695ms
+Unicast reply from 192.168.1.2 [<DUT MAC address>]  144.496ms
+Unicast reply from 192.168.1.2 [<DUT MAC address>]  166.788ms
+Unicast reply from 192.168.1.2 [<DUT MAC address>]  2.283ms
+Unicast reply from 192.168.1.2 [<DUT MAC address>]  2.234ms
+Sent 5 probes (5 broadcast(s))
+Received 5 response(s)
+
+This shows working broadcast data connectivity.
+
+$ arping -b -I wlan0 192.168.1.2 -c 5
+ARPING 192.168.1.2 from 192.168.1.1 wlan0
+Sent 5 probes (5 broadcast(s))
+Received 0 response(s)
+
+This shows not working broadcast data connectivity.
+
+If testing results do not look consistent, the testing state can be
+cleared by disconnection and reconnecting the station (the test tool or
+the DUT) to the network.
+
+
+Sniffer and wlantest
+--------------------
+
+It is useful to get a wireless sniffer capture from the operating
+channel of the AP to be able to confirm DUT behavior if any of the data
+tests indicate reason to believe something is not working as expected.
+
+wlantest (from the wlantest directory of hostap.git) can be used to
+decrypt and analyze a sniffer capture. For example:
+
+wlantest -r wlan0.pcap -n decrypted.pcap -p 12345678
+
+The debug prints and comments in the generated file indicate where
+unexpected behavior has been detected, e.g., when the test tool ends up
+clearing its packet number to test replay protection. That can help in
+checking whether the DUT actually replies to a frame that it was
+supposed to drop due replay.
+
+
+Testing replay protection on a station device
+---------------------------------------------
+
+Start hostapd and use hostapd_cli on the test device to control testing
+operations. Connect the DUT to the network.
+
+<3>AP-STA-CONNECTED <DUT MAC address>
+
+This indicates that the connection was completed successfully.
+
+Verify that broadcast and unicast traffic works correctly (if not,
+something is wrong in the test setup and that needs to be resolved
+before being able to run any tests).
+
+Verify that unicast traffic works and issue the following command in
+hostapd_cli:
+
+> raw RESET_PN <DUT MAC address>
+OK
+
+Verify that unicast traffic does not work anymore. If it does, the DUT
+does not implement replay protection correctly for unicast frames. Note
+that unicast traffic can recover once the packet number from the test
+device increases beyond the value used prior to that RESET_PN command.
+
+
+Verify that broadcast traffic works and issue the following command in
+hostapd_cli:
+
+> raw RESET_PN ff:ff:ff:ff:ff:ff
+OK
+
+Verify that broadcast traffic does not work anymore. If it does, the DUT
+does not implement replay protection correctly for broadcast
+frames. Note that broadcast traffic can recover once the packet number
+from the test device increases beyond the value used prior to that
+RESET_PN command.
+
+
+Testing replay protection on an AP device
+-----------------------------------------
+
+Start the AP (DUT) and start wpa_supplicant on the test device to
+connect to the network. Use wpa_cli to control the test device.
+
+<3>SME: Trying to authenticate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
+<3>CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
+<3>Trying to associate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
+<3>Associated with <DUT MAC address>
+<3>WPA: Key negotiation completed with <DUT MAC address> [PTK=CCMP GTK=CCMP]
+<3>CTRL-EVENT-CONNECTED - Connection to <DUT MAC address> completed [id=0 id_str=]
+
+Verify that unicast traffic works and issue the following command in
+wpa_cli:
+
+> raw RESET_PN
+OK
+
+Verify that unicast traffic does not work anymore. If it does, the DUT
+does not implement replay protection correctly. Note that unicast
+traffic can recover once the packet number from the test device
+increases beyond the value used prior to that RESET_PN command.
+
+IEEE 802.11 protocol uses unicast frames in station-to-AP direction, so
+there is no need to test AP replay protection behavior separately with
+the broadcast IPv4 traffic (which would be converted to unicast frames
+on the link layer).
+
+
+Testing GTK reinstallation protection on a station device (group handshake)
+---------------------------------------------------------------------------
+
+Use the procedure describe above for testing replay protection, but with
+the following hostapd_cli commands:
+
+Test broadcast connectivity; should work
+
+> raw RESET_PN ff:ff:ff:ff:ff:ff
+OK
+
+Test broadcast connectivity; should not work; if it does, replay
+protection is completely broken and the following step cannot be
+executed reliably. The following command needs to be run before there
+has been large enough number of new frames to increment the PN on the
+test tool. It would also be possible to execute "raw RESET_PN
+ff:ff:ff:ff:ff:ff" again after the initial sanity testing to get back to
+PN 0 for the next step.
+
+> raw RESEND_GROUP_M1 <DUT MAC address>
+OK
+
+Test broadcast connectivity; should not work; if it does, the device
+does not implement protection for delayed retransmission of Group Key
+Message 1/2.
+
+
+Testing GTK reinstallation protection on a station device (4-way handshake)
+---------------------------------------------------------------------------
+
+Use the procedure described above for testing replay protection for
+broadcast traffic, but with the following hostapd_cli commands:
+
+Test broadcast connectivity; should work
+
+> raw RESET_PN ff:ff:ff:ff:ff:ff
+OK
+
+Test broadcast connectivity; should not work; if it does, replay
+protection is completely broken and the following step cannot be
+executed reliably. The following command needs to be run before there
+has been large enough number of new frames to increment the PN on the
+test tool. It would also be possible to execute "raw RESET_PN
+ff:ff:ff:ff:ff:ff" again after the initial sanity testing to get back to
+PN 0 for the next step.
+
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+Test broadcast connectivity; should not work; if it does, the device
+does not implement protection for delayed retransmission of 4-way
+handshake EAPOL-Key Message 3/4.
+
+Variant 1: Include extra Message 1/4
+
+Otherwise same as above, but replace RESEND_M3 command with:
+
+> raw RESEND_M1 <DUT MAC address>
+OK
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+Variant 2: Include two extra Message 1/4
+
+Otherwise same as above, but replace RESEND_M3 command with:
+
+> raw RESEND_M1 <DUT MAC address> change-anonce
+OK
+> raw RESEND_M1 <DUT MAC address>
+OK
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+
+Testing TK reinstallation protection on a station device (4-way handshake)
+--------------------------------------------------------------------------
+
+Use the procedure described above for testing replay protection for
+unicast traffic, but with the following hostapd_cli commands:
+
+Test unicast connectivity; should work
+
+> raw RESET_PN <DUT MAC address>
+OK
+
+Test unicast connectivity; should not work; if it does, replay
+protection is completely broken and the following step cannot be
+executed reliably. The following command needs to be run before there
+has been large enough number of new frames to increment the PN on the
+test tool. It would also be possible to execute "raw RESET_PN <DUT MAC
+address>" again after the initial sanity testing to get back to PN 0 for
+the next step.
+
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+Test unicast connectivity; should not work; if it does, the device
+does not implement protection for delayed retransmission of 4-way
+handshake EAPOL-Key Message 3/4.
+
+Variant 1: Include extra Message 1/4
+
+Otherwise same as above, but replace RESEND_M3 command with:
+
+> raw RESEND_M1 <DUT MAC address>
+OK
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+Variant 2: Include two extra Message 1/4
+
+Otherwise same as above, but replace RESEND_M3 command with:
+
+> raw RESEND_M1 <DUT MAC address> change-anonce
+OK
+> raw RESEND_M1 <DUT MAC address>
+OK
+> raw RESEND_M3 <DUT MAC address>
+OK
+
+
+Testing ANonce generation on an AP device
+-----------------------------------------
+
+Start the AP (DUT) and start wpa_supplicant on the test device to
+connect to the network. Use wpa_cli to control the test device.
+
+<3>SME: Trying to authenticate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
+<3>CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
+<3>Trying to associate with <DUT MAC address> (SSID='test-psk' freq=5240 MHz)
+<3>Associated with <DUT MAC address>
+<3>WPA: Key negotiation completed with <DUT MAC address> [PTK=CCMP GTK=CCMP]
+<3>CTRL-EVENT-CONNECTED - Connection to <DUT MAC address> completed [id=0 id_str=]
+
+Show the ANonce from the first 4-way handshake, request PTK rekeying,
+and show the ANonce from the second 4-way handshake:
+
+> GET anonce
+df8c61d1f1f7aca9f1739dd888199547f4af2b8b07f8bf15b45ea271da0072b2
+> raw KEY_REQUEST 0 1
+OK
+> GET anonce
+d8ddcb716f28abfdf1352a05d51e7a70f58802122e99d13c730c3c0f09594aac
+
+If the ANonce values are same, the AP did not update the ANonce for
+rekeying (it should have as shown in the example above).
+
+
+Testing FT Reassociation Request frame retransmission on an AP device
+---------------------------------------------------------------------
+
+This test case requires a sniffer to be used and manually analyzed.
+
+Enable FT on the DUT AP (likely two AP devices needed), connect test
+tool to the AP using FT protocol (e.g., connect to another AP first and
+then use the "ROAM <BSSID>" command), and do the following steps:
+
+- verify unicast traffic from the AP to test station (either ping from
+  the AP or from a device behind the AP); this needs to work
+- wpa_cli "raw RESEND_ASSOC"
+- verify unicast traffic from the AP to test station (either ping from
+  the AP or from a device behind the AP); this is likely to fail, but
+  the real analysis is done based on the sniffer capture
+
+In the sniffer capture, find the last Reassociation Request frame from
+the test station (this is more or less identical to the previous one and
+the only one that should not have Authentication frame exchange before
+it). Look at the last used PN in a unicast Data frame from the AP to the
+test station before the last Reassociation Request frame and the PN in
+the following unicast Data frame after the last Reassociation Request
+frame. If the PN goes down (e.g., is reset to 1), this would be a sign
+of a likely security vulnerability. The AP's TK configuration should be
+verified (i.e., whether it is configuring the same TK again and then
+allowing it to be used with reused PN values).