Implementation and testing of a conformance platform for IEEE 1901.1 Power Line Communication Standard

IEEE 1901.1 Standard for Medium Frequency (less than 12 MHz) Power Line Communications for Smart Grid Applications was officially released on 2018. This standard brings new possibilities for power line communications, as the used frequency range allows for broader band and higher bit rates when compared to other PLC communication standards. In this scope, this paper presents a conformance platform that allows testing compliance of devices implementing the standard. This conformance-testing platform is then used to test the station and concentrator devices from three different manufacturers, using the data link layer tests as one example, highlighting the capabilities of the platform and simultaneously evaluating the current status of development of these devices (with regards to standard compliance). Results of these tests are also included.

. This BB-PLC standard uses OFDM modulation and includes different physical layer specifications including innovative solutions as dual turbo coding, time-frequency interleaving, diversity copy and data link layer specifications including channel timing optimization, tree-topology network and multi-area network coordination. The standard also defines an operation frequency band of 2 MHz to 12 MHz. Due to its novelty, IEEE 1901.1 is not largely deployed yet. Nonetheless, this standard presents a promising solution for PLC as it presents high bandwidth, high reliability and low delay as main operational characteristics.
Different manufacturers are already producing new chipsets and devices, which aim to be fully compliant with IEEE 1901.1. In order to test the device compliance, IEEE has been working in a test procedure, having recently released the IEEE Approved Draft Standard Test Procedures for IEEE 1901.1 Standard, named IEEE 1901.1.1 [8], which contains all tests to be performed to allow verifying the compliance of these devices. Thus, all devices should follow these procedures, in order to reach increase market revenues for the manufacturers and foster market share and large-scale deployment of the standard solutions. This is necessary even at the device development level, prior to certification testing procedures, as it helps manufacturers to solve possible issues with the devices speeding up the manufacturing and market uptake process.
To contribute to this development, this paper presents a conformance-testing platform for IEEE 1901.1 standard device testing. It will part of an IEEE 1901.1 certification laboratory to be established throughout this year. This conformance testing platform is constructed according to IEEE 1901.1.1 [8]. In order to demonstrate the platform capabilities, different modules from three manufacturers are tested. These devices include three single-phase station (STA) devices, three three-phase STA devices and three central coordinator (CCO) devices. A sub-set of tests of those defined in IEEE 1901.1.1 is selected to test these devices, including all data link layer tests. Outcomes of this work will allow demonstrating the platform implementation (software and hardware), its functionalities when performing compliance tests and simultaneously demonstrating the outcomes of these compliance tests for different devices which implement IEEE 1901.1 PLC standard.

Conformance testing platform
In order to test the standard conformance of devices, the team developed a conformance testing platform, which follows the general rules defined in IEEE 1901.1.1. The general schematic of the testing platform is depicted in Figure 1.  The platform is composed of a set of hardware and software components. From the software side, a dedicated software was created to allow the execution of the test cases defined in IEEE 1901.1.1. At this stage, the current software version allows performing a sub-set of all required tests, namely those of the data link layer. Hardware included contains the necessary slots to connect the devices under test (DUT), a transmitter and a receiver to send/receive the communication frames necessary to perform the tests, and the physical connections (cables) between those. Additionally, the platform includes an embedded computer to run the software platform and all required peripheral components. Figure 2 illustrate the implemented testing platform.  Each device needs to be tested separately and the platform allows performing tests to CCO, singlephase STA or three-phase STA devices. Figure 3 depicts a detailed view of the slots used to connect each one of these three devices. In this particular case a single-phase STA was connected.

Testing procedure
In order to be tested, IEEE 1901.1 devices need to enter in test mode. IEEE1901.1.1 defines seven different test modes, which should be selected according to the tests to be performed (functionality to be tested). Testing data frames are identified by including a testing parameter (value '50') in the MAC Service Data Unit (MSDU) header. Additional parameters identify the testing mode selected. Upon restart, a device should return to normal operation mode (leaving automatically test mode). Table 1 contains the number of performed tests, for STA and CCO devices according to each one of the tested functionality in the standard. As previously indicated, these only include data link layer tests. For each test to be executed, IEEE 1901.1.1, and therefore the implemented testing platform, clearly identifies the DUT eliminating any possible error in testing the devices. A final report is then generated, including the results of all tests and possible remarks associated to these. Thus, the platform provides an automated way to execute all tests associated to a single device (STA or CCO), providing a summary in terms of standard compliance by the DUT. Using this testing procedure, nine different devices were tested, from three different manufacturers: single-phase STA, three-phase STA and CCO from Leaguer, Eastsoft and Topscomm.

Conformance tests results
A summary of all obtained results is presented in Table 2, including the total number of tests passed, failed and errors obtained for each tested device and each manufacturer. The next sections provide additional details on the performed tests and obtained results for each class of devices: single-phase STA, three-phase STA and CCO.

Single-phase STA tests
Tests executed to single-phase STA devices include the 34 protocol conformance tests (data link layer) included in IEEE 1901.1.1 and quickly summarized in Table 1. As already described, these 34 tests were repeated, for both tested frequency bands: band 0 (2 MHz -12 MHz) and band 1 (2.4 MHz -5.6 MHz). Furthermore, in order to eliminate possible sporadic errors (e.g. due to communication issues from the platform itself) all tests were duplicated. This means that in fact each device needs to go through 68 different tests and 136 tests are performed per device due to this duplication procedure. Figure 4 depicts a snapshot of the software included in the conformance testing platform indicating the result of the tests performed to the STA from Eastsoft. Sixty-height protocol conformance tests were performed and successfully passed. The full set of tests was executed two times for all devices.
The three single-phase STA devices tested using this procedure demonstrated to be fully compliant with IEEE 1901.1 standard regarding the conformance tests of the data link layer, as all performed tests were successfully considered as passed in a single test execution. Each device took around 75 minutes to complete the 68 tests using the platform described previously. After executing all selected tests, the software provides a summary of achieved results while displaying also other relevant information as the DUT name and manufacturer, the test procedure duration and a log with all executed actions. Test results are also saved in a CSV file for further analysis.
Analyzing all obtained results, it is possible to verify that devices from Leaguer and Topscomm passed all executed tests. The STA from Eastsoft, failed in one test in frequency band 0 during the second execution of the test procedure. The test failed is the test 8.9.8 of IEEE 1901.1.1, named "The STA Goes Offline When the Communication Success Rate Is 0 with Four Consecutive Routing Periods". The test failed because the software platform did not receive the "MMEAssocReq" packets from the STA (step 7 of the test procedure), but this could be due to a communication problem between the STA

Three-phase STA tests
The three-phase STA from Leaguer and Topscomm passed all executed tests. The device from Eastsoft only passed 18 test cases, and 50 test cases failed or timed out. As the devices should support 68 testing cases defined in IEEE 1901.1.1, it is then considered that the three-phase STA from Eastsoft is currently non-compliant with the standard.

CCO tests
Three available CCO devices were effectively tested but none of them passed all test cases, and thus are considered non-compliant. We take Topscomm's test results as an example as this device had a lower number of failed tests.
The results obtained for the tests performed to the Topscomm CCO allow concluding that this device is also not compliant with IEEE 1901.1, since from the 68 tests performed, 50 were successfully passed while 14 failed or gave origin to errors, indicating that there might be some errors in the device firmware. The device presented a different behavior for the tests performed in each one of the two frequency bands tested.
In frequency band 0 (2 MHz -12 MHz), the device passed in 21 tests, failed in 10 tests and the IEEE 1901.1.1 test 8.8.6 -"CCO Allowing Multiple Level-1 STAs to Be Connected" terminated due to an error in the device. Failed tests are related to different functionalities, including: i) beacon mechanism, ii) timeslot management, iii) unicast and broadcast data processing and iv) multi-network coexistence and coordination. Although the failed tests belong to these four groups of functionalities, an analysis on the test results allows concluding that all these tests failed due to an error in the beacon mechanism of the CCO device, which is not generating and broadcasting beacon signals properly, leading to timeout errors. Figure 5 depicts a snapshot of the tests executed to the CCO, in which the errors related to the beacon mechanism and timeslot management can be seen.
It is relevant to mention that as can be confirmed in this figure, the conformance testing platform always executes the full round of selected tests, not stopping when there is a non-compliance. This is important as the results of all tests can provide additional feedback to manufacturers on which are the issues that need to be overcome with the devices, in order to reach IEEE 1901.1 compliance.
In frequency band 1 (2.4 MHz -5.6 MHz), achieved results were different. The CCO passed in 29 tests, failed in two and one test gave origin to an error in the device, being unable to be executed. The test that raised an error was the same that raised also an error in frequency band 0, related to the beacon mechanism. Access". Both tests failed due to timeouts from the conformance testing platform side while waiting for beacon signals, which should have been generated by the CCO, further indicating the results already discussed for frequency band 0. However, in this frequency band the number of failed tests is considerable less. This can be an important feedback point for the manufacturer as the device is presenting some erratic behavior (in terms of IEEE 1901.1 compliance) with the broadcast of beacon signals and this behavior is different in the two analyzed frequency bands. Figure 6 depicts a snapshot of the results obtained in the testing procedure, including a highlight of the failed test 8.1.3. Figure 6. Results of conformance tests to CCO from Topscomm (freq. band 1). Aiming at contributing to the development of the standard and associated solutions, this paper presents a protocol conformance testing platform, including results of conformance tests executed to different devices. This platform was created following the rules defined in 1901.1.1 and contains all necessary hardware and software to test the conformance of DUT in a plug and play way, providing a flexible and straightforward solution from the user perspective. The current version of the platform software allows performing the whole set of data link layer tests, but it is expected to be upgraded for all tests defined in IEEE 1901.1.1. The platform will be integrated in a IEEE 1901.1 certification laboratory currently being developed and expected to be fully operational during this year.

Conclusions
Using this platform, nine different devices from three manufacturers were tested. These include the most relevant PLC related devices: including three single-phase STA, three three-phase STA and three CCO. Results obtained indicate that the three single phase STA tested seem to be compliant with the IEEE 1901.1 standard (regarding data link layer tests). The tested three-phase STAs achieved different results, and only two devices seem to be compliant with the standard. Finally, tests executed to all three CCO resulted in non-compliances, indicating that there is the need for further development of these devices.
Obtained results also allow providing relevant feedback for manufacturers on which are the functionalities that are not compliant with the standard and therefore need further implementation, as the conformance testing platform provides relevant insights on failed tests. This can contribute to an agile and faster device development process, thus contributing as well for the market uptake of IEEE 1901.1 related solutions.