SFP+

The Ten64 has two SFP+ slots:

Lower slot is connected to DPMAC2 (mapped to eth8 by default)
Upper slot is connected to DPMAC1 (mapped to eth9 by default)

Both slots are in 10G mode (XFI<->SFP+) by default, to use them in 1G mode (SGMII<->SFP) currently a reset config word setting is needed.

The SFP's are connected through a retimer (TI DS110DF410) which implements the clock recovery needed for SFP+. This is not a PHY device - it merely conditions the bitstream.

GPIOs needed for fiber SFP+¶

On the Ten64, SFP's are directly driven by the SoC, there is no 'PHY' device that handles their control signals. Hence when a fiber SFP is used, the control signals do need to be driven by the host.

Eventually we hope there will be a Phylink implementation for the DPAA2 MAC's in the kernel which can handle this transparently.

SFP+ control GPIOS (partial extract from GPIO)¶

GPIO Controller	Pin	Linux GPIO number (with default device tree)	Default Direction	Purpose	Meaning of ‘0’	Meaning of ‘1	Notes
TCA9539 (U39)	0	368	Input	Lower SFP+ TXFAULT	No transmitter fault	Transmitter Fault	Has pull up
TCA9539 (U39)	1	369	Output	Lower SFP+ TXDISABLE	SFP+ transmitter disabled	SFP+ transmitter enabled	Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables
TCA9539 (U39)	2	370	Input	Lower SFP+ PRESENT	Lower SFP present	Lower SFP not present
TCA9539 (U39)	3	371	Input	Lower SFP+ LOS	Lower SFP lost signal	Lower SFP has signal
TCA9539 (U39)	4	372	Input	Upper SFP+ TXFAULT	No transmitter fault	Transmitter Fault	Has pull up
TCA9539 (U39)	5	373	Output	Upper SFP+ TXDISABLE	SFP+ transmitter disabled	SFP+ transmitter enabled	Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables
TCA9539 (U39)	6	374	Input	Upper SFP+ PRESENT	Upper SFP present	Upper SFP not present
TCA9539 (U39)	7	375	Input	Upper SFP+ LOS	Upper SFP lost signal	Upper SFP has signal
LS1088 GPIO4	11	395	Output	SFP+ Left Down (usually bound to lower port)	LED off	LED on
LS1088 GPIO4	12	396	Output	SFP+ Left Up (usually bound to lower port)	LED off	LED on
LS1088 GPIO2	29	477	Output	SFP+ Right Down (usually bound to upper port)	LED off	LED on
LS1088 GPIO2	30	478	Output	SFP+ Right Up (usually bound to upper port)	LED off	LED on

Link and activity LEDs are set up with the ledtrig subsystem.

Example SFP setup - OpenWrt¶

OpenWrt already has the SFP+ GPIOs defined by default, these are set in /etc/config/system:

$ cat /etc/config/system
config gpio_switch 'lower_sfp_txidsable'
        option name 'Lower SFP+ TX Disable'
        option gpio_pin '369'
        option value '0'

config gpio_switch 'upper_sfp_txdisable'
        option name 'Upper SFP+ TX Disable'
        option gpio_pin '373'
        option value '0'

config led 'led_sfp1down'
        option name 'SFP 1 RX'
        option sysfs 'ten64:red:sfp1:down'
        option trigger 'netdev'
        option mode 'link'
        option dev 'eth7'
...

Example SFP setup - standard Linux¶

To enable the lower SFP:

# Export lower SFP+ TXDISABLE to userspace:
echo 369 > /sys/class/gpio/export
# Make it an output
echo out > /sys/class/gpio/gpio369/direction
# Set value to 0 (enable TX)
echo 0 > /sys/class/gpio/gpio369/value

Switching SFP speeds between 1G and 10G¶

Currently, a change of board configuration (RCW and DPC) is required to change the SFP+ speeds from 10G to 1G. These will be incorporated into a future firmware-builder release which will generate the required binaries for all possible configurations.

By default, the board runs SerDes 1 protocol 0x1d/1144 (2xXFI + 2xQSGMII).

The following alternative configurations are available:

SerDes 1 Protocol	Upper SFP	Lower SFP
0x1d / '1144'	10G	10G
0x1e / '1344'	10G	1G
0x1f / '2344'	1G	1G

(There are also 2.5G modes available, but omitted here as these aren't 'standard')

The hexadecimal number is the SerDes protocol number (see Table 23-1 in the LS1088 Reference Manual), while the second decimal number is the RCW setting per lane.

The Upper SFP is Lane D (the first number), the lower SFP is lane C (the second number).

Configuration changes for 1G¶

When a 1G 1000BASE-X SFP is used, the following options need to set in the DPC:

    board_info {
        ports {
             mac@1 {
                 enet_if = "1000BASEX";
             };
        };
    };

The Lower SFP is MAC2 while the Upper SFP is MAC1.

Note that some multi-bit rate SFP's (such as Mikrotik's S+RJ10 10GBase-T) will always present a 10G bitstream, so there is no need to change the SerDes protocol for them.

SFP I2C¶

The I2C bus for SFP's are connected through a PCA9540BD I2C multiplexer, the bottom SFP is channel 2 and top SFP is channel 1. Under Linux, this will be on /dev/i2c-4 and /dev/i2c-3 respectively, with the parent bus (LS1088 I2C4) on /dev/i2c-2.

Our device tree has the required setup for this.

For example, when an SFP is installed in the bottom cage:

# i2cdetect 4
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --

Providing there are no kernel consumers for the SFP I2C (e.g PHYLINK), you can dump the SFP EEPROM:

# i2cdump 4 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4, address 0x50, mode byte
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 22 00 00 00 00 00 08 00 00 06 67 00 00 00    ??".....?..?g...
10: 00 00 01 00 4d 69 6b 72 6f 54 69 6b 20 20 20 20    ..?.MikroTik
20: 20 20 20 20 00 00 40 20 53 2b 52 4a 31 30 20 20        ..@ S+RJ10
30: 20 20 20 20 20 20 20 20 32 2e 30 37 05 00 00 b0            2.07?..?
40: 02 1a 00 00 39 43 35 43 30 33 42 34 45 46 44 30    ??..9C5C03B4EFD0
50: 20 20 20 20 31 38 30 39 32 36 20 20 68 00 06 50        180926  h.?P
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

The same SFP when installed in the top SFP cage:

# i2cdetect 3
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-3.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --

Note that some SFPs will disable their EEPROM when SFP TXDISABLE is active - you should use the SFP PRESENT to determine the status of the SFP cage rather than scanning the I2C bus.