while ago did blog post about
Found at: gopher.blog.benjojo.co.uk:70/dns-filesystem-true-cloud-storage-dnsfs
DNSFS. Store your files in others DNS resolver caches
===
A while ago I did a blog post about how long DNS
resolvers hold results in cache for, using RIPE Atlas probes testing
against their default resolvers (in a lot of cases, the DNS cache
on their modem/router).
That showed that some resolvers will hold DNS cache entries
for a whole week if asked to
(https://blog.benjojo.co.uk/post/dns-resolvers-ttl-lasts-over-one-week), and I joked at the end that one could use this for file
storage.
Well, I could not stop thinking about doing this. There
are surely a lot of open DNS resolvers out on the internet,
that are just _asking_ to be used for storing random things in
them. Think of it. Possibly tens of gigabytes of cache space
that could be used!
This is not the first time something like this has been
done, Erik Ekman made PingFS, a file system that stores data _in
the internet itself_ .
gif of how pingfs works
This works because inside every ping packet is a section
of data that must be sent back to the system that sent the
ping, called the data payload:
Wireshark view of a ping packet
Because you can put up to 1400-ish bytes in this payload,
and pings take time to come back, you can use the speed of
light in fiber as actual storage.
Now obviously this is not a great idea for long term data
storage, since you have to keep transmitting and receiving the same
packets over and over again, plus the internet gives no promise
that the packet won’t be dropped at any time, and if that
happens then the data is lost.
However. DNS has caches. It has caches _everywhere_.
This means that the DNSFS looks a lot of the same as
PingFS, but once a query is sent it should be cached in the
resolver, meaning you don’t have to keep sending packets to keep
the data alive!
# Resolver strategy
because they are a DDoS reflection risk
For this to work we need a lot of open DNS resolvers.
Technically DNS resolvers (except the official ones that a ISP gives
out) should be firewalled off from the public internet because
they are a DDoS reflection risk , but a lot of devices out
there ship with bad default configuration that allows their built
in DNS resolvers to be reachable from outside the LAN.
The more open DNS resolvers there are, the more redundancy
(or storage space) we have.
and ranges of those who do not want to be scanned
For this, we need to scan the whole internet for
resolvers. This is a slightly daunting task, however when you take
into account the ranges on the internet that are not routable
and ranges of those who do not want to be scanned , it
amounts to about 3,969,658,877 IP addresses.
In addition to that we are looking for open resolvers,
this means that the DNS server on the other end must be able
to look up public domain names, most DNS servers are setup to
be authoritative for a single domain name, and can’t be used
by us for file storage.
# Getting a list of DNS resolvers
For this, I am using Robert Graham's masscan to send DNS
queries to all applicable IP addresses on the internet.
The massscan config I used
However this command has a problem, I am looking for open
resolvers, not just things that will reply to port 53 on UDP.
My solution is to use a great feature of the linux kernel
called BPF filters (you can read a great article about BPF
filters and their use to filter traffic on the Cloudflare blog).
You can use them with iptables to drop any traffic you don’t
want, but programmatically! One BPF rule can do a whole chain
worth of work.
I managed to write a tcpdump filter that only matched the
DNS responses that I wanted (ones with a single results inside
them).
`tcpdump -ni eth0 port 53 and udp and ip[35] != 0x01 and
net 185.230.223.69/32`
I then compiled it to a raw BPF rule using a small
helper program:
```
root@xxxx:~/masscan# ./bpf-gen RAW 'port 53 and udp and
ip[35] != 0x01 and net 185.230.223.69/32'
25,48 0 0 0,84 0 0 240,21 21 0 96,48 0 0 0,84 0 0
240,21 0 18 64,48 0 0 9,21 16 0 132,21 15 0 6,21 0 14 17,40
0 0 6,69 12 0 8191,177 0 0 0,72 0 0 0,21 2 0 53,72 0 0
2,21 0 7 53,48 0 0 35,21 5 0 1,32 0 0 12,21 2 0
3118915397,32 0 0 16,21 0 1 3118915397,6 0 0 65535,6 0 0 0
```
and then inserted it into IPTables:
```
root@xxxx:~/masscan# iptables -I INPUT -m bpf --bytecode
"25,48 0 0 0,84 0 0 240,21 21 0 96,48 0 0 0,84 0 0 240,21 0
18 64,48 0 0 9,21 16 0 132,21 15 0 6,21 0 14 17,40 0 0
6,69 12 0 8191,177 0 0 0,72 0 0 0,21 2 0 53,72 0 0 2,21 0
7 53,48 0 0 35,21 5 0 1,32 0 0 12,21 2 0 3118915397,32 0
0 16,21 0 1 3118915397,6 0 0 65535,6 0 0 0" -j DROP
```
Now masscan will only see and then log results I am
interested in. No need to do a 2nd pass to qualify servers.
gif of masscan running
After waiting about 24 hours for the scan to complete I
got only two abuse notices!
One was automated and very formal, the other not so much:
dir="ltr">I think this guy misunderstands that a NOC abuse contactisn't for sending that kind of abuse tohref="https://t.co/t0ZIbgKyai">pic.twitter.com/t0ZIbgKyai— Ben Cox (@Benjojo12)href="https://twitter.com/Benjojo12/status/931626850031472646?ref_src=twsrc%5Etfw">November 17, 2017
At the end I was left with **3,878,086** open DNS
resolvers, from all ranges and places in the world. Visualised nicely
from Cloudflare's DNS traffic analytics:
cloudflare dns analytics
Basically, where there are Cloudflare data centers, there
are open DNS resolvers.
The country breakdown for open resolvers is as follows:
```
ben@metropolis:~/Documents/dnsfs/bulk-mmlookup$ pv
../uniqiplist.txt | bulk-mmlookup | awk '{$1 = ""; for (i=2; i printf $i " "; print $NF}' | sort | uniq -c | sort -n | tac52.9MiB 0:01:03 [ 850KiB/s][=======================================>] 100%1498094 China285749 United States233549 Republic of Korea168979 Russia167145 Brazil153170 Taiwan142655 India83581 Italy76894 Turkey69300 Poland62542 Philippines53266 Indonesia46055 Japan43331 Romania40434 Bulgaria39548 Australia36099 Iran32996 Canada29971 South Africa```And ISP:```ben@metropolis:~/Documents/dnsfs/bulk-mmlookup$ pv../uniqiplist.txt | bulk-mmlookup -type isp | awk '{$1 = ""; for (i=2;i -n | tac52.9MiB 0:00:15 [3.37MiB/s][=======================================>] 100%830615 4134 No.31,Jin-rong Street355713 4837 CHINA UNICOM China169 Backbone146755 4766 Korea Telecom140772 3462 Data Communication Business Group86075 4812 China Telecom (Group)67874 9829 National Internet Backbone62325 209 Qwest Communications Company, LLC59514 9121 Turk Telekom56682 3269 Telecom Italia55965 9299 Philippine Long Distance Telephone Company54555 4847 China Networks Inter-Exchange50841 12389 PJSC Rostelecom48674 5617 Orange Polska Spolka Akcyjna47039 4808 China Unicom Beijing Province Network40641 26599 TELEFÔNICA BRASIL S.A35171 8866 Vivacom33075 5650 Frontier Communications of America, Inc.31921 9318 SK Broadband Co Ltd30181 8708 RCS & RDS29821 9808 Guangdong Mobile Communication Co.Ltd.28777 17974 PT Telekomunikasi Indonesia28731 7738 Telemar Norte Leste S.A.25634 701 MCI Communications Services, Inc. d/b/a VerizonBusiness23598 35819 Bayanat Al-Oula For Network Services23427 26615 Tim Celular S.A.23322 42610 PJSC Rostelecom22488 7018 AT&T Services, Inc.17009 4713 NTT Communications Corporation14707 12880 Information Technology Company (ITC)14259 24560 Bharti Airtel Ltd., Telemedia Services13989 14420 CORPORACION NACIONAL DE TELECOMUNICACIONES -CNT EP13462 7303 Telecom Argentina S.A.12069 6713 Itissalat Al-MAGHRIB11865 13999 Mega Cable, S.A. de C.V.11439 45899 VNPT Corp10711 7922 Comcast Cable Communications, LLC10256 28006 CORPORACION NACIONAL DE TELECOMUNICACIONES -CNT EP9993 45595 Pakistan Telecom Company Limited8896 4739 Internode Pty Ltd```However, this data is kind of meaningless. Because all itis really showing is the biggest ISP and countries. So let’stry looking at the open resolvers per internet user in acountry:| Country | Open Resolvers |Internet users | People per resolver ||-----------------------|----------------|----------------|----------------------|| Saint Kitts and Nevis | 729 | 37210| 51.0 || Grenada | 771| 41675 | 54.1 || Marshall Islands | 158 |10709 | 67.8 || Bulgaria | 40434 |4155050 | 102.8 || Belize | 1063| 165014 | 155.2 || Republic of Korea | 233549 |43274132 | 185.3 || Bermuda | 308| 60047 | 195.0 || Maldives | 968| 198071 | 204.6 || Guam | 601| 124717 | 207.5|| Antigua and Barbuda | 262 | 60306| 230.2 || Ecuador | 28923| 7055575 | 243.9 || Romania | 43331| 11236186 | 259.3 || Hong Kong | 18698 |5442101 | 291.1 || Barbados | 739| 228717 | 309.5 || Guyana | 954| 305007 | 319.7|| Cayman Islands | 136 |45038 | 331.2 || Seychelles | 158 |56168 | 355.5 || Liechtenstein | 100 |36183 | 361.8 || Tunisia | 13733| 5472618 | 398.5 || Poland | 69300| 27922152 | 402.9 || Georgia | 5078| 2104906 | 414.5 || Malta | 787| 334056 | 424.5|| Andorra | 155| 66728 | 430.5 || Moldova | 4396| 1946111 | 442.7 || Italy | 83581| 39211518 | 469.1 || China | 1498094| 721434547 | 481.6 || Gabon | 358| 182309 | 509.2|Or, if we strip the countries with less than 1 millioninternet users:| Country | Open Resolvers | Internetusers | People per resolver ||--------------------|----------------|----------------|----------------------|| Bulgaria | 40434 |4155050 | 102.8 || Republic of Korea | 233549 | 43274132| 185.3 || Ecuador | 28923 |7055575 | 243.9 || Romania | 43331 |11236186 | 259.3 || Hong Kong | 18698 |5442101 | 291.1 || Tunisia | 13733 |5472618 | 398.5 || Poland | 69300 |27922152 | 402.9 || Georgia | 5078 |2104906 | 414.5 || Moldova | 4396 |1946111 | 442.7 || Italy | 83581 |39211518 | 469.1 || China | 1498094 |721434547 | 481.6 || Australia | 39548 |20679490 | 522.9 || Turkey | 76894 |46196720 | 600.8 || Russia | 168979 |102258256 | 605.2 || Singapore | 7552 |4699204 | 622.2 || Bolivia | 7184 |4478400 | 623.4 || Panama | 2673 |1803261 | 674.6 || Ukraine | 27718 |19678089 | 709.9 || Philippines | 62542 |44478808 | 711.2 || Kuwait | 4493 |3202110 | 712.7 || Lebanon | 6333 |4545007 | 717.7 || Costa Rica | 3486 |2738500 | 785.6 || Saudi Arabia | 25837 |20813695 | 805.6 || Brazil | 167145 |139111185 | 832.3 || Sweden | 10872 |9169705 | 843.4 || Uruguay | 2654 |2238991 | 843.6 || Dominican Republic | 6476 | 5513852| 851.4 || Morocco | 23189 |20068556 | 865.4 || Armenia | 1743 |1510906 | 866.8 |You can find the data yourself here or as a CSV here# How long do resolvers hold cache for?Before testing this, I waited 10 days to allow for all ofthe resolvers who could be on dynamic IP addresses to changeand become unusable. This waiting lost me 37.9% of my IPlist.a decent amount of work to show
RIPE Atlas did a decent amount of work to show that anon dismissable percentage of Atlas probes (normally onresidential connections) change the public IP address once a day:
Bar chart of how long probes keep the same IP addressAfter that, I replicated the old method, a very long TTL(in this case, 68 years) and a TXT record containing thecurrent unix timestamp of the DNS server.The initial query rate on my DNS server was interesting,showing a large initial "rush", likely due to some DNS resolvershaving multiple levels of caches (and thus those upper cacheswarming up)
grafana graph of DNS rpsI then queried every server in the list every hour forabout a few days using a simple tool linkThen refactored some code on the RIPE atlas post to workwith my dataset. This showed that 18% of resolvers can holditems in cache for about a day:```ben@metropolis:~/Documents/dnsfs$ cat retention.csv | awk -F',' '{print $1}' | grep -v time | ./mm -p -b 6Values min:0.00 avg:470.30 med=56.00 max:1778.00 dev:635.18count:2387821Values:value |--------------------------------------------------percent0 |**************************************************45.28%1 |0.29%6 |** 2.08%36 |********** 9.70%216 |************************** 24.27%1296 |******************** 18.38%ben@metropolis:~/Documents/dnsfs$ cat retention.csv | awk -F',' '{print $1}' | grep -v time | ./mm -b 6Values min:0.00 avg:470.30 med=56.00 max:1778.00 dev:635.18count:2387821Values:value |-------------------------------------------------- count0 |**************************************************10811091 |68166 |** 4972136 |********** 231737216 |************************** 5796131296 |******************** 438825```The next task is to make a basic guess on how big thecache is in these 400k open resolvers. To assist in thisguesswork, I queried `version.bind` on every one of these resolvers,filtered down to the major brands of DNS server, and ended up withthe following:
DNS Server breakdownThe default cache configurations for these devices are asfollows:| Brand | Cache Size ||---------------|:-------------:|| DNSMasq | 150 Entries || BIND | 10MB worth || PowerDNS | 2MB worth || Microsoft | Unlimited??? || ZyWall | Unknown || Nominum | Unknown |At this point I figured a maximum of 9 TXT records, each250 character base64 string long (187 bytes ish) would bereasonable.This means we get approximately (3 * 438825 * 187) = 250~MB. Assuming we replicate to 3 different resolvers for each TXTrecord this should also prove "stable" enough to store files forat least a day.# Building the file systemAs much as I like FUSE I found that due to modernassumptions of desktop linux it’s impractical to use it for highlatency backends, which DNSFS would be since its data rate is verylow (but still faster than a floppy disk!). So I chose asimple HTTP interface to upload and fetch files to and from theinternet.The DNSFS code is a relatively simple system, every fileuploaded is split into 180 byte chunks, and those chunks are“set” inside caches by querying the DNSFS node via the publicresolver for a TXT record. After a few seconds the data is removedfrom DNSFS memory and the data is no longer on the clientcomputer.
How DNSFS works gifTo spread the storage load, the request filename and chunknumber is hashed, then modulated over the resolver list to ensurefair (ish) spread.# DemoFor the demo, I will store one of my previous blog postsin the internet itself!
Demo of DNSFSAs you can see, my file has made quite a name foritself.As always, you can find the code to my madness on mygithub here: https://github.com/benjojo/dnsfsAnd you can follow me on Twitter here for micro doses ofmadness like this: https://twitter.com/benjojo12