However, there have been a small number of consequences which we have been working on. Obviously not show stoppers otherwise the planned work would have been reversed, but oddities.
One of them was that we were having difficulty getting SNMP from some of our LNSs, which meant some of our monitoring was unavailable. This had left us scratching our heads somewhat as the LNSs were not rebooted or reloaded or anything.
Then, another snag was that today one of our servers that does syslog started to run out of disk. Again, a puzzle. But this was easier to understand just by looking at the logs.
It turns out these are related. We have some debug logs from the LNSs related to setting up PPP sessions and allocation of IP addresses. These are kept for a couple of days to help resolve any connection problems.
One of the things logged is the IPv6 allocation, and this is logged by logging the DHCPv6 request/reply exchange from the customer router. Usually these either happen once after connection or maybe once an hour.
The problem, it seems, is rather odd. Some customers still use the Technicolor ADSL broadband routers that we used to sell from years ago. It seems many of these got upset in a rather odd way after the work on Thursday. We can see no logical reason for this, but they are now in a state where they are on-line and working, but generating approximately 1GB of uplink traffic a day, each, sending DHPCv6 requests! We were logging all of these. It seems the logging may actually have been so much load that it was impacting the SNMP responses.
The fix is rebooting the Technicolor routers, which, thankfully, we can do remotely.
But this gives me a slight insight in to the difficulty of collecting Internet Connection Records. Each of these DHCPv6 exchanges would be something that might well be logged as an ICR.
In practice, just trying to log this one type of packet we could not keep up - the log file was only 16GB (158 million entries) since 4am today. Looking at the traffic levels, that is a tiny fraction of the number of requests being sent by these routers. Our LNS logging system has built in limiting to try and avoid overloading things, and it was being pushed to the limits.
If we had to log every session (TCP/UDP/SCTP/IPSEC/ICMP, etc) there is just no way any of our existing kit could keep up. Of course it wasn't designed to! It was designed to shift packets quickly and provide Internet access to our customers, not snoop on anybody.
This also highlights the issue with any deliberate generation of ICRs by s/w on customer networks. It is easy with relatively low levels of traffic to cause a lot of ICRs to be created, if the #IPBill passes.
I first signed up with you 2½ years ago and was given a Technicolor TG582n. Do you not give these out any more? Any particular reason? It seems mine was indeed affected as it has a connection uptime from around 6am on Thursday. Apart from this, it works fine, just curious!
ReplyDeleteWe currently do a ZyXEL model, but that will change some time I am sure. Times change, and these days we need something that does ADSl and VDSL... Yes, the Technicolors were pretty good.
DeleteAll of these cheap domestic routers have their problems. The TG582n was very unreliable, I had a mains power switch on mine to power cycle it daily at 06:20. But at least the IPv6 workd well. The Zyxel is much more reliable and has a better webui (unless you want to use it on a tablet or smartphone), but the IPv6 stops working a few hours t a day after the router is powered on. One day there might be a cheap domestic router that works reliably, but equally pigs might fly.
DeleteSo, do those customers on a usage based plan get 1gb of credit to their account for this, as it was outside their control? ;)
ReplyDeleteIf we metered upload we might have done that.
DeleteMy bad, I always thought upload was metered on the units tariffs!
DeleteIf they were sending 1GB/day of DHCPv6 requests, weren't you sending a similar amount of DHCPv6 responses?
DeleteIt looks like not - responses at much lower level because of rate limiting.
DeleteIf IP bill is finally passed, this can be a way to DDos ISPs data logs.
ReplyDeleteSeems to me like a good way of DoSing a network if it's badly configured. Obviously it would be sensible to ensure any logging hardware crashing is not going to take your network down (i.e. you'd have to port mirror off the core switches to dedicated logging devices), but the actual capacity needed for a larger network would be staggering no doubt.
ReplyDeleteWhat about storage space? With that much data you'd have to look at LTOs really. Are those going to have to be vaulted for 12 months? You'd be sending truckloads of them each day. What about all the auditing and tracking? What happens when tapes get lost?
Bittorrent would drive the system absolutely crazy.
This also raises the question about what happens if an ISP cannot log the traffic due to lack of storage/capacity or someone deliberately attacking it? Do they have to shut their entire service off until the matter is under control? Are they held liable and treated as criminals if a single session gets through the log?
One hopes our beloved politicians will finally see the light. I did rather bluntly state in my submission that they're the product of someone without the slightest clue, and that personally I don't fear my activities being logged since they won't see anything I don't want them to see. I suppose if there's one good thing that comes of all this, I get to go on record nationally telling a politician they're an idiot...
If I were to send a single one-byte UDP data-packet (plus IP-header) I presume that you would need to log date-time, source IP+port, destination IP+port, protocol, and a lot more useless information.
ReplyDeleteBecause of course, that "might" be an Internet-connection, and I "might" have decided that any one of the "header-fields" was the important data (not to be confused with the "content" that they apparently don't want to log)
What a lot of trouble for sending somebody the number '1'
...
Now if I were to send you an multi-gigabyte file this way [i.e. not necessarily via TCP, so not necessarily able to be defined as a "connection") (perhaps one a day for 12 months) .. Good luck in logging it all :-)
"Logging system not designed to log large number of connections fails to log large number of connections. Government policy proven impossible to implement."
ReplyDeleteReally?
The point is that these systems are not designed to log this stuff. The Internet is growing and the logging is normally for diagnostics or statistics, not for law enforcement. Routers often use custom hardware and ASICs to shift packets to their destination quickly.
DeleteI always felt that the best way to solve this problem was for the Government to run the storage side of things (economies of scale, etc), establish a private interconnect with the ISP and for the ISP to encrypt/sign/relay customer traffic to that service.
ReplyDeleteThe ISP would maintain a cryptographic keypair where the government was handed the public key and the ISP would not disclose the private key under any circumstances.
This keypair would be used to sign each encrypted payload for the purposes of ensuring evidence has not been tampered with.
In addition, the ISP would also generate a cryptographic keypair on a per-customer basis with public keys retained by the ISP and private keys stored offline on a secure system - these keys are rotated monthly by the ISP - these keys are used to encrypt customer traffic prior to signing with the ISPs' key and before transmission to Government storage - older keys are only kept for as long as they are able to decrypt data required by the legal retention period (12 months or whatever).
Once the retention period has passed, the Government will no longer be able to read the data as the ISP will have deleted the private key required to decrypt the data.
A warrant, properly signed, would merely require the ISP to disclose the private keys to law enforcement for that specific customer and covering the specific date range mentioned in the warrant; of course, with the Government being in possession of the private keys, it would be possible for them to 'fake' traffic if they wanted to 'fit up' a particular individual - hence why each genuine encrypted payload would be signed by the ISPs' private key.
Law enforcement would then have to obtain the encrypted data from the Government storage service or perhaps supply the keys directly to that service and they automatically return all data they have which was signed with the designated ISP key and decrypted with the supplied keys.
I believe that this method would have easily met the Governments' *published* requirements of being able to examine a users' historic data trail, they would have had to shoulder all of the expense of storing the data and the ISP still gets the final say as to who can see their customers' traffic or not - in other words, it allows collection of all the data but not necessarily the ability to view any of it without the appropriate legal process being followed.
Of course, this is not what they have done so some of us have other plans to combat this stupid bill if/when it passes.
This would also have the added objective of creating the single biggest honeypot in human history, and a lot of ISP workers being threatened with the greatest backdoor ever - the $5 wrench, in order to reveal the private key.
ReplyDeleteNote, though, that ISP workers are at risk of the $5 wrench backdoor if the ISP does the logging, too - if they'd have access to the private key in Terry F.'s proposal, they'd have access to the data store in the current proposal.
DeleteOut of interest, what does Tor Project traffic look like to an IP Bill compliant logging system? I would expect it to be a huge number of small flows, as you see the Tor client access Tor relays, and relay to relay traffic, which breaks any reasonable logging system.
ReplyDelete