Government Moves Forward with UK ISP Internet Snooping System
The UK Home Office and the National Crime Agency (NCA) appear to be making progress on the development of a new internet snooping system and database, which could soon require broadband ISPs and mobile operators to provide various agencies with access to view a log of your online activity.
In case anybody has forgotten, the 2016 Investigatory Powers Act (aka – “snoopers charter“) introduced, among many other things, a new power to force ISPs – upon being ordered to do so by a senior judge – into logging the Internet Connection Records (ICR) of all their customers for up to 12 months (e.g. the IP addresses of the servers you’ve visited and when), which can be accessed without a warrant and may occur regardless of whether or not you’re suspected of a crime.
The Communications Data Code of Practice, which was finalised in 2018, largely indicated that an ICR would “only identify the service that a customer has been using” and this is likely to involve the retention of various different pieces of data (varying between ISPs/networks – depending upon capability).
Advertisement
As we recall, the core ICR data is likely to include a customer’s account reference, the source IP address, destination IP address + port and the date/time of the start and end of an event or its duration. Other data may additionally be added if available (e.g. volume of data transferred and partial URLs – i.e. only that which contains communications data, not content).
Simplified Interpretation of an ICR Log
| Account ID |
Date (Time) | Source IP (You) |
Destination IP:Port | Data Volume | URL |
| 1 | 19/01/2017 (12:01) | 84.56.232.71 | 123.45.62.86:80-HTTP | 800KB | omgprivacyss.com |
| 1 | 19/01/2017 (13:12) | 84.56.232.71 | 65.123.45.90:21-FTP | 0.2KB | ftp.fakefiletest.co.uk |
| 65 | 19/01/2017 (13:14) | 84.79.130.47 | 190.45.62.86:80-HTTP | 1700KB | icanhasyourdata.net |
Records like this are useful because they can help to identify who was using a particular internet service, what communication service they were using (e.g. Facebook) and what devices were connected (e.g. Laptop). Tech-savvy users may of course work around this by using a Virtual Private Network (VPN), DNS over HTTPS (DoH), Proxy Servers or various other anonymising tools.
Recent court challenges mean that, technically speaking, such data can only be stored (or ordered to be stored) if it is considered necessary and proportionate to do so, such as in the course of helping to fight serious crime. But the Government’s definition of what is and is not a “serious” crime has sometimes been called into question. Some of the past court challenges also involved EU courts, but such rulings in post-Brexit land have less sway in the future.
Advertisement
Extract from the IPAct’s Comms Data Code
For the purposes of Parts 3 and 4 of the Act “serious crime”, defined in section 86(2A) of the Act means: an offence for which an adult is capable of being sentenced to one year or more in prison; any offence involving violence, resulting in a substantial financial gain or involving conduct by a large group of persons in pursuit of a common goal; any offence committed by a body corporate; any offence which involves the sending of a communication or a breach of privacy; or an offence which involves, as an integral part of it, or the sending of a communication or breach of a person’s privacy.
Progress so far
Last year we reported that two unnamed broadband or mobile providers (we’d put good money on BT being one of those, as a logical choice) had begun early “small scale” trials of such ICR snooping technology (here), which would be the precursor to launching it as a national service for ISPs to adopt.
Sadly, finding out anything further about those trials, as well as the system in general, is difficult because the IPAct effectively prohibits ISPs from talking about it. But this also causes problems for ISPs too, since it’s much harder for them to share experiences when developing best practices for the code.
However, Public Technology recently uncovered a new procurement notice by the Government’s National Communications Data Service (NCDS), which invited firms to develop and provide support for a tool that would allow authorities to search for ICR information and filter results. Five firms (all SMEs) have been selected as potential providers and up to £2m has been budgeted for the development of the tool and the migration of systems into an AWS environment (i.e. data centre storage provided by Amazon Web Services).
Extract from the Procurement Notice
Under the provisions of the Investigatory Powers (IP) Act, it is now possible for the Law Enforcement Agency (LEA) community to lawfully obtain Internet Connection Records (ICR) in support of their investigations. Following the completion of some initial trial activities, work is now underway to provision a national ICR service. As part of this national service, a central Filtering Arrangement and Results Platform is required, which will be hosted in our NCDS AWS Environment.
To ensure maximum reuse of the trial capabilities, work to evaluate which elements can be migrated to NCDS and which elements need to be rebuilt is ongoing and will be delivered by the existing wider ICR Team. The Technical Migration team (to be sourced under this Outcome) will be responsible for the technical implementation of the national service once this analysis has been completed.
Work on all this is expected to get underway in July and run for between 6-24 months. The notice states that it is working in line with the expectation that a private Beta version of the ‘Filtering Arrangement‘ and ‘Results Platform‘ capability will be available for use against Telecom Operator (TO) data by the “end of 2022“.
Advertisement
Still, we’re sure that all our data will be in good hands as nobody would ever want to abuse a mass national snooping system, no not at all (*tongue firmly in cheek*). As above, getting any kind of official comment on all this is extremely difficult, so we’re doing our best to fill in the blanks with what we know about the law.
As a side note, ICRs can be accessed by various agencies (local authorities and council officials are NOT permitted to access them), albeit primarily this lot (i.e. not only the police and security services):
• Police and National Crime Agency (NCA)
• Intelligence agencies: MIS, MI6 and GCHQ
• HM Revenue and Customs (HMRC)
• Department of Transport
• Department for Work and Pensions
• Serious Fraud Office
• The Scottish and Welsh Ambulance Service Boards
• National Health Service (NHS)
• The Ministry of Defence
• Department of Health
• Ministry of Justice
• Competition and Markets Authority
• Criminal Cases Review Commission
• Food Standards Agency
Finally, the code of practice for all this does consider the size of the telecommunications operator, which may affect whether a provider becomes subject to such a data retention notice and what kind of data they will be expected to supply via ICRs. Put another way, the biggest ISPs are much more likely to become the subject of such a notice and will probably be expected to provide the most detailed level of ICRs.
“Ultimately, however, a notice can only be given where the Secretary of State, having taken into account relevant information, considers it necessary and proportionate to do so and where the decision to do so has been approved by a Judicial Commissioner,” states the code.
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on X (Twitter), Mastodon, Facebook, BlueSky, Threads.net and Linkedin.
« Top Fastest UK Mobile and Home Broadband ISPs for H1 2022
Latest UK ISP News
- FTTP (6053)
- BT (3649)
- Politics (2729)
- Business (2444)
- Openreach (2412)
- Building Digital UK (2336)
- Mobile Broadband (2157)
- FTTC (2086)
- Statistics (1914)
- 4G (1827)
- Virgin Media (1776)
- Ofcom Regulation (1591)
- Fibre Optic (1471)
- Wireless Internet (1464)
- 5G (1417)
I think they’re just trying to legalise what GCHQ was already doing.
Might want to consider a VPN if you haven’t already…
I have been using a VPN for around 2 months now, I never thought about one before, but decided one for my phone when connected to public Wi-fi. Since I have a VPN I started using it for my computer as well.
Totally agree, I’ve had a VPN for a few years now but only activate it when needed (Netflix region, buying plane tickets etc). With this change I’m going to get a tomato router with inbuilt VPN for my entire network, choose whatever DNS I see fit (looking at you BT) and use pihole to strip out the add bollocks (again up yours BT).
@James:
How many separate VPN companies do you think there are?
Who do you think owns them?
Do you really think BT is injecting ads?
Take any measures that make you feel safe, but be under no illusions.
Gary
What about small ISPs?
You would imagine small ISP’s would have to comply, after all its to save us all from the terrorists
Why on earth would a local council need access to your internet connection records? That sounds extremely dystopian to me.
As the article states: “local authorities and council officials are NOT permitted to access them” :). But when I copied and pasted the list of organisations, I neglected to remove one council reference, as that was before the change. Corrected now.
Seems to have been corrected but …. Food standards agency !? What’s that about.
“We would only ever access citizens data in cases of terrorism and heinous crimes of a sexual nature.”
“Also: It is vital that the Food Standards Agency, The Scottish Ambulance Service, the ‘NHS Business Services authority’ and Slough local Council also have access to your data; to ensure they win the fight against terrorisms!”
I don’t get it. Above it says local councils cannot access it. Then directly below it there’s a list where local councils is included. Are they or are they not permitted access? Either way, why would the health service or FSA need your browsing data?
Yes it’s FSA here, we’re executing a search warrant to see if you’ve been looking at pork prices in Europe again.
Removed the council reference, was an older list from before the revision.
Boris the Prying Minister!
Politicians really do dream-up our worst nightmares.
Even using a VPN doesn’t thwart this detection method. VPN providers will need different Ingress and Egress IP’s otherwise the association along with data volume is all too easy.
Yes, but that would be it, just a ton of traffic from your IP to the IP address of the VPN provider, nothing can be inspected inside the tunnel. And no, if you send DNS traffic inside the tunnel there is no URL either (there is no HTTP traffic to be seen).
This will likely become a problem in the future if/when they decide to control VPN usage, in China they know you’re using a VPN but whether they do anything about it depends on how much of a threat the state perceives you to be (like living in Xinjiang).
They could just link it to social credits eventually, 1MB of unlicensed VPN usage = 1 less credit.
Seems that VPNs would be able to thwart this detection method.
This government are playing a very good game of “how to lose your voters” more than they already have. Do they think people will blindly support this in the name of ‘stopping muh baddies’?
Never have VPNs looked so promising, even if they do cooperate with international law enforcement I’d rather that than anything locally!
OK so people vote for other parties… which have the same agenda…
As usual, the only people batting for the people in this area are the Liberal Democrats, in this case proposing that “serious crime” should actually be serious, and not, say, anything that you *can* be prosecuted for a year for, even if that would be extremely disproportionate to the offence: https://www.theyworkforyou.com/lords/?id=2018-10-30a.1289.1#g1291.0
Both Labour and the SNP appear to have abstained, which in essence is agreeing to it without having it on your record. So you know who to vote for next time; but you probably already did, so maybe tell your friends?
@Laurence the problem is that the libdems openly collude with Labour, see the recent by election where they’ve traded candidates, so essentially they are the same party
LibLabConSNP broadly share the same agenda and collude with each other to the general direction more or less the same direction.
The Tories pretend to be pro Brexit while doing their best to undermine it from within, the LibLabSNP pretend to be neutral but will drag the UK back into the EU at the closest opportunity, parliament is filled with snakes.
Clearly they’re not the same party, because as you can see at the end of the link I provided, Labour abstained while the LibDems proposed and voted for the amendment. This is not the first time that this has happened for such topics. Indeed, one of the reasons it took so long for this to start being implemented is because it was prevented by the need to keep them in coalition.
The Liberal Democrats would surely prefer to be in Europe, for the most part, but I do not see how that is relevant. In any case, contrary to what you say, they appear to have been entirely honest and upfront about that, with part of their 2019 platform being “Stop Brexit”: https://www.libdems.org.uk/plan
@Laurence I was referring to the recent by election where Libdem and Labour colluded, trading votes in Wakefield and Tiverton
Sure they may have some different views like the ones you point it out, but it’s all superficial when they collude like this. Abstaining is the same as supporting
In both cases Labour abstaining was effectively supporting the goals of the largest party, i.e. the Conservatives. In practice it might not have made a difference if they hadn’t, but the intent was seemingly to avoid being seen to oppose *or* support the main motion, which might be used against them in the future.
As for the vote trading, I don’t see the issue, as it makes sense under game theory. The end result was one more LibDem elected to replace a Conservative, and one Labour to replace a Conservative (which is at least not any worse). If they had not done that, it is likely that the Conservatives would have got back in, which is counter to both the Labour and LibDem goals.
Routing all traffic down a VPN tunnel just won’t be an option for the masses.
ISP-supplied routers won’t support this setup.
Sadly, a lot of people I speak on the matter don’t seem to care about privacy. I really don’t get it. They seem to have an attitude that they have nothing to hide so they don’t care. They just can’t comprehend the scale at which every aspect of their lives is being spied on.
People who are bothered by this but are not that technical will set up VPN clients on a per device basis and most services will only allow a limited number of simultaneous connections.
Then you have services like Netflix which blanket block IP addresses belonging to known VPN services.
Setting up a private VPN server on a service like Linode or Digital Ocean can get around that but that is for the advanced user.
“Then you have services like Netflix which blanket block IP addresses belonging to known VPN services.”
So does ISPreview. Eg this website blocks users running vpn.ac
Unfortunately, a lot of attack traffic and spambots also make use of VPNs and proxies, which is why some IP ranges do get blocked in order to help protect the site and our visitors. But this is more likely to happen against non-UK/EU based servers. Some particular web servers are also very poor at clamping down on such activity.
I have Surfshark VPN and I have no services blocked
I assume other commercial VPNs open the door for the masses to have internet security.
Another disgusting move by the government’s war on free speech. This is not a slippery slope, it is a recognized and confirmed pattern
It won’t be long until we become Australia/Korea where posting “the government is dumb” on facebook can lead you to get arrested (happened multiple times in those countries, go look it up) and eventually become like China where the govt picks and chooses what content is okay and arrests anyone looking at anti-govt content
It’s insane how little opposition to this there is. Klaus Schwab has all the main UK parties and press under his thumb
Quite a few VPN apps now allow app whitelistin for Netflix etc.
The average joe has always been the foundation upon which tyranny is built.
I presume that Apple’s private browsing will defeat this? 51% (depends who you ask, 50% with some sources) of mobile phone users have an Apple phone in the UK. This equates to millions of people who can defeat it just by owning that brand of phone. I hope Google does the same with Android.
Then we can all get another laugh watching the gov wanting backdoors in encryption so they can “save the children” or whatever other subterfuge they want to use.
MI5/GCHQ etc, kindly take a long walk off a short pier.
Some people right now would say you have nothing to hide but your comment could earn you 10 years in the gulag in the future.
Who knew that DWP helped fight terrorists with MI6 that list of agency’s that can access it is insane
These are the kinds of records that telephone companies already need to keep for your usage of the phone. Date, time, duration, and destination.
DNS-Over-HTTPs won’t really mask much, because the IP address you ultimately connect to can be reversed. It’s not very accurate if the IP hosts multiple sites, but it’s probably not a big obstacle.
I wonder if the European Convention of Human rights will kick in on this one like they always kick in when we try and deport foreign child murderers out of the UK….
I wonder if they’ll also step in to save Assange…
Now you know why they want to get rid of that.
EUCHR is not binding.
i.e. countries are free to ignore their rulings.
related note: the EU is planning on reading everyone’s emails/texts/whatsapp they voted yes on it last year.
I can see no oversight, the sheer amount of organisations that are on the permitted list is extensive, I’m not aware of a SINGLE instance where HMRC have thwarted a terrorist attack on the country.
Really we are not far away from VPNs being outlawed and encryption turned off, mind a net benefit from that, our banks will have to reopen branches.
Some tongue in cheek and sarcasm in my comments.
If VPNs are banned then after that the govt installs a mandatory camera inside people’s homes and microchips everyone with GPS trackers. Privacy is a dream of the past
It will be hard for them to outlaw VPNs and turn off encryption.
@Anonymous
Hasn’t stopped the Chinese government from doing it.
@Mike, yes it does.
Lots of Chinese people use VPN’s and encryption is still a thing in China.
Their main way of controlling it is by making it hard to get access to a VPN service to purchase it. Also sending any VPN connection DNS requests into a black hole.
However VPN’s that allow the use of IP addresses to connect to their servers still work there.
Source – my partner is Chinese and I have been there many times.
I’m a bit confused here as a non network specialist. I’m not that familiar with DoH so I’ve done some light reading and enabled it on my phone, however there are a lot of pros and cons so I understand. Mark has suggested DoH ISA way to circumvent tracking but I’m not sure it is, can we have some clarity?
If I recall correctly, at the moment when you type a website into your browser that is then sent to a DNS server (typically your ISP’s) who then finds the IP address of that website and sends it back to your browser so you can then connect to/view it.
DoH sends that query to Cloudflare/Google (or another DoH provider) so the ISP will only know that you connected to an IP, they won’t know what website on that IP you connected to, although most of the time there is 1 website per IP, so it’s not very good at stopping tracking.
I think the main reason for the government/ISP’s complaining about it was that they’d have to reconfigure their surveillance apparatus and that in theory a defendant could say they connected to something else on the IP, not the website itself.
With a VPN the ISP will only see your connected to the VPN IP regardless of which website/server you connected to whilst connected to the VPN, which is why if you want to stop ISP/government snooping this is generally the preferred option.
DNS-over-HTTPS may make government surveillance of the names of websites that you attempt to access slightly harder (and avoids their attempt to block those which are seen to be illegal by giving you a false address).
This proposal is instead to monitor connections, i.e. what happens after you get the name converted to an address. It’s like monitoring your actual call records rather than use of the phone directory.
Using a different name directory with DNS-over-HTTPS is less likely to impact this kind of tracking, although it might potentially help if the end result is the use of services hosted outside of the UK. Probably not much, though.
In theory one address may host many services on many domain names, but most do not and some of the protocols disclose what domain name the request is for (including HTTPS, usually, because otherwise it can’t respond with the right certificate).
So I suppose DoH is perhaps slightly better than not using it at all but no comparison to a VPN? Don’t get me wrong I like the government attempts to maintain national security but this move is turning us into a surveillance state without justification. All that my records are likely to indicate are news sites and some work related stuff but this is the principle of what’s necessary and proportionate to keep us safe; I don’t feel this carte blanche approach is.
National security is often the alibi of tyrants.
In the 30s/40s a few (axis) nations used punch card machines, they fed in the census data, then set the criteria (ie: Jews), it then spat out a list of all the people they wanted, who they then rounded up and imprisoned/exterminated etc. also turns out the countries they conqueror had similar systems which they could then use the data from to continue the persecutions there.
Now imagine what a government could do today with much more advanced computers and much more information on you.
People often make the mistake of thinking they have nothing to hide, but it is not the individual who decides what you have to hide, it’s the state.
Early tabular cards before they had punch card readers in mainframes.
History lesson. Yes it’s the state and you don’t know how future politics will change to persecute individuals with certain beliefs or sexuality. Beliefs, religion or gender identity or sexuality. The them and us of politics and state.
A penny off a pint will convince many to welcome the intrusion and the government “accidentally” selling of your data to anyone that will pay for it.
Better make sure your TV licence is valid if you’ve accessed iPlayer etc… This isn’t about terrorism it’s about revenue generation.
This is from the same people who broke lockdown laws. They’ll break this rule.
I’ve already seen a post from Stassie Karanikolaou criticizing our government, calling them “hypocrites who treat little people like toys”.
We need to go back to how the Internet was from 2007-2013, content-driven and less tribalistic.
Stassie Karanikolaou is an expert on Internet things, and she’s already called for China to drop its Great Firewall.
Then you have public figures like Bella Hadid who have already said that the British government is so corrupt they can’t run the country properly and treat citizens as a joke.
The public have more trust in Stassie Karanikolaou than they ever did in Boris Johnson and Nicola Sturgeon, which shows how unpopular the government is.
Looks like Christine Evangelista was right in 2014 when she said the British government didn’t understand the Internet!
Incidentally, anyone remember DontStayIn.com, Jane S-D of the Sugah Dee Dancers which was a meme from 2006 to 2014? Well, even she’s spoken out about the government, saying if they’ll break their own lockdown rules, they’ll do so about the Internet.
Firefox has a ton of VPN addons, which work well on a Mac but most people wouldn’t know of them.
Still, VPN’s have MANY LEGITIMATE USES, but the government don’t understand that.
I think I might know who the other ISP is.
I’m one of these sad people who ACTUALLY READS the terms & conditions and when we got Vodafone last year I was a bit surprised to see exactly how much information they say they *may* acquire & store. Not only is it external metrics like the sites you visit & when but also the MAC addresses of devices connected to their router along with their computer / network name (presumably obtained via either NetBIOS lookup or the DHCP lease request). I don’t understand how this information could be useful for ISP diagnostics but I can easily see how it might be necessary for RIPA compliance.
If your DNS is via HTTPS or TLS just how does the ISP monitor what sites you visit?
Yes. Most https connections use SNI, sending the site address (not the full URL) in plain text. Essentially they know that you visit this site, but not that you’ve watched this page.
If a site is big enough to have a dedicated service with a dedicated IP, then the SNI info isn’t needed, but then it’s easy to associate connections to that IP with a website.
On top of this one can deduce what the user is doing by just looking at the traffic (eg: reading a page, streaming video, making a call, etc).
We should protest against this by causing a MASSIVE amount of logging data for the ISPs.
It wouldn’t be too difficult to make a small app which calls millions of random websites an hour. Similar to trackthis.link but on a massive scale which runs 24/7 on a raspberry pi. If enough people did this it would make logging a nightmare for all involved.