In 2014 Dan Geer, a computer security analyst, gave a speech at the RSA Conference, an annual gathering of cyber-security specialists, titled: ‘We Are All Intelligence Officers Now’. It described the ways in which computers were insinuating themselves into every aspect of life, the resulting haemorrhage of data, and the change in what it meant to be a collector of intelligence. In his talk, Geer asked: ‘Is it possible that in a fully digital world it will come to pass that everyone can see what once only a director of national intelligence could see?’

Fast forward and it is possible to see Geer’s vision being realised. For a flavour of this, consider an episode that unfolded in 2021. Analysts noticed that CCTV cameras in Taiwan and South Korea were digitally talking to crucial parts of the Indian power grid – for no apparent reason. On closer investigation, the strange conversation was the deliberately indirect route by which Chinese spies were interacting with malware they had previously buried deep inside the Indian power grid. The analysts were in a position to observe this because they had been scanning the entire internet to find command and control (C2) nodes – such as the offending cameras – that hackers tend to use as pathways to their victims.

The attack was not foiled by an Indian intelligence agency or a close ally. It was discovered by Recorded Future, a company in Somerville, Massachusetts, which claims to have knowledge of more global C2 nodes than anyone in the world, and which it uses to constantly disrupt Chinese and Russian intelligence operations. The firm, like others, also scrapes vast amounts of data from the dark web – a part of the internet that can only be accessed using special software – collects millions of images daily, extracts visible text to find patterns, and hoovers up corporate records.

The Chinese intrusion serves as a microcosm for intelligence in the modern age. The cameras in Taiwan and South Korea are among more than one billion around the world, forming a metastasising network of technical surveillance – visual and electronic, ground-level and overhead, real-time and retrospective – that has made life far harder for intelligence officers and the agents they need to develop, recruit and meet. That those cameras could be used to sabotage India’s electricity supply shows how digital technology has enabled covert action on a grand scale; what previously required front companies, physical infrastructure and agents carrying tools of sabotage can now be done virtually. That this could be watched in near real-time by a private company illustrates the revelatory quantity and quality of data that oozes out of the digital world. Intelligence is being democratised – blurring the boundary between what is secret and what is public.

As society has migrated to the internet, so have its secrets, and, therefore, so has intelligence. Consider the deep web, a part of the internet that is not indexed by search engines, and the dark web, which requires specialised software to access. They offer a degree of anonymity attractive to a variety of unsavoury people: terrorists, paedophiles, drug dealers, and cyber-criminals. But that anonymity is superficial.

Consider the example of Flashpoint, a so-called threat intelligence firm. Its original work involved building fake personas, such as an analyst pretending to be a jihadist, to infiltrate extremist groups online and gather information about their plan – a form of virtual human intelligence. It now normally deals in data. By tracing extremist groups’ cryptocurrency ‘wallets’, for instance, you can spot anomalous movements that might hint at an impending attack. This kind of intelligence can be semi-secret: tucked out of sight, accessible but often ephemeral. Joseph Cox, a journalist, notes that administrators of criminal and hacker chat rooms on Telegram, a social media platform, frequently wipe messages in one channel and establish another. ‘It really is like missing a whispered conversation in the bar.’ Collecting those messages requires vigilance or automation.

If one approach is to observe what is happening out there – on the internet, on the deep and dark week, in particular places – then another is to combine that with what is happening inside one’s own networks  – ‘in host’. The firms that build key hardware and software – Google for email, Microsoft for operating systems, and Amazon for cloud computing, to name a few – have unprecedented and unmatched insight into the traffic moving over their networks. The result is that these companies are, in one sense, the largest signals intelligence agencies on the planet. Microsoft tracks more than 78 trillion ‘signals’ per day.

These companies observe not just the traffic on their own networks but, like counter-intelligence services, map and track the activities and signatures of their adversaries, including state-linked hacking groups known as advanced persistent threats or APTs. It was Microsoft, not the American government, which publicly revealed that ‘Volt Typhoon’, a Chinese hacking group, had targeted American critical infrastructure since at least 2021, including water and energy facilities, probably as preparation for wartime sabotage. The fact that Western cyber-security companies have been involved in the defence of Ukrainian networks from the earliest days of the war means that they also see some Russian cyber threats that Western agencies might not be aware of.

Private intelligence companies are not unconstrained, however. They are subject to the law. They may not break into buildings, as domestic security services can. They may not breach computer networks in violation of hacking laws, as a cyber intelligence agency might do. Many of them are also proprietorial and cagey about protecting their methods, data and clients. Yet the open nature of the private sector can also be an advantage. Thomas Rid of Johns Hopkins University has noted that counter-intelligence work was once ‘highly secretive’ and ‘cloistered in small teams and communities’ – think of the CIA’s notorious James Angleton, a spycatcher who became a reckless paranoiac.

What changed in the 2010s was the maturation of ‘digital counter-intelligence’, most notably in the field of cyber threat intelligence. Companies began openly countering Russian and Chinese hacking, often publishing their findings in great detail. The debate, explains Rid, became ‘more evidence-based and far less secretive’. These companies were often hunting the same groups of hackers from China, Russia, North Korea and Iran and they created a community of learning and tradecraft, in which different parts of the jigsaw could be put together. People often moved between firms, but also between intelligence agencies and the private sector, bringing know-how with them.

All this is an opportunity for spycraft. For one thing, it expands collection capacity. Take the example of the Falklands War. America found that its spy satellites, designed to watch the Soviet Union, were in the wrong orbit to point at the South Atlantic (‘Nobody ever thought there’d be a damn war in the Falklands for God’s sake’, noted Robert Gates, later the CIA director). The private sector has since solved that problem. The spectacular growth of the commercial satellite industry allows states to enjoy near-blanket coverage. Britain has gone from buying hundreds of thousands of dollars of commercial satellite images every year to multiple millions. Other examples abound. In Gaza, for instance, Israel’s armed forces and signals intelligence units have used private firms, including Google Photos, to assist with facial recognition of Palestinians.

A second advantage is that secrets acquired by non-secret agencies can be shared more widely. In space intelligence, for instance, according to the historian Aaron Bateman, the United States rarely shared satellite images with its NATO allies except Britain. In some cases it did not acknowledge certain sorts of satellites, such as those which collected radio emissions or which used synthetic aperture radar, even existed. That began to change in 1991 during the first Gulf War. But it is now routine for governments to buy and publish high-resolution satellite images to expose malfeasance by an adversary.

Governments can also tip off outside analysts to look for certain things that they want to be publicised, and those analysts often stumble on intriguing things themselves. In August 2021, there were rumours that China was building new ICBM launch sites. Decker Eveleth, a young analyst, looked for them using common sense: they would be on flat land, and far from American radars in Japan and South Korea. Having slogged through satellite images of Inner Mongolia without luck, he found what he was looking for in next-door Gansu: 120 missile silos under construction. Open-source analysts later found the same telltale grid pattern in a remote part of Xinjiang.

Intelligence agencies offer recruits the allure of working for organisations with a sparkling history, a mandate for public service and a licence if not to kill then to break domestic and foreign laws in service of the state. The drawbacks have grown more prominent. ‘It’s a hard sell to anybody who’s in a leading AI lab to join the intelligence community and then be told you’ll have to wait a year to get a security clearance,’ says Jason Matheny of RAND. The chasm in salaries is another issue. Working conditions are a third. ‘We cannot offer certain conditions that are taken for granted today,’ notes Bruno Kahl, the head of the BND, Germany’s foreign intelligence service. ‘Remote work is barely possible… and not being able to take your cellphone to work is asking much from young people.’ When Joe Morrison of Umbra, a radar satellite start-up, was asked by Western officials why they ought to work with commercial unclassified satellite vendors, his reply was both glib and truthful: ‘Access to talent that likes to smoke weed.’

The most radical interpretation of all these changes is that Western intelligence is broken and needs to start again from scratch. ‘The UK intelligence community (UKIC) is facing an existential challenge,’ argued Lucy Mason, a former British defence official, and Jason M, a semi-anonymous serving intelligence official, in a paper published by the Alan Turing Institute, a research centre in London that works closely with the intelligence services, in November. ‘It is being out-competed by providers of open-source intelligence and data companies.’ The authors proposed a completely new model ‘away from one where national security is done only by some cleared people in highly centralised, closed, organisations, to one which is open, collaborative, and joined up by design’.

This is probably going too far. To be sure, non-secret sources are increasingly important. Open source contributed around 20 per cent of British defence intelligence ‘current processes’, noted General Jim Hockenhull, then chief of the service, in late 2022, ‘but the availability and opportunity means that we’ve got to invert this metric.’ The same appetite exists in the non-military intelligence world. ‘If I’d gone and collected all of China’s military procurement records, I’d probably have got an OBE,’ says a former British intelligence officer. ‘The fact that they were, for many years, just sat there in open source just completely bypassed everybody.’ A flourishing trade in personal location data harvested by advertising brokers from apps on mobile phones is a rich seam for state agencies around the world. In April 2024 America’s communications regulator levied $200m in fines on the country’s largest telecoms firms for selling such data without permission to firms who then sold it on again.

There are limits to private-sector intelligence. The fact that public data can answer many questions that would once have required secret intelligence does not mean they can answer all such questions. Open sources did shine a light on Russia’s military build-up before the invasion of Ukraine in 2022. Nonetheless, only states had access to the most incriminating evidence, such as intercepts of Russian war plans and indicators that Russia was, for instance, moving blood plasma to the front lines at a crucial moment in mid-January 2022. No commercial or public source has established Russia’s development of an orbital nuclear weapon, Iran’s provision of ballistic missiles to Russia, or Iran’s computer-modelling work relevant to the design of nuclear weapons – all recent stories in the public domain that are based on secret intelligence collected by states.

The second problem is that it is misleading to think about open and secret sources as two separate things, kept apart from one another. Sometimes the former can substitute for the latter, at least to a reasonable degree. Public estimates of losses of Russian military equipment in Ukraine appear to be pretty accurate. But public data is often most useful and revealing when it is fused with something that is non-public, or secret. The problem is that bridging the unclassified (the ‘low side’, as government officials call it) and the classified (‘high side’) world is both technically and institutionally difficult. Consider, for instance, the case of a spy agency which has its own data on the movement of Russian intelligence officers, perhaps acquitted by tracking phones or devices. It may wish to juxtapose that with a publicly available database of visa or travel records – perhaps one leaked on the dark web.

‘What’s actually sensitive is the question you ask,’ says a person familiar with this sort of operation. ‘As soon as the question comes from the high side down onto the low, that question is detectable and the data you pull up is detectable.’ In other words, interrogating the public dataset can reveal what you do or do not know about  Russian spies, perhaps tipping them off. But pushing all the data up onto the high side is too expensive because cloud computing built to handle highly classified data is a scarce resource for all but the very wealthiest of governments. Western agencies are still grappling with this problem, with many reformists frustrated at the slow pace of change in their organisations. ‘If you’re not willing and able to engage with the world of data’, complains the insider, ‘you just cannot be efficient, and your costs go up’.

The third issue has to do with the legal and ethical challenges that arise when states are competing over access to data and its exploitation. China has long seen the acquisition of data as a key resource in its strategic competition with America and the wider West. In 2015 Chinese hackers stole more than 22 million American government security clearance records held by the Office of Personnel Management. In 2017 they acquired the records of 148 million Americans and 15 million Britons from Equifax, a credit reporting agency. In 2021 they targeted Britain’s electoral commission. In February 2024 files leaked from iSoon, a Shanghai-based firm that hacks and then sells data to Chinese government entities, showed the range of its ambition: immigration data from India, phone logs from South Korea, and road-mapping data from Taiwan.

This activity spans a broad range. Much of it is traditional intelligence gathering. Some of it enables China to catch Western spies. Both of those things are no different to what Western spy agencies would do in the other direction, but it also offers other possibilities. ‘Building databases of society has been [Chinese] intelligence… methodology since the 1930s,’ writes Peter Mattis, a China expert and former CIA analyst. ‘Start with the broadest possible data on individuals, then filter and target them for intel and influence.’ Some people would like the West to learn from this approach. ‘If we do not find a way to merge the great capabilities of Western governments and the private sector to defend our own values and interests’, argues Duyane Norman, a former CIA officer, ‘these adversaries will continue to close the gap.’

That is easier said than done. Democracies tend to impose stringent requirements on the sort of thing that may or may not be collected. In Britain the intelligence agencies do collect ‘bulk personal data’, but if they want to ‘retain or examine’ it then they must jump through a few hoops: they need to get a warrant and then show that getting, keeping and using it is proportionate to some specific aim. It is not enough to believe that it might prove useful. Some data is thus ‘more easily accessed and used by the private sector than by government organisations’, write Lucy Mason and Mr M, the authors of the paper published by the Alan Turing Institute.

American spies are similarly constrained. It is ‘hard or impossible’ to ‘identify and scrub’ data on Americans from large datasets, notes Emily Harding, a former CIA analyst now at CSIS, a think-tank, making it hard to comply with the law. American agencies are thus ‘far behind private sector entities with no such restrictions’, she says. One former European intelligence official observes that the VENONA project, a celebrated Allied effort to collect and slowly decrypt Soviet wartime intelligence transmissions, which eventually revealed a number of Soviet agents in the West, would not have been possible under the law as exists today in some European countries.

In 2013 the disclosures by Edward Snowden, a disgruntled contractor working for America’s National Security Agency, prompted an intense and unexpected public debate over the activities of intelligence agencies and their ability to collect, if not actively read, vast amounts of phone, internet and other traffic. In the decade since, much has changed. The majority of internet browsing and personal messaging now takes place with the protection of end-to-end encryption, making it harder for spies to read what they might intercept. More data is also being encrypted ‘at rest’ – on devices, and in use. That trend, too, has been driven by the private sector, as large tech companies – Apple, Google and Meta, above all – have embraced encryption and user privacy in the face of opposition from law enforcement agencies around the world.

At the same time, daily life relies more than ever on digital technology: more things run on software (fridges, cars, phones), those things have a greater array of sensors (GPS receivers and radio transmitters) and they are increasingly connected, often over the internet, allowing data, often embodying our most personal secrets, to flow to and fro. The paradox of the modern world is that, while we have more means to keep our data secret, there is so much more data to contend with and so many more places from where it can seep out into the world, where a sprawling ecosystem of private intelligence can collect, analyse and use it.