Tens of thousands of US-based organizations are running Microsoft Exchange servers that have been backdoored by threat actors who are stealing administrator passwords and exploiting critical vulnerabilities in the email and calendaring application, it was widely reported. Microsoft issued emergency patches on Tuesday, but they do nothing to disinfect systems that are already compromised.

KrebsOnSecurity was the first to report the mass hack. Citing multiple unnamed people, reporter Brian Krebs put the number of compromised US organizations at at least 30,000. Worldwide, Krebs said there were at least 100,000 hacked organizations. Other news outlets, also citing unnamed sources, quickly followed with posts reporting the hack had hit tens of thousands of organizations in the US.

Assume compromise

“This is the real deal,” Chris Krebs, the former head of the Cybersecurity and Infrastructure Security Agency, said on Twitter, referring to the attacks on on-premisis Exchange, which is also known as Outlook Web Access. “If your organization runs an OWA server exposed to the internet, assume compromise between 02/26-03/03.” His comments accompanied a Tweet on Thursday from Jake Sullivan, the White House national security advisor to President Biden.

Hafnium has company

Microsoft on Tuesday said on-premises Exchange servers were being hacked in “limited targeted attacks” by a China-based hacking group the software maker is calling Hafnium. Following Friday’s post from Brian Krebs, Microsoft updated its post to say that it was seeing “increased use of these vulnerabilities in attacks targeting unpatched systems by multiple malicious actors beyond HAFNIUM.”

Katie Nickels, director of intelligence at security firm Red Canary, told Ars that her team has found Exchange servers that were compromised by hackers using tactics, techniques, and procedures that are distinctly different than those used by the Hafnium group Microsoft named. She said Red Canary has counted five “clusters that look differently from each other, [though] telling if the people behind those are different or not is really challenging and unclear right now.”

On Twitter, Red Canary said that some of the compromised Exchange servers the company has tracked ran malware that fellow security firm Carbon Black analyzed in 2019. The malware was part of an attack that installed cryptomining software called DLTminer. It’s unlikely Hafnium would install a payload like that.

Microsoft said that Hafnium is a skilled hacking group from China that focuses primarily on stealing data from US-based infectious disease researchers, law firms, higher-education institutions, defense contractors, policy think tanks, and nongovernmental organizations. The group, Microsoft said, was hacking servers by either exploiting the recently fixed zeroday vulnerabilities or by using compromised administrator credentials.

It’s not clear what percentage of infected servers are the work of Hafnium. Microsoft on Tuesday warned that the ease of exploiting the vulnerabilities made it likely other hack groups would soon join Hafnium. If ransomware groups aren’t yet among the clusters compromising servers, it’s almost inevitable that they soon will be.

Backdooring servers

Brian Krebs and others reported that tens of thousands of Exchange servers had been compromised with a webshell, which hackers install once they’ve gained access to a server. The software allows attackers to enter administrative commands through a terminal Window that’s accessed through a web browser.

Researchers have been careful to note that simply installing the patches Microsoft issued in Tuesday’s emergency release would do nothing to disinfect servers that have already been backdoored. The webshells and any other malicious software that have been installed will persist until it is actively removed, ideally by completely rebuilding the server.

People who administer Exchange servers in their networks should drop whatever they’re doing right now and carefully inspect their machines for signs of compromise. Microsoft has listed indicators of compromise here. Admins can also use this script from Microsoft to test if their environments are affected.

This week’s escalation of Exchange server hacks comes three months after security professionals uncovered the hack of at least nine federal agencies and about 100 companies. The primary vector for infections was through software updates from network tools maker SolarWinds. The mass hack was one of—if not the—the worst computer intrusions in US history. It’s possible the Exchange Server will soon claim that distinction.

There’s still much that remains unknown. For now, people would do well to follow Chris Krebs’ advice to assume on-premises servers are compromised and act accordingly.

First it was SolarWinds, a reportedly Russian hacking campaign that stretches back almost a year and has felled at least nine US government agencies and countless private companies. Now it’s Hafnium, a Chinese group that’s been attacking a vulnerability in Microsoft Exchange Server to sneak into victims’ email inboxes and beyond. The collective toll of these espionage sprees is still being uncovered. It may never be fully known.

Countries spy on each other, everywhere, all the time. They always have. But the extent and sophistication of Russia’s and China’s latest efforts still manage to shock. And the near-term fallout of both underscores just how tricky it can be to take the full measure of a campaign even after you’ve sniffed it out.

By now you’re probably familiar with the basics of the SolarWinds attack: Likely Russian hackers broke into the IT management firm’s networks and altered versions of its Orion network monitoring tool, exposing as many as 18,000 organizations. The actual number of SolarWinds victims is assumed to be much smaller, although security analysts have pegged itin at least the low hundreds so far. And as SolarWinds CEO Sudhakar Ramakrishna has eagerly pointed out to anyone who will listen, his was not the only software supply chain company that the Russians hacked in this campaign, implying a much broader ecosystem of victims than anyone has yet accounted for.

“It’s become clear that there’s much more to learn about this incident, its causes, its scope, its scale, and where we go from here,” said Senate Intelligence Committee chair Mark Warner (D-Virginia) at a hearing related to the SolarWinds hack last week. Brandon Wales, acting director of the US Cybersecurity and Infrastructure Agency, estimated in an interview with MIT Technology Review this week that it could take up to 18 months for US government systems alone to recover from the hacking spree, to say nothing of the private sector.

That lack of clarity goes double for the Chinese hacking campaign that Microsoft disclosed Tuesday. First spotted by security firm Volexity, a nation-state group that Microsoft calls Hafnium has been using multiple zero-day exploits—which attack previously unknown vulnerabilities in software—to break into Exchange Servers, which manage email clients including Outlook. There, they could surreptitiously read through the email accounts of high-value targets.

Bit flips are events that cause individual bits stored in an electronic device to flip, turning a 0 to a 1 or vice versa. Cosmic radiation and fluctuations in power or temperature are the most common naturally occurring causes. Research from 2010 estimated that a computer with 4GB of commodity RAM has a 96 percent chance of experiencing a bit flip within three days.

An independent researcher recently demonstrated how bitflips can come back to bite Windows users when their PCs reach out to Microsoft’s windows.com domain. Windows devices do this regularly to do things like making sure the time shown in the computer clock is accurate, connecting to Microsoft’s cloud-based services, and recovering from crashes.

Remy, as the researcher asked to be referred to, mapped the 32 valid domain names that were one bitflip away from windows.com. He provided the following to help readers understand how these flips can cause the domain to change to whndows.com:

Of the 32 bit-flipped values that were valid domain names, Remy found that 14 of them were still available for purchase. This was surprising because normally, Microsoft and other companies buy these types of one-off domains to protect customers against phishing attacks. He bought them for $126 and set out to see what would happen. The domains were:

• windnws.com
• windo7s.com
• windkws.com
• windmws.com
• winlows.com
• windgws.com
• wildows.com
• wintows.com
• wijdows.com
• wiodows.com
• wifdows.com
• whndows.com
• wkndows.com
• wmndows.com

No inherent verification

Over the course of two weeks, Remy’s server received 199,180 connections from 626 unique IP addresses that were trying to contact ntp.windows.com. By default, Windows machines will connect to this domain once per week to check that the time shown in the device clock is correct. What the researcher found next was even more surprising.

“The NTP client for windows OS has no inherent verification of authenticity, so there is nothing stopping a malicious person from telling all these computers that it’s after 03:14:07 on Tuesday, 19 January 2038 and wreaking unknown havoc as the memory storing the signed 32-bit integer for time overflows,” he wrote in a post summarizing his findings. “As it turns out though, for ~30% of these computers doing that would make little to no difference at all to those users because their clock is already broken.”

The researcher observed machines trying to make connections to other windows.com subdomains, including sg2p.w.s.windows.com, client.wns.windows.com, skydrive.wns.windows.com, windows.com/stopcode, and windows.com/?fbclid.

Remy said that not all of the domain mismatches were the result of bitflips. In some cases they were caused by typos by people behind the keyboard, and in at least one case the keyboard was on an Android device, as it attempted to diagnose a blue-screen-of-death crash that had occurred on a Windows machine.

To capture the traffic devices sent to the mismatched domains, Remy rented a virtual private server and created wildcard domain lookup entries to point to them. The wildcard records allow traffic destined for different subdomains of the same domain—say, ntp.whndows.com, abs.xyz.whndows.com, or client.wns.whndows.com—to map to the same IP address.

“Due to the nature of this research dealing with bits being flipped, this allows me to capture any DNS lookup for a subdomain of windows.com where multiple bits have flipped.”

Remy said he’s willing to transfer the 14 domains to a “verifiably responsible party” and in the meantime will simply sinkhole them, meaning he will hold onto the addresses and configure the DNS records so they are unreachable.

“Hopefully this spawns more research”

I asked Microsoft representatives if they’re aware of the findings and the offer to transfer the domains. The representatives are working on getting a response. Readers should remember, though, that the threats the research identifies aren’t limited to Windows.

In a 2019 presentation at the Kaspersky Security Analysts Summit, for instance, researchers from security firm Bishop Fox obtained some eye-opening results after registering hundreds of bitflipped variations of skype.com, symantec.com, and other widely visited sites.

Remy said the findings are important because they suggest that bitflip-induced domain mismatches occur at a scale that’s higher than many people realized.

“Prior research primarily dealt with HTTP/HTTPS, but my research shows that even with a small handful of bitsquatted domains you can still siphon up ill-destined traffic from other default network protocols that are constantly running, such as NTP,” Remy said in a direct message. “Hopefully this spawns more research into this area as it relates to the threat model of default OS services.”