Rackspace Hosted Exchange Still Offline Over Security IssueThousands of Affected Customers Urged to Use Microsoft 365 - At Least Temporarily
Update: Dec. 5, 2022 21:24 UTC: Rackspace shares finished trading Monday down 15.46% with no additional update on the status of the company’s ongoing Exchange outage. The Nasdaq Composite, an index tracking performance of the tech company-heavy exchange on which Rackspace trades, ended the day down 1.9%.
Thousands of Rackspace customers globally continue to face Microsoft Exchange Server outages that the managed services giant says trace to a security incident.
Rackspace says it will keep its hosted Exchange service offline indefinitely while it investigates the specific problem, which it has not yet publicly identified.
Texas-based Rackspace is the world's largest managed cloud provider, counting more than 300,000 customers worldwide, including two-thirds of the world's 100 largest publicly traded businesses.
The problems began Thursday night, with Rackspace reporting Friday afternoon that it had "experienced a significant failure in our Hosted Exchange environment" and that it "proactively shut down the environment to avoid any further issues while we continue work to restore service" and identify "the root cause of the issue." Subsequently, the company reported that the root cause involved an apparent attack.
Rackspace has recommended all affected customers move to Microsoft 365 as a temporary mitigation and says it is providing them with "Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice" at no additional cost. Services affected by its Hosted Exchange offering that remain offline include MAPI/RPC, POP, IMAP, SMTP, ActiveSync and the Outlook Web Access interface used to access Hosted Exchange instances for managing email online.
Early Monday, Rackspace reported it had made progress with service restoration - again, provided hosted Exchange customers cease using its hosted Exchange service. Instead, they must either temporarily forward their email to an external address for every user or migrate from hosted Exchange to Microsoft 365.
"We have successfully restored email services to thousands of customers on Microsoft 365 and continue to make progress on restoring email service to every affected customer," Rackspace says in a Hosted Exchange Disruption update posted early Monday. "At this time, moving to Microsoft 365 is the best solution for customers who can now also implement temporary forwarding."
Rackspace says that while its own Rackspace Email service is unaffected by the security incident and its mitigation efforts, any customer with a hybrid hosting environment that uses both Rackspace Email and Exchange for a single domain "will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change."
On Sunday, Rackspace told customers that it had "committed extensive internal resources and engaged world-class external expertise in our efforts to minimize negative impacts to customers."
"We continue to make progress in addressing the incident," it added. "The availability of your service and security of your data is of high importance."
Multiple users reported long delays and dissatisfaction with migrating their emails to M365, as well as frequently getting disconnected from telephone support. "Sat on hold today for 5 hours. Yes 5 hours," one customer reported via Twitter.
Another customer also detailed an hourslong wait time. "I need my emails recovered and access to my email account. I have lost so much money over the past three days," the customer tweeted.
Customers have also threatened class action lawsuits over the downtime.
All @Rackspace Customers: I have retained lawyers 2 hold Rackspace/Officers/Directors Accountable 4 Our Damages 4 Unacceptable Outage/Terrible Treatment of Customers. C Picts of email I just sent: follow me or DM 4 more info #RackspaceOutage #rackspace @stephatkins @JaieAvila pic.twitter.com/MkFo9SDe0k— Scott C. Smith (@WaterWarriorOne) December 3, 2022
Rackspace didn't immediately respond to a request for comment.
Did Attackers Bypass ProxyNotShell Mitigations?
While the company has yet to share details of the security incident, British cybersecurity expert Kevin Beaumont says evidence suggests Rackspace was running Microsoft Exchange servers that remained vulnerable to the two flaws that are known as ProxyNotShell.
Microsoft patched the pair of Exchange zero-day vulnerabilities in early November, after they were first publicly disclosed in late September and known to have been exploited in the wild by a threat actor displaying indicators of Chinese origin (see: Microsoft Patches ProxyNotShell Exchange Vulnerabilities).
Beaumont says research with the Shodan internet of things search engine suggests Rackspace was using at least some Exchange clusters with build numbers from August, which predate Microsoft's patches.
Rackspace may have put Microsoft-recommended mitigations in place until it could patch all systems, such as creating rules using the IIS Rewrite module and implementing them via PowerShell scripting or the Exchange Emergency Mitigation service.
But if so, the systems would still have been at risk.
"The Microsoft-supplied mitigations for ProxyNotShell are bypassable," Beaumont says. "IIS Rewrite, which Microsoft used for mitigations, doesn't decode all URLs correctly and as such can be bypassed for exploitation. If you relied on the PowerShell mitigation or the EEMS application, your Exchange Server is still vulnerable - Microsoft just haven't told you this clearly. The fix is to patch."
Executive Editor Mathew Schwartz contributed to this report.