Facebook’s seven-hour blackout that cost the company an estimated $100million in lost revenue alone was made worse because the staff were working from home, an insider has claimed.
The global outage – which hit Facebook, Instagram, WhatsApp and Messenger on Monday – was caused when a faulty update that disconnected its servers from the internet, meaning engineers had to travel to its Santa Clara data centre to fix the glitch in person, an insider told Daily Mail.
But the repair was delayed, according to one insider who was posting on Reddit, because of ‘lower staffing in data centres due to pandemic measures’.
According to the newspaper, the glitch, which has prompted calls for a break-up of big tech firms, also brought down messaging services that remote-working staff use to communicate, so those who knew how to fix the servers couldn’t get that information to the teams inside the data-centre, the insider said.
Also disabled were key-fob entry systems at Facebook’s main campus in Menlo, meaning those who had been WFH but rushed back to the office could not get inside while those already inside were unable to access conference rooms and other areas that required a pass.
‘There are people now trying to gain access to… implement fixes, but the people with physical access is separate from the people with knowledge of how to authenticate the systems and people who know what to do, so there is now a logistical challenge,’ the insider said.
Industry sources who have worked closely with the tech giant say Facebook is suffering from two major problems: Staff working from home and over-reliance on artificial intelligence.
The social media site has been beset by bugs, glitches and AI issues for months – exacerbated by staff not being on-premises to deal with or correct issues.
One source said that Facebook is simply unprepared to deal with emergencies and ‘is very weak on the technical side’ Another added Facebook is currently ‘a shambles’ and has been beset with tech problems ‘for months’.
They added: ‘They think they can do everything with AI – but their tech isn’t up to scratch. I’m inclined to think it’s because they’re WFH.’
Monday’s outage was partly to blame for a nose-dive in Facebook’s share price that saw $47billion wiped from its value in its second-worst day ever on the stock market, also driven by a whistleblower testifying about the harms the site does to teenagers in Congress this week.
Mark Zuckerberg – who lost around $7billion amidst the carnage – has previously vowed to make work from home a permanent part of Facebook, telling staff back in June that ‘anyone whose role can be done remotely can request remote work.’
The multi-billionaire said he plans to spend around half his time working remotely in 2022, and predicted that half of his staff could be permanently off-site by 2030.
Facebook’s office are currently open but only to 25 per cent capacity, after plans to open fully by October were pushed back to at least January 2022 amid the spread of the Delta Covid variant.
Of the staff who are not currently in the office, it is not clear how many will become permanent remote workers.
But a Facebook executive previously told the Wall Street Journal that the company has approved 90 per cent of WFH requests. The only caveat is that salaries may be cut to reflect the locations where people are actually working, as opposed to where the office is based.
Data centre staff are among those who cannot request a permanent WFH.
Facebook’s problems began around midday Eastern Time (5pm GMT) on Monday, shortly after its servers were updated, and lasted until around 5.45pm (10.45pm GMT) when the servers came back online. It took several more hours for all users to be able to access Facebook’s sites and apps.
Following Monday’s outage, Zuckerberg issued a personal apology to Facebook users – telling them ‘sorry for the disruption’ while adding: ‘I know how much you rely on our services.’
But his message was immediately attacked from all sides, with those who use Facebook business saying he failed to take the issue seriously while casual users accused him of ‘making yourself more important than you are’.
Still others said they had enjoyed the outage, and were planning to spend more time off social media in the future. ‘Life was way simpler without these services,’ wrote one.
John Graham-Cumming, the chief technology officer of web security firm Cloudflare, said Facebook made a series of updates to its border gateway protocol (BGP) which caused it to ‘disappear’ from the internet.
The BGP allows for the exchange of routing information on the internet and takes people to the websites they want to access.
Dane Knecht, senior vice president of the firm, said earlier the Facebook Border Gateway Protocol (BGP) routes had been ‘withdrawn from the internet.’
Cybersecurity expert, Kevin Beaumont, wrote on Twitter: ‘This one looks like a pretty epic configuration error, Facebook don’t exist on the internet right now. Even their authoritative name server ranges have been BGP withdrawn.’
WhatsApp, Instagram and Facebook Messenger, run on a shared back-end infrastructure, creating a ‘single point of failure’ according to experts.
It wasn’t just the main Facebook apps going down, other services, including Facebook Workplace and the Oculus website were also down.
The EU’s competition commissioner said it shows why large tech firms should be broken up to avoid a similar failure of multiple platforms at once.
EU competition commissioner Margrethe Vestager said the incident highlighted the negative impact of big tech firms controlling large swathes of the online world.
‘We need alternatives and choices in the tech market, and must not rely on a few big players, whoever they are,’ she wrote on Twitter.
The dominance of a handful of large social media and internet companies has come under scrutiny from competition watchdogs on several issues, with many campaigners in the UK, Europe and the US urging governments and regulators to take steps to break up larger firms to prevent monopolies being created.
IT experts have also called on the tech industry to come up with better systems to prevent a single error from having such a wide impact.
Ms Vestager, who is also the European Commission’s executive vice-president for a Europe fit for the digital age, added that the incident showed it was also sometimes good to step away from social media and talk to people ‘offline’.
Facebook’s Chief Technology Officer, Mike Schroepfer, offered his ‘sincere apologies for the outage on Monday afternoon. The scandal-hit company’s shares had dipped by 5 per cent on Monday amid the outage and after a whistleblower went public on Sunday night with claims that the firm prioritises growth over safety.
