The impact of the recent Salesforce outage, which lasted 15 hours and 8 minutes, was widespread and severe – thousands of organizations were impacted and #salesforcedown was a top Twitter trend. Given that Salesforce is the world’s most popular CRM provider, the aftershocks are yet to manifest themselves. All the more reason to do an insightful postmortem. What are the takeaways from Salesforce Outage? How can we better secure our organizations?
Salesforce Outage Heat Map courtesy of downdetector.com
Examining the Outage
Last week, a faulty database script deployed by Salesforce “inadvertently gave users broader data access than intended.” “Broader access” is an understatement; it gave users, both custom and standard profiles, full permission – read and write access – to restricted data, thereby giving full access to sensitive company and customer data to any errant employee to steal or tamper with.
The script affected customers of Salesforce Pardot a business-to-business (B2B) marketing-focused CRM. Salesforce responded by blocking access to all current and former Pardot marketing automation software clients – impacting a huge swathe of organizations globally. That meant no access to your organization’s data for more than a day — Salesforce was truly down.
We have had to disable access to our service to customers affected in order to help resolve the issue. We expect to be able to restore access soon as we continue to work through this issue.
— Parker Harris (@parkerharris) May 17, 2019
Points from the Postmortem
1. You are Responsible for your Data
This was not a one-off case. Salesforce had a debilitating outage in 2016 that wiped out hours of data. And it doesn’t apply only to Salesforce – all SaaS providers are susceptible. G Suite experienced an outage this year, not its first. Microsoft too had a recent shutdown, not its first either. Salesforce, G Suite and O 365 are best-in-class SaaS providers and have stringent security measures and audits, however, mistakes, outages, and disruptions can happen to even the most secure applications. The bottom-line is that organizations are ultimately responsible for their own data’s security and ensuring business continuity.
2. Fine-tune User Privileges
The outage highlights the importance of not over-provisioning privileges. Restrict privileges to a small number of employees, on a strictly need-to-have basis. Regular audits and reviews of user access controls can prevent or at least restrict the damage caused due to errors.
As Balaji Parimi, CEO at CloudKnox said, “Enterprises need to understand that their biggest security risk is not from the attackers targeting them or even malicious insiders – it’s identities with over-provisioned privileges.”
3. Backup is your Safety Net
Even though Salesforce deployed automated provisioning to restore permissions where possible, unless Salesforce data was reliably backed up, admins had to manually update the profile and permission settings. An onerous task at best – particularly after having spent the past two days responding to employees who can’t log in to their Salesforce instances.
Moreover, even if you were fortunate enough to not have your data rifled through after the permissions breach, zero data access for almost two days has a pervasive impact on business continuity and customer experience. However, if you had a reliable SaaS backup and restore solution, you could restore the latest version to production and business could continue as usual. In fact, Salesforce’s responses was: “Organizations with a valid backup of their profiles and user permission data can deploy that information directly from a Sandbox and copy to the production environment.”
Backup and recovery is a necessary business-saver. As Salesforce says: “Yes, we recommend that you use a partner backup solution that can be found on the Appexchange.”