IT administrators and executives who have experienced ransomware attacks have an important story to tell — their lessons learned from the battle can also turn into helpful advice for others.
Ransomware protection and recovery was top of mind at VeeamON in Miami and online Tuesday and Wednesday, as it has been at previous iterations of Veeam’s user conference and other recent tech shows. Customers stressed planning, immutability, security and education.
“Never waste a good crisis,” said Edwin Moraal, chief information security officer (CISO) at VNOG, a safety region in the Netherlands that oversees fire stations and works with other public sector organizations.
Create, practice, act on plans
Months before a ransomware attack on the city of New Orleans, attacks hit across Louisiana in 2019. As a result of the imminent threat, the city rehearsed its backup and recovery strategy, said Kim Walker LaGrue, CIO of the city of New Orleans, which has nearly 5,000 employees and 400,000 residents.
“We had a disaster recovery and continuity plan in place,” LaGrue said during a panel discussion. “We walked through those procedures and knew how we would handle an attack.”
When an attack did hit, LaGrue said the city followed “every part of that protocol,” which included notifying authorities such as the FBI, disconnecting the environment from the internet and shutting down the data center.
“That was the plan and we fought with it,” LaGrue said.
Cristal KawulaPresident, TriCon Elite Consulting
Cristal Kawula, president of TriCon Elite Consulting in Canada, which also sustained a ransomware attack, said she liked hearing that LaGrue and her team practiced their recovery scenario.
“It’s great to have a DR plan. It’s great to have a cyber attack DR plan,” Kawula said. “But if you’ve never actually had everyone run through it, trust me when I say: When you’re under attack is not the first time you want your team seeing that plan. It is not the first time you want everyone figuring out what to do.”
In the city’s case, the attack spread and affected its entire environment, putting 120 sites, 300 to 400 servers and 2 TB of data at risk.
“We made an immediate shutdown, and that was the best decision for us at the time,” LaGrue said. “I would recommend that you put yourselves in a position not to negotiate with ransomware [attackers]. It was a decision we made early on that we would not do that. And the best way to do that is to know what your strategy will be [and have] a good, strong backup solution.”
As part of the overall backup and recovery strategy, Moraal recommended testing restores.
“If you cannot restore something, it’s a useless backup,” Moraal said in a panel discussion.
Immutability can help admins ‘sleep at night’
IT administrators said immutable backups play a key role in ransomware protection and recovery. Immutability protects against alteration and deletion of files.
Kawula said immutability in the cloud with Microsoft Azure has been a helpful tool, especially given issues with supply chain and infrastructure availability over the last few years.
“You don’t have to wait six months, you don’t have to wait six weeks, you don’t have to wait six minutes,” Kawula said. “You can get your backups up and ready now.”
Dave Kawula, principal consultant at TriCon Elite Consulting who works alongside Cristal, said he’s seen delays with customers upward of a year just to procure a server.
“Having that delay for hardware and that procurement delay — it’s not really acceptable, especially if you’re going to get hit,” he said.
LaGrue said her city deployed Veeam after the ransomware attack to help with the recovery process, as its previous backup platform was “pretty complex.” Immutability provided a layer of security.
“Immutable backups were really the thing that made me sleep at night during our 30-day recovery process,” LaGrue said.
Security, security, security
While implementing a comprehensive ransomware protection and recovery strategy is difficult, it’s worth it, IT authorities said at VeeamON.
“Security is not meant to be a matter of convenience,” Dave Kawula said. “Monitoring your infrastructure, trying to protect against those cyberthreats — this is not something that comes easily to us as IT professionals. We almost all have to put on a hat as security professionals now. And although it’s hard, it’s also supposed to be hard for the bad guys to get in.”
Complexities might include deploying privileged access workstations and having to log in to multiple accounts.
“It’s a little more inconvenient, but hopefully securing your infrastructure,” he said.
Thanks in part to ransomware attacks, organizations are escalating backup and disaster recovery to C-level executives such as the CIO and CISO, said Johnny Yu, research manager at IDC.
“It’s important to look at it as more than just an IT problem,” Yu said. “It is a business problem. You do need to get people further on up involved in the process. That doesn’t necessarily mean that they need to be performing the recoveries, but they need a bit more investment than just signing the approval to purchase the data protection product.”
Paul Crocetti is an executive editor at TechTarget Editorial. Since 2015, he has worked on TechTarget’s Storage, Data Backup and Disaster Recovery sites.