Apache had to scramble in early December 2021 to be ready to release patches for Log4Shell when it made the situation public on December 9 last year. As a result, researchers quickly found hard cases and workarounds for patches, and Apache was forced to release multiple iterations, which added to the confusion.
“This stuff is everywhere, really everywhere,” said Jonathan Leitschuh, an open source security researcher. “Attackers jumped on it, the security community jumped on it, payloads flew everywhere.”
Still, the researchers say that Apache’s overall response has been solid. Nalley adds that Apache has made changes and improvements in response to the Log4Shell story and hired dedicated staff to extend the security support it can provide to open source projects to catch bugs. before they send the code and respond to incidents as needed.
“In a short amount of time, two weeks, we got it fixed, which is great,” Nalley said. “In a way, this is not a new situation for us and I would say we have handled it perfectly. But the fact is, even at the Apache Software Foundation, this highlights our responsibility to the people who use our software.”
Going forward, the more worrisome aspect of the situation is that, even a year later, about a quarter or more of Log4j downloads from the Apache Maven Central and other repositories are still full. Log4j instances are vulnerable. In other words, software developers are still actively maintaining systems running vulnerable versions of add-ons or even building new vulnerable software.
“The reality is most of the time when people choose an open source software component,” said Brian Fox, co-founder and chief technology officer of software supply chain company Sonatype, which operates Maven. vulnerable, a fix is available. Central and is also a third-party Apache repository provider. “I’ve been working for a long time and I’m tired, but it’s really shocking. And the only explanation is that people really don’t understand what’s inside their software.”
Fox says that after the initial scramble to resolve Log4Shell, version downloads in Maven Central and other repositories reached a peak where about 60% of downloads were patched versions. and 40% are still vulnerable versions. Over the past three months or so, Fox and Apache’s Nalley say they’ve seen the numbers drop to about 75/25% for the first time. However, as Fox put it, “After a year, a quarter of the downloads are still pretty bad.”
“Some feel Log4j is a huge wake-up call for the industry, a collective awakening and bewilderment,” he said. “And it helped us really expand the message about software supply chain security, because people no longer deny it. What we’re all talking about now is real’ we’re all living with it. But Log4j’s peer pressure alone forced everyone to upgrade, so if we can’t upgrade this one to 100%, what about all the others?”
For security researchers, the question of how to solve the long tail of security holes is always present. And the problem applies not only to open source software, but to proprietary systems as well. Just think how many years it took to move the last 10% of Windows users out of XP.
“With these worst-case scenarios—black swan events in open source—you know they will continue to happen, because the community has responded so much better, but the pace at which open source has evolved is even worse. even faster,” said Lorenc of ChainGuard. “So we have to find a balance between prevention and mitigation, while continuing to work on reducing frequency as much as possible. It looks like The Simpsons meme when Bart says, ‘This is the worst day of my life.’ And Homer said no, ‘Worst day of your life hitherto.’”
