External security assessments are one of my favorite parts of working at Atredis. I love the entire process, from sifting through mountains of data to identify the customer’s scope to digging deep into commercial products that we find deployed on the perimeter, it is challenging work and a lot of fun.
A recent service of interest was an externally-exposed IBM BigFix Relay Server. This service provides a HTTP-over-TLS endpoint on TCP port 52311 that enables system administrators to deploy patches to devices outside their firewall, without forcing the use of a VPN. This is great when an update needs to be deployed that involves the VPN itself, but can be problematic from a security perspective.
After identifying an external BigFix Relay Server, Chris Bellows, Ryan Hanson, and I started to dig into the communications protocol between the relay and the client-side agent. We found that unauthenticated agents could enumerate and download almost all deployed packages, updates, and scripts hosted in the BigFix environment. In addition to data access, we also found a number of ways to gather information about the remote environment through the relay service.
The TL;DR of our advisory is that if BigFix is used with an external relay, Relay Authentication should be enabled. Not doing so exposes a ridiculous amount of information to unauthenticated external attackers, sometimes leading to a full remote compromise. Also note than an attacker who has access to the internal network or to an externally connected system with an authenticated agent can still access the BigFix data, even with Relay Authentication enabled. The best path to preventing a compromise through BigFix is to not include any sensitive content in uploaded packages. IBM also addresses this issue on the PSIRT blog.
BigFix uses something called a “masthead” to publish information about a given BigFix installation. The masthead is available on both normal and relay versions of BigFix at the URL https://[relay]:52311/masthead/masthead.axfm.
The masthead includes information such as the server IP, server name, port numbers, digital signatures, and license information, including the email address of the operator who licensed the product. This information can be immediately useful on its own, but its just the tip of the iceberg.
BigFix uses a concept called Sites to organize assets. A full index of configured Sites can be obtained through the URL https://[relay]:52311/cgi-bin/bfenterprise/clientregister.exe?RequestType=FetchCommands. This site listing provides deep visibility in the organization’s internal structure.
Going further, an attacker can obtain a list of package names and versions by requesting the URL https://[relay]:52311/cgi-bin/bfenterprise/BESMirrorRequest.exe. This tells an attacker exactly what versions of what software are installed across the organization. The package list is split into specific Actions, which each have the following format:
Action: 21421
url 1: http://[BigFixServer.Corporate.Example]:52311/Desktop/CreateLocalAdmin.ps1
url 2: http://[BigFixServer.Corporate.Example]:52311/Desktop/SetBIOSPassword.ps1
In order to download package contents from a relay, the package must first be refreshed in the mirror cache. This can be accomplished by requesting URL ID "0" of the Action ID in the URL https://[relay]/bfmirror/downloads/[action]/0
Once the data has been cached, individual sub-URLs may be downloaded by ID https://[relay]/bfmirror/downloads/[action]/1
Automating the process above is straightforward and allows an attacker to obtain copies of the published packages. As hinted above, sometimes these packages include sensitive data, and sometimes this data can be used to directly compromise the organization.
In order to determine how common this issue was, we conducted an internet-wide survey of the IPv4 space, looking for the BigFix masthead file on externally exposed relay servers. Of the ~3.7 billion addressable IPv4 addresses, we found almost 1,500 BigFix Relay servers with Relay Authentication disabled. This list included numerous government organizations, large multinational corporations, health care providers, universities, insurers, major retailers, and financial service providers, along with a healthy number of technology firms. For each identified relay, we queried the masthead and obtained a package list, but did not download any package data.
Shortly after conducting the survey, we reached out to the BigFix product team to start the vulnerability coordination process. The BigFix team has been great to work with; quick to respond and interested in the best outcome for their customers. Over the last three months, the BigFix team has improved their documentation and notified affected customers. As of March 18th, that process has been completed.
In total, our survey found 1,458 exposed BigFix Relay Servers, with versions 9.5.10.79, 9.5.9.62, and 9.5.8.38 being the most common. Looking at just “uploaded” packages (custom things uploaded into BigFix by operators), we identified over 25,000 unique files.
Quite a few of these uploaded files appear to contain sensitive data based on the filename.
Encryption and authentication keys
bitlockerADkey.ps1
SSH_KEYtar.tmp
AES.key
_BC4Key.txt
Scripts to set the administrator password
secChangeadminpsw.bat
localadmin_pw.bat
AddWorkstationAdmins.bat
AdminPassword.exe
change_admin_password.exe.tmp
SetConfigPasswordRemote.vbs
In summary, anyone using BigFix with external Relay Servers should enable Relay Authentication as soon as possible. All BigFix users should review their deployed packages and verify that no sensitive information is exposed, including encryption keys and scripts that set hardcoded passwords. Finally, for folks conducting security assessments, keep an eye out for port 52311 on both internet-facing and internal networks.
-HD