A Brute Forced New Year

At 11 a.m. on New Year’s Day, I received an email from Uptime Robot that our primary webserver is down. It’s a small server. Just an M2 Mac Mini at a colocation facility. It is an affordable setup that works well for us. We actually have two machines, one in a Southwest data center and another in a Midwest data center.

I connect to it via Apple Remote Desktop to see what’s happening. The CPU usage is high. Too high. Something is causing a lot of work. I dig deeper and find the ARDAgent—the Apple Remote Desktop service I’ve used to log into the computer—running amock.

Little Snitch shows heavy traffic from East Asia on the ARDAgent port 3283. Somebody has a bot farm trying various username/password combinations—millions of connections in just a few hours—trying to break into our server. Fortunately, we don’t use obvious usernames like “admin” or “superuser” and we use LONG passwords. It’s unlikely the bots will succeed, but it was interfering with the web server.

Our web traffic then switches to our secondary server. It turns out the second machine is also under attack. Wow! It’s not even in the same data center. Why are they attacking us? Is it personal? That seems unlikely. Could they be going after systems in the same hosting facilities? Who knows? Who cares? What we need is a strategy to kill the attack.

I’ve always had a love/hate relationship with Apple Remote Desktop. ARD provides basic information and administration of remote computers. If I log into a computer using ARD, it hands off my connection to the virtual network computer (VNC) to remotely display the graphical user interface (GUI). I technically don’t need ARD. I can connect using VNC directly, but the traffic is unencrypted. ARD encrypts the traffic, making it much safer to use on the open Internet.

Unfortunately, you cannot change the ARD port. It’s hard-wired to port 3283. I can change the VNC port and the SSH port. But not ARD’s port.

I know we shouldn’t rely on security through obscurity, but it certainly helps. I used to manage a full class-C range of IP addresses through an internet service. I would watch bots scan port 22 across all IP addresses—looking for SSH servers. Another scan would search for standard Windows sharing ports. Yet, another would scan for SMTP servers. Today, these scans include port 3283 in their search.

Bots scan across IP ranges on the same port, searching for common exploits. If you have a poorly configured SSH service, they can find the server through port 22 and try breaking in. If it works, they own your computer. If it doesn’t, they move on to the next machine.

These scans are the most common. Every computer exposed directly to the internet will be scanned multiple times a day.

The second type of scan is a comprehensive port scan, where a bot tries a range of ports on an IP address—sometimes all 65,536 ports. These scans are noisy and time-consuming.

Some bot farms run both kinds of scans. First, a scan for a single port across various IP addresses. It marks all the found servers and whether they could break in. With this list of known servers, it could scan each port to find hidden services.

But these ranged port scans on a single address are less common.

Most of the time, it’s a drive-by scan of standard ports like 22 (ssh), 80 (web), 443 (https), 5900 (vnc), 3283 (ard), and more.

By changing the default port for a service, you avoid the standard scans. Some people think that it’s dumb to move ports, but I’ve watched how it thwarts so many scans. At the very least, it forces the attack to spend way more resources trying to determine the active ports and what services are available at these alternative ports.

I’ve known for a while that ARDAgent is being scanned and probed—often. It’s not uncommon for ARDAgent to balloon memory usage into gigabytes. I already have a scheduled restart of the service to severe connections, but it’s not something I can just shut off.

Or can I?

My experience is that macOS does NOT like ARD being turned on and off via the command line. I’ve tried in the past, but it never seemed to work. It may be time for another look. I have a home server to test this on.

These are the commands to start and stop ARD.

# Start ARD
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -allowAccessFor -allUsers -privs -all -clientopts -setmenuextra -menuextra yes

# Stop ARD
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate -configure -access -off

With ARD running on my home server, I SSH in and run the Stop ARD command. It stopped. That’s funny, it never worked for me before. I’m not sure why it’s working now. I tried the Start ARD command, and it works, too. What’s going on? It was a problem in the past.

Reading through the various support documents leads me to believe that ARD can only be started and stopped via the command line IF it was first activated manually at the machine through Settings > General > Sharing > Remote Management.

That’s the ticket. You can start it first on the local machine and then control it remotely.

Think of it as a door that cannot be opened from the outside because it is barred shut on the inside. You must first go inside and remove the bar, and then you can lock/unlock it from the outside using the right key.

I tested it on the secondary web server, and all the bot traffic disappeared. Of course, I can’t see it via the GUI. I use htop via ssh to graph the CPU usage. I turn ARD back on. Instantly, the attack resumes. Hmm. I turn it off. I test managing ARD on the primary server as well. Stop. Start. Stop. It works. Everything is quiet, and my web servers are back to their ordinarily fast selves.

We left it alone for 24 hours and turned on ARD again. The attack is over. The bots realized the jig was up and moved on. I turned ARD back off.

This is the optimal solution I was looking for all along. I don’t like running ARD unless I need to remote through the GUI into the machine. Most of the time, SSH is sufficient.

What about brute force attacks on SSH? Isn’t it susceptible to the same problem? Nope. Not if it’s configured correctly. The trick is to set up access via passkeys—no passwords allowed. You can’t brute force SSH if it doesn’t allow password authentication.

# sudo vi /etc/ssh/sshd_config.d/passphrase_only
# Disable all password authentication
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

To check that it works, use the verbose flag when you ssh back into the server.

ssh -v [email protected]

There should be a line that says:

debug1: Authentications that can continue: publickey

Just publickey by itself. No passwords are allowed.

Then, for extra security—through obscurity—change the SSH port by sudo editing the /etc/services file. Find the SSH port 22 lines (tcp and udp) and change to your preferred port number. I won’t make any suggestions here. Just find one you like and use it.

Then, set up your personal ~/.ssh/config file to use the changed port.

Host example
HostName example.com
User USERNAME
Port NEW_PORT_NUMBER

Then it’s as easy as ssh example to log into the example server. When configured this way, I’ve never seen a bot attack on my SSH services.

As for the New Year’s attack, after it abated, I changed the passwords, then reviewed all running processes for odd behavior. Everything seems as smooth as ever. Actually, it’s better than before because nothing can take swipes at ARD anymore.

All of this work sounded pretty easy, but it wasn’t. It took time to track each issue and identify how the attack worked. It wasn’t a fun day, but we learned a lot, and now everything is more secure.

Happy New Year!