Ever since I took the Certified Wireless Security Professional (CWSP) class from Robert Bartz, I have been fascinated with Information Security (InfoSec). For me, someone who hadn’t really paid much attention to the InfoSec world before (other than to complain about the firewall) this was a huge turning point in my career.
Now, to be fair, I have had some encounters with Malware before but it was more around being on the receiving end of things, not the person on the sideline, watching on. Additionally, I have seen it one other time in the past, but it was a while ago, and my only concern was to tell someone else and move on with my life.
All of that changed about a week ago.
So it turns out, malware doesn’t announce itself when it first shows up on devices or networks, you need to have other devices already in place that can monitor the traffic and pick up key traffic that matches patterns, and then alert someone about it. Think of it like a spectrum analyzer that can detect and identify a wireless video camera or BlueTooth devices but in packets going out over the wire.
Luckily, such devices exists and can alert when something is amiss.
Now, as a side note, I need to point something out here. These tools alert on something because it matched something it was told to look for. Just because there is an alert, it doesn’t mean that there is a problem, just that it detected something. Conversely, no alert doesn’t mean there isn’t a problem, just that the traffic doesn’t match what it is looking for. The difficulty is that malicious players in the world can generally read posts like these and work to hide their tracks, expressly to prevent the tools from finding them. It is the ultimate cat and mouse game, and half the people playing this game don’t have to follow any rules.
Now that we have that out of the way, let’s move on.
So about a week ago, there is something that triggers an alert. It turns out the source IP address of this suspect traffic was a NAT address from a wireless network, which is why I was contacted. One more example of how wireless folks get involved in things that aren’t necessarily wireless, but are still expected to play a part in things. Luckily, I like to use pfSense firewalls for scenarios like this, which gave me a great place to start my troubleshooting.
The first step that I took in this troubleshooting scenario was to start with “analysis” (thanks ECSE Troubleshooting) and part of the diagnostic tools in pfSense is a packet capture feature that allows me to pick an interface to capture and some basic pre-capture setups. I chose to look at the “LAN” side and not to filter anything.
Pro Tip – If you don’t know what you are looking for, don’t ever use a pre-capture filter.
The only downside I have with the pfSense capture feature is you can’t view the traffic live; you start it, wait for some random time, then stop it. It’s only after stopping the capture that you can view it. It also gives you an option to download the capture and view it in Wireshark. This is the option that you want, it’s so much easier to navigate and use display filters as you start your troubleshooting. At this point, keep in mind this isn’t supposed to be a write up on how to troubleshoot any issue, just how I worked and discovered this particular issue.
After my first capture, this is what I saw:
Now, working in Wireshark can be intimidating, but don’t let that scare you off. Save your original capture, and start to get to work. I’ll try to throw in some tricks I used to help you out, it certainly helped me out! (Thanks ECSE Troubleshooting and Eddie Forero!)
When you first look at this capture, it might not appear to have any red flags, but what I have learned is that there are a couple of things that Wireshark does to help you out, and colors are one of those things. In looking at the main screen, I noticed there was a lot of blue colors, that blue color being related to DNS. The other cool thing is Wireshark will show you a bigger picture of colors on the right hand side. See that narrow, vertical part on the right? That shows a bigger section of the capture (if it’s large it won’t show the whole capture, just a bigger section) but do you notice how much of that narrow section is blue? That is a ton of DNS queries!
Quickly I realized that the source of all of these DNS inquiries were all from the same device. For any device, that just isn’t normal. After reading through it a little bit more, I found a line that was a little concerning. It’s on line 70 and 71 of the capture above, but seen below:
While the majority of the other DNS inquiries are for a website that made sense to me (knowing the function of this device), the fact that it was looking for a website like “susedundjust.ru” was a concern. For those of you not in the know, a domain of “ru” is for Russia. More troubling, it resolved to an IP address and shortly after, starting to talk with this server. I’m not here to disparage any nationality, but that’s just odd. However, now that I have a direction, I did a second capture. Focusing just on the DNS entries, this is what I saw:
Now, I’m not an information security professional, I’m more of the interested bystander, but that result just isn’t right! Look at the time! All of that in 9 seconds! And check out the names it’s trying to resolve! I don’t care what the name ends in, there is a problem!
OK, now that I have a possible problem child, I used a Wireshark display filter of “ip.addr == 172.21.2.212” to look at just that one device. It took my capture from a whopping 929 packets to just 399 packets.
Pro Tip – Display filters don’t delete the other packets, just hides it from the display. Don’t be afraid of trying out other display filters, just delete them if they don’t work, hit enter, and try another one.
Now that’s done, I can focus, but on what? A trick that I picked up in my ECSE Troubleshooting class is this one:
View -> Name Resolution -> Resolve Network Addresses
What used to be a “random” collection of IP addresses now appear some names, but not all the names.
Bummer, but it’s just one step in the process.
The best part is when it resolves a name that you didn’t capture during a DNS lookup (more on that later.) What you see after that is something that looks like this:
Again, just an InfoSec wannabe over here, but do you really want any device you are responsible to be sending anything to a website named “anubisnetworks[.]com”? Of course, being an InfoSec wannabe, it turns out this is a red herring. There really is a security firm called Anubis Networks, and they are legit.
What is going on here? Honestly, even a week after this started, I’m still trying to piece this puzzle together. Line 301 contains this nugget:
Remember those 2 lines from above, Lines 70 and 71? This device did a DNS lookup for that server, and then found an IP address for it. Why would Anubis want that? More importantly, how does Anubis fit into any of this?
Just what is that website? Why is this device that’s doing crazy DNS lookups sending data to Anubis Networks about some website named “susedundjust[.]ru”? I still am not sure. However, further down that capture, with the names resolved, led me to this:
This isn’t just a simple push like above, this is that same device establishing an encrypted tunnel with a website named “server[.]fomikuron[.]com”. This is REALLY odd. Doing some internet research, I ran across this entry on a website named malwareurl.com:
I don’t know what a “Poseidon botnet controller” is but absolutely nothing sounds good about that, and the entry is less than a month old. Back to the Google, and this is what I found out about PoSeidon (malware) from the Wikipedia:
“The malware attempts to steal both keystrokes and credit card numbers stored in system memory, by scanning RAM for Discover, Visa, MasterCard and AMEX issued credit cards. The credit card data is then encrypted and sent (exfiltrated) to a number of predefined Russian servers.
If the commercial remote administration software LogMeIn is installed, the LogMeIn settings are modified, forcing the next remote user to enter a username and password. This allows the username and password to be read into the keylogger and exfiltrated.”
Remember all the Russian DNS requests? This same device talking to a botnet controller? I learned about RPA’s during #TFD19 from Automation Anywhere, and I learned that RPA’s are also called “bots”. Did I really just discover a bot doing malicious work? Even now it leaves a heavy feeling in my stomach. This isn’t good.
Back to Wireshark we go.
Now, when I did the name resolution, remember how I mentioned that it doesn’t resolve all the IP’s that you see, just some of them? Time for some manual detective work. Looking back at the original capture picture, I started with the TLSv1.2 stuff and started plugging those IP Addresses into a WhoIs IP address look up tool. One of the first IP’s I ran across, line 54 above, was 126.96.36.199. That guy came up as “fastandtstrongwolf[.]com” and is reported as malicious.
The next IP I looked up was 188.8.131.52. Want to take a wild guess who owns that IP? LogMeIn.
Now that I have these 2 pieces of information, how can I correlate that with what my capture has? IP’s are good, and techie people are sometimes good at remembering these, but in a situation where there are non techie people, names are important. Since the tool didn’t auto-populate the name in the step above, there is a manual way to do this. Follow along at home if you like, but here it is.
- Find a packet in the packet list field that contains the IP address in question.
- Right click in that packet, but on the IP address that you have looked up.
- In the pop-up window, select “Edit Resolved Name”.
- Below the display filter line at the top, a new line will appear named “Name Resolution Preferences.”
- On the right side, there will be the IP address you selected. If it’s the wrong one, click the down arrow and switch to the IP you wanted.
- In the blank “Name” field, type your new name and hit enter.
WARNING – Once you edit this field, you can’t change it as of this writing! If you name that IP address something you would prefer a customer not see, you can’t change it back! Once a name is assigned you can’t change it so name the device accordingly!
Now, after that exercise, you will see your new name populate across the capture everywhere that IP address shows up. Quick filter with the syntax of:
ip.addr == 172.21.2.212 && !dns
IP address of 172.21.2.212 is the local, unhealthy device and since I knew DNS was a galactic mess from earlier, I filtered that out with the “!”, I came up with a screen that looks like this: For the old people with eyes that aren’t as sharp as they used to be, I’ll type out what is in the source and destination field:
- Our poor infected device (with a hidden name of “DC-“
- the Default Gateway answering a DHCP request
Pro Tip – using brackets at the end of a malicious website is a good way to type what you mean, without your device suddenly trying to go to said “baddie server.” Picked that up from researching the sites dedicated to this stuff, and by accidentally trying to go to a baddie website during my research!
So there it is. That is some bad stuff. 30 seconds of traffic and not one packet is related to what this device is supposed to do for a living. Now, I don’t own the infected device, it belongs to someone else, so I can’t get it and see if I can find the malware on the device itself, but that is more than enough smoke for me to say there is a fire.
Cisco’s “Talos” computer research laboratory discovered and named the PoSeidon malware back in 2015, and it looks like I hit EVERY element of it in just a couple of captures. My day is now going differently than I expected.
(Disclaimer – even if I had the device I’m not sure I could find the malware, but I bet someone else could.)
Long story short, I contacted them, showed them my evidence, and suggested they do something about it. Turns out it doesn’t take much more than what you see above to freak out a normal person, and they took what I deem to be the appropriate action – they unplugged every cable on the device, to include the power cable. I hoped they would have smashed it with a hammer as well, but it wasn’t an actual or perceived spider, so that might have been overkill! (Love you Amy!)
That the end of it, right?
Unfortunately, no. If you have made it this far – get up, stretch your legs, refill your drink, walk the dog, whatever, because it only gets weirder from here.
Now that I know which device is/was the problem, I decided to check for anything I might be able to see. Now, I know from earlier that I can exclude something from the display by entering a “!” (aka: an exclamation point or “bang”) before something in the filter element to exclude it from the display, so that’s what I do.
!ip.addr == 172.21.2.212
Now, with all of my other naming stuff and other cool tricks still in place, I keep scrolling down, and I run into this:New IP’s in both columns that I haven’t resolved, but the local IP is one number off from my original infected device. The pattern, though, with the colors and how it looks, is vaguely familiar, so it triggers something in my brain. Doing some searches, I resolve 184.108.40.206 to “fans.fmhschool[.]com” and 220.127.116.11? 8 days after the fact and I am still trying to figure out what that IP address goes to. Quick Google search and fans.fmhschool turns out to be a technical college in India.
This is getting more intense than I ever realized, I need a break.
Back At It
With renewed vigor after a break, and some adult beverages, I went back to packet captures. At this point, there was a realization that I was doing a TON of packet captures, and I reached out to a friend of mine who works in InfoSec, inquiring if this is normal. Their response was yes, sometimes, you will spend A LOT of time in Wireshark, so get comfortable with it, put your head down, and get to work.
So that’s what I did. I did a bunch of captures, and I made some mistakes. I want to share that with you, if you don’t mind. Think of these as “Jim’s pcap lessons for the amateur InfoSec sleuth.”
- Have a plan. Capturing is good and all, but have a plan of what you are capturing.
- Document the plan. Having a plan is good and all, but write it down! After a while, you forget what you are trying to find and/or do.
- Save your captures. A couple of times I violated my rule and didn’t save the raw capture immediately, and I lost that data when I either closed the file or saved it with modifications.
- Name your files. With so many captures, trying to remember which file went with what thought process can be difficult. This ties back into #2 about documentation.
- Assume that you are going to come up with a bunch of files, you might as well create a folder to save everything to at the beginning. Your desktop is great until 3 days into an “engagement” you can’t find anything on your desktop.
- Keep notes. Along with the raw captures, modified captures, your plan from #2, keep good notes. It helps to remind you what you found, when you found it, and what your next steps should be.
Key thing to keep in mind with this type of work is that (I’ve heard) that this type of stuff can be used in both civil and legal proceedings. If called to discuss what happened and when, good documentation is very key. Of course, having good documentation isn’t a new thing to anyone in IT in general, but this is a different realm than I am used to.
Having typed all of this, I have to think this is also a good approach for doing any capture, wired or wireless. Maybe I change the name of the rules to any and all pcap adventure. Anyways.
Multiple captures later, and using the tricks that I have covered earlier, I was able to get a little closer to what I needed to find, but I was still having some problems. While this second device had problems, it wasn’t nearly as chatty as the first, so I had to use some additional Wireshark tricks to clean up the display in Wireshark. The trick I came across was letting Wireshark do the filtering syntax for me.
If you have ever tried to do filters in Wireshark, you know that it can be tricky. Don’t type exactly what it’s looking for and it either filters nothing or everything. I found a trick that I employed a to clear out the legitimate traffic and only show me the malicious stuff. That trick is right clicking on ANYTHING will get the following the pop up menu.
When you find something in the top packet list screen you know is legit and you want to eliminate it, click on that packet. In the packet detail screen below that, find that IP address in either the source or destination field in the Internet Protocol Version 4 section, and right click on it. In the section with filters, pick what you want. If you already have an IP address you are looking for, but want to exclude an address, simply clicking on “…and not Selected” will populate the display filter with all the crazy syntax you need! Now, you might have to change it from “ip.src” to “ip.addr” but everything else is done for you! Good news it messing up the display filter isn’t the end of the world, just remove the filter entirely, hit enter, and then start over.
I don’t want to go through every filter that I applied, and the research into all these websites, but suffice it to say that I could say, with a 99% certainty, that I had found a second device that was doing something it shouldn’t.
The websites were there, fastandtstrongwolf[.]com, hesbetloran[.]ru, the still unsure IP address of 195.161.41[.]19, LogMeIn. What wasn’t there was the DNS stuff that alerted me days ago. In fact, that seemed pretty stable.
*Side note at this point. Log Me In is a legitimate company with a legitimate product, and what I found isn’t a comment into the company or their product. Like other products, their product has been hijacked by malicious entities to do bad things. What I found is a symptom of a bigger problem, not the problem in and of itself.
My New Pet Malware
At this point I am coming into 4 days of constant Wireshark work, watching the malware through packet captures, and feeling like I have a good idea of what is happening. The anecdotal evidence points to some intelligence with this malware, so I decide to do what any “normal” Wi-Fi person would do – I decided to play with my “malware pet” and see what it does.
Now, to be fair, I knew that at any moment the device could be removed from the network and remove the threat, but I wanted to better understand this creature I’ve been working on before I lost the opportunity to see it in action, in the wild if you will.
Now, my most recent packet capture was showing the communication going on, and as I applied display filters, I had noticed that the bottom right hand corner of Wireshark would show the percentage of the traffic that I was looking at, as it applied to all of the traffic. Remember, this can only happen when you don’t use pre-capture filters. If you don’t capture all the traffic, then this can be misleading. What I was finding was displays that look like this:
What I was able to determine that of all the traffic collected, 8.6% of the traffic was heading to one single IP address, 81.25.71[.]88. As a reminder, that IP address resolved to “fastandtstrongwolf[.]com” which is known to be associated with the PoSeidon Malware.
PoSeidon Malware Structure
Now, if you didn’t take the time to read about it earlier, I want to give you a quick explanation of how this particular malware functions. It might be this way for all of them, but I don’t want to overstep. The malware works by using 2 different, external “resources” in order to function. One of them is known as the “Command and Control (C2) Server” and the other is the “Data Exfiltration Server”. Once a device is infected with the malware, it reaches out to the C2 server for updates, but sends any information it gathers to data exfiltration server. Based on what I saw earlier when I said there was some intelligence to the malware, it might be coming from the C2 server.
Websites that it was using earlier had disappeared, and new websites were appearing. URL’s that used to resolve to one public IP were now resolving to a new IP. These malicious players are smart, and it doesn’t surprise me to think this is how they get around futile attempts like I was about to employ.
Poking the Bear
If you remember from the beginning, I mentioned that I had a pfSense firewall in play on this network. The existing firewall rules allowed very little inbound traffic, but traffic initiated from the LAN heading outbound is generally allowed. That being said, I do have the ability to add rules blocking specific IP’s as needed. So that’s what I did!
I started a packet capture, and then went ahead and added 2 rules blocking the fastandtstrongwolf[.]com IP and the weird IP that we haven’t identified yet, 18.104.22.168, because there are 2 servers involved, why not pick on that one? With a quick couple of IP’s being blocked, it was simply a waiting game to see how my pet malware would behave to this poking. I’m inpatient so I waited about 5 minutes, stopped the capture, and then took a look.
All I can say is my pet malware did not disappoint! After dropping in a couple of display filters to remove the legitimate traffic, what I was left with was glorious, and hopefully for you will reward you for sticking with me through all of this.
Step 1 – Act normal
Here we can see our 2nd Device talking to LogMeIn, and then attempting to contact fastandtstrongwolf[.]com, which I blocked. Give the malware credit, it tried 6 times, with an abnormal DHCP request thrown in for good measure. Based on Step 2, I’m pretty sure it is supposed to try for 30 seconds before it kicks that server to the curb, and moves on with it’s mission. No honor amongst thieves!
Step 2 – PANIC!
Now, I’m not going to actually type out the names of the servers it’s looking for at this point, but it is a persistent bugger. Keep in mind the duration of this image is only 6 seconds.
Step 3 – Continue the PANIC!
What we see here is the rest of the DNS queries that didn’t fit well in the first image. All told, these 2 images represent 13 URL’s over the course of 10 seconds. Can we all agree this isn’t a person on a keyboard?
Step 4 – Making Progress
Turns out Server # 14 is still alive and online, and he lives at 208.100.26[.]251 and we are now introduced to “feednotusdry[.]ru”. Pretty sure that is the exact opposite of what we hope it does to us, but sometimes the baddies just run out of cute names to use, I’m guessing. What we can also see is it doesn’t take long to re-establish contact and try to resume it’s normal programming. For those keeping track, it took just about 53 seconds to notice something was wrong, ensure that there really was a problem, go on a search for a redundant server, and then re-establish contact and keep doing it’s thing. I know purpose built failover that take longer than that! Props to the malware guys, they aren’t stupid!
Step 5 – But Wait!
Turns out just one server isn’t good enough for my pet, it wants two. Remember that whole C2 and Data Exfiltration Server? My pet needs more! Got to hand it to them, redundancy wasn’t missed in this guy at all!
Step 6 – Lets Go For Two!
So 66 seconds after the first failed attempt to communicate with “fastandtstrongwolf[.]com”, my pet malware has found its second server of “soletrobuse[.]ru” at 72.44.80[.]19.
Step 7 – Put A Bow On It
By the time we get to 290 seconds after the start of the capture, and 265 seconds after the first failed contact with one of it’s servers, my pet malware is back online, sending encrypted data back and forth to 2 servers we probably don’t want it to talk to, all the while re-establishing its LogMeIn stuff.
It both sickens and impresses me, all at the same time. Damn.
End Of My Journey
I think until you have had the chance to go through an experience like this, it’s hard to understand how a person feels about what they are seeing. I routinely went back and forth between amazement of watching the malware in action, seeing how it would behave as the global internet environment changed around it (sites being shut down, new sites created), constantly changing, to almost wanting to throw up as I watched data being shipped to places I knew I didn’t want it to go. How can one be both amazed and sickened at the same time? I don’t know how the professionals do this day in and day out. I tip my cap to them.
Shortly after this capture, the device was powered off and all the cables disconnected, sitting there like a caged animal waiting to be let loose again. I don’t know what they did with either of the boxes actually. One can hope they met their maker with a hammer and then shredder, or maybe some other less than lethal solution that renders it safe again, but hopefully clean.
But therein lies the rub. I can hope it’s clean, but if you skimp on your IT team and they delete one file and call it good, it could be released into the wild again, maybe on another unsuspecting user. Why is the best I can do is “hope” that the right thing happens?
During this adventure, a friend called me an “accidental Blue Teamer” and it made me think. Blue Teamer’s are the security folks that try to protect networks and devices against the “Red Team” or the malicious actors trying to harm us for real. But in reality, we are all “Blue Teamer’s” by simply being in Information Technology (IT). I don’t know about you, but I’m not here by accident. That means I’m not an “accidental” member, I’m just a Blue Teamer by definition.
Security, now days, is in layers. We use micro segmentation to try and contain the breaches and issues, but at the end of the day it is up to all IT professionals to be aware of what is going on. Merely hiding our head in the sand and hoping its not us won’t cut it anymore. Learn how systems operate normally so when you see something that doesn’t make any sense, you can use your awesome abilities for good, not for evil.
I want to thank you, the reader, for making it this far! I also want to thank Eddie Forero for all of his instructions on using Wireshark, which would have made this a really boring adventure. I also want to thank all the InfoSec people out there who work on this stuff day in and day out and share their knowledge, online, giving me a place to go and look stuff up.
Lastly, nothing in this should be construed as professional InfoSec advice. If you think you have a problem, find a competent InfoSec professional and get help!
I’m not an InfoSec Professional, I just play one on Twitter!