Aruba Takes The Lead With WPA3

Let’s just call a spade and spade.  And by that I mean that Aruba Networks played their trump card at Mobility Field Day 3 (MFD3), and it was the Ace of Spades.

For three days, delegates of MFD3 listened to vendors talk about A.I. and machine learning and analytics and location and BLE and a little on 802.11ax.  What wasn’t talked about much is the new WPA3 standard/certification that we have been waiting on since WPA2 was first introduced back in 2004.  WPA2 is now a surly teenager in human years but in the world of technology has now been likened to a pensioner heading into retirement.  14 years is a LONG time to have anything in technology a standard.  In the 802.11 realm, it’s just slightly younger than 802.11g!

Let that sink in for a second.

Then comes along this company called Aruba Networks as the last presenter at MFD3.  They started with the standard quick introduction and then talked about a new product that is intended to be used as a Point-To-Point (PTP) link with dual radios; one at 5 GHz (802.11ac) and the other at 60 GHz (802.11ad) that by itself is cool enough to stand on it’s on.  You can watch that presentation here if you want to learn more about it (which you should, it’s cool) and then a quick hit on their 802.11ax stuff.  Again, pretty cool along with a good slide about dates surrounding 802.11ax.  If it wasn’t for what came next, this would be my focus but the next topic changed everything for me, so their hardware will have to wait for a different post.

The next presenter was a gentleman named Chuck Lukaszewski, and he brought the goods.  Chuck presented on Aruba’s efforts in the realm of security, WPA3, and more importantly, Opportunistic Wireless Encryption (OWE).  OWE was recently changed from a requirement by the Wi-Fi Alliance for WPA3 certification to an optional feature, much to the chagrin of wireless professionals everywhere.  The general consensus was if it’s optional then no vendor is going to put any effort into it because why would they?  Chuck changed all of that with this slide:

Aruba WPA3 Summary

I’m sure that myself and others will talk about these other new terms, “SAE” and “Suite B/CSNA” which are still part of the required certifications, but I want to focus on OWE, the optional part that we all wanted but had given up hope on.  If you watch the presentation, it’s easy to pick up on how excited we all were to not only know that this wasn’t dead, but that Aruba was actually able to demo this in action, live, and in front of a technical audience.  802.11ax might promise crazy QAM rates (1024 QAM to be precise) along with BSS coloring and OFDMA (allowing clients to utilize LESS than a full channel if they don’t need it, LOVE that one by the way) but sometimes the improvements that are needed are not always the sexy and marketing bullet points that C-Series executives want to see on their hit sheets.  1.21 JiggaBytes Per Second (JBPS, I just made that up) is much cooler on a marketing sheet than “hey, we did something that is cool but your will never be able to tell because it is seamless to you” but let me assure you, it’s the one thing that we NEED in the wireless industry.

Everyone knows that you don’t use the Wi-Fi in public places because you are going to get hacked by the guy sitting a couple of chairs away and your life will be in shambles.  Guess what OWE solves?  Exactly!  This feature is not meant to authenticate the user, nor account for what they are doing.  That will always be left to the Enterprise version of the WPA2 and now WPA3 standard, this is meant purely for the guest client/user that you want to allow onto your Wi-Fi and you want to ensure that no one can sniff their traffic while they are onsite.  This doesn’t interrupt captive portals (shudder) since that operates further along the network path so you can still stop users from accessing the internet; no, this is intended as a feature that shores up the bane of the Wi-Fi world – guest Wi-Fi is insecure due to the open nature of the network.Aruba OWE Protocol Flow

I call this my “Mom” feature.  My mom uses technology, but she doesn’t understand much more than other mothers of her generation.  She doesn’t understand why having a 4 Way Handshake (seen above) right after the association packets is a good thing, but I don’t need her to know why or understand.  All she needs to know is that if she selects a device that supports OWE from the WPA3 certification and is at a location that supports OWE, she can now have some level of assurance that when she surfs the internet and sees the lock symbol in the upper left, she doesn’t have to call me freaking out.  People not calling me freaking out is a good thing by the way.

So what’s next?  Good question.  Start by going and watching Chuck’s presentation at Mobility Field Day 3.  Watch the reaction from the delegates at what they presented, and then watch the video again to let it sink in of what you just saw.  GCMP/CCMP protected data over the air on a guest Wi-Fi network.  The user only has to select the network and the protocols then take care of the rest.

Next, start bothering your infrastructure vendor of choice to find out what they are doing in the realm of OWE.  Is it on their roadmap?  When are they going to be releasing something about OWE?  If it’s not something they working on, why not?  Aruba has taken the lead on getting this into the public space, it’s now inherent on us, especially those of us in the Large Public Venue (LPV) realm to push ALL vendors to support this.

After the infrastructure vendors, start working on the client side.  Remember, you need a client side device that can do this, or the ability to add it, to make this work.  Ask Apple, Samsung, Motorola, LG, Dell, HP, and all the others, what their roadmap is for supporting this.  Predictions are we will see 802.11ax clients next year and I really hope they have the supplicant side ability to do this.  If not, as an industry we are missing a HUGE opportunity here and I for one won’t sit idly by and watch this opportunity slip away simply because we don’t have to.

I am sure I will harp on this subject more in the future, I think that it’s just that important.  When and if I come up with anything new I will make sure that I share it, but for now I want to thank Aruba Networks and their engineers for taking the lead in this effort.

It’s not the new features you thought you wanted, it’s the features that you didn’t think you needed!

**Disclaimer – I have not received any financial compensation or consideration from Aruba Networks for my thoughts here.  These are my thoughts and opinions alone and Aruba Networks is not responsible or forewarned about what you read.**

Mobility Field Day 3 – A Delegate

So I just finished my first Mobility Field Day 3, #MFD3, put on by Tech Field Day (Gestalt IT.)  It was one of the most amazing experiences in my life!

At this point, you are probably thinking “well, here comes the book report on each presentation and how I was ‘blown away’ by all the vendors that presented” so I will stop reading now.  Sorry to disappoint you but this is not one of those.  Don’t worry, I took many notes and thanks to the way that Tech Field Day is recorded, there are video recordings of all the presentations so I can go back, review the tape and write extensive novels on every presentation and bore you that way.  War and Peace wasn’t written in one weekend (maybe it was, I’m too tired to check right now) so give me some time.

What I want to discuss is the behind the scenes, the “sausage making” that I was completely unaware of, and the rest of the delegates that joined me in our merry jaunt through Silicon Valley (or “Silicone Valley” as one person called it.)

First up is Gestalt IT (@GestaltIT) and the team that was here on the ground with us.  If you don’t know, Gestalt IT is the company formed by Stephen Foskett (@SFoskett) to facilitate getting people together and discussing the topic of the day.  The Gestalt IT team for my first MFD was made up of Tom Hollingsworth (@networkingnerd) and Ben Gage (@BenTGage).  With detailed emails before the event and then from the moment each delegate touched down at the airport, they had everything under control and guided the two new people, me and Scott Lester (@theITrebel) with what we needed to do and wrangled the veterans like the herd of wild cats they are.  I’m pretty sure that without their patience, understanding and just the right amount of snarkiness and sass that none of this would work.  At least not with the crowd I was fortunate enough to join.  Luckily it didn’t take Scott and I very long to shake the newness and jump right into the herd of wild cats to ensure that Tom and Ben had a full compliment of crazy, irreverent wireless geeks to wrangle through the rough and tumble streets of technology town and not just 10.

You’re welcome Tom and Ben!

For three days they tried to get us to focus on the task at hand and behave like adults that we have fooled people into thinking we are.  All I can say is if you watched the live stream and the videos posted on YouTube and Vimeo after the fact it looked they they did a fantastic job with their task; it’s because they did.  Off camera we lived up to the wild cat moniker.  Me personally, I loved it.  More than one time the group had me laughing so hard I cried.  To reveal a secret, even Tom Hollingsworth joined the wild cat pack at times.  Luckily Ben Gage was there to keep us inline.  Ben’s a musician so I find it funny that the musician was the adult of the group.  Tell me how many times you get to say that!  Seriously, Ben deserves an award after our group!

Now for the delegates.

You can go read about them on the site but I want to round this up by filling you in on a few surprises I found out by hanging with the rest of the delegates.

  1. Robert Boardman – I’ve known Rob as the quirky part of the Wi-Fi of Everything duo of him and Rowell Dionicio (I have a man crush on Rowell, and he was here as a delegate as well, but this is about Rob.)  In one of the bigger surprises for me this week, Rob is actually really smart!  My perception of him completely changed when the camera went live and the heat was on.  His questions and understanding of the vendors and the technology was impressive and I have a new found respect of him. Don’t get me wrong, he stilled delivered comedy relief while the camera was on, and he can be an even bigger dork off camera, but my whole perception of Rob has changed for the better because of this.
  2. Jennifer Huber – Jennifer worked a project for us one time a while ago, and I always thought of her as a proper wireless expert and very knowledgeable.  I’m sorry Jennifer, but I might even say a little “boring.”  She teaches yoga and eats healthy, and I really thought we wouldn’t get along much.  Boy was I wrong!  First day we sat next to each other through the Apple product launch, and she can turn on the angry woman in a flash!  The words I heard coming from the professional sitting to my left was impressive!  By the end of the Apple product launch I realized that I could really dig sitting next to her for the next three days and listen to her go off.  Major props Jennifer!
  3. Keith Parsons – Everyone knows Keith, and everyone loves Keith.  I have written many blog posts that talk about Keith and his role in my wireless journey, but this isn’t about that.  I sat next to Keith a couple of times and I don’t know why this surprises me, but this is what I learned about Keith.  That man can multi-task!  I would look over and he would be doing something on his computer that made me think he wasn’t paying attention and then BANG!  Keith would be asking tough questions about the presentation and taking their experts to task on what they said, and never letting them hide.  One more thing to add to his myriad of skills and why it’s good to be friends with him.
  4. Lee Badman – Lee, from #WIFIQ fame, is the crotchety Wi-Fi man that you think of when you think of the hero of the suffers of bug infested wireless code.  What I didn’t realize about Lee until this week is the man has a dry sense of humor that I find very appealing.  That guy, if you give him the time, has some of the funniest ideas that I heard all week.  I don’t want to give them away, but I really hope he follows through with the idea he presented during lunch on the last day.  Lee, please help out the community and move that idea to the top of the list.  It’s what the community needs, even if they don’t know it.

As for the other delegates, please don’t get me wrong, they were all the rock stars you think of when you hear their name.  No disrespect to Amy Arnold, Johnathon Davis, Mitch Dickey, Rowell Dionicio, Sam Clements, Scott Lester, and Stew Goumans for not making my list above.  Every single time there was something that needed to be said, a point made, calling out a vendor for not answering a question, whatever, they were always willing to step up and say what needed to be said.  Sam was the gracious expert that I needed, Rowell was the quiet professional you think of, Stew was the 802.11eh representative we lacked.  Amy was the star of the wired side when we needed the support for that subject, JD was constantly there keeping things moving and Mitch was the SCA champion you know him to be (and the Red Bull champion of the week, hope your heart is ok!) and Scott was always ready with a question or additional guidance and input.

I know that without the full compliment of delegates that the team was able to assemble, my week wouldn’t have been as great.  In the end, I got just as much from the group that I traveled with as the vendors that presented.  My hope is that as the group of delegates, we were able to represent the community at large and did our best to ensure that those that didn’t attend in person were able to benefit from our work as well.

MFD3 Group pic_jpg


A Story of Three Companies

During Mobility Field Day 3, we were fortunate enough to visit with three different companies that were in different stages of mergers/acquisitions.  To be fair, the third company, NETSCOUT, hadn’t announced anything while we were onsite, it was business as usual.  This post is being written with the benefit of hindsight.  Luckily for me, it bookends my thoughts nicely so winner-winner, chicken dinner for me!  I’ll get back to NETSCOUT here in a bit.

In chronological order, we met with Arista first.  I know that some might ask why the Mobility Field Day delegates met with Arista.  Some might know why and some might ask who is Arista.  For those not in the know, you can get caught up here.  Shortest story is Arista acquired Mojo last month, so now they are a “cognitive Wi-Fi” company.  I don’t know what that means, and honestly after sitting through a 2 hour presentation with mostly Arista folks and not enough Mojo folks, I still don’t know what that is supposed to mean.  I get why they presented, but I know that as a group we were mostly confused on what was going on during the first hour.  As a first time delegate I didn’t know if the majority of the presentations were going to be this dry and wandering or not.  (Luckily for me, they weren’t.)

My thought after the presentation was here was a company that wanted to be able to support full stack across the enterprise with some version of Artificial Intelligence (A.I.) or Machine Learning (M.L.) that I am still not clear about.  Either way, they wanted/needed a wireless product they could slap their name on and go forth and prosper.  Granted, until they get an access layer switch that provides PoE (RIGHT?!?!) that won’t happen, but I suspect it won’t be too long before that is announced.  At least that is my hope after our feedback.

The next day we met with Fortinet.  Most everyone knows about the Fortinet acquisition of Meru since it did happen back in May of 2015.  What I want to discuss is how their presentation went during #MFD3 and what was learned.  After WLPC 2018 in the US, Mitch Dickey of @Badger_Fi fame wrote an open letter of displeasure to Fortinet asking them to step up and do a better job of explaining what they were as a wireless company, not just a security company.

Boy did they listen!

Fortinet did a great job presenting how their wireless product integrated with the rest of their portfolio and how it was more than something bolted on as an afterthought.  They also announced (OK, it has always been a thing, they just pointed it out) that the Fortinet wireless line was cable of running in both SCA AND MCA configurations!  I know for some of the delegates in the room, this was a new thing to learn.  I also know from some phone calls since #MFD3 that others didn’t know that as well.  The message that was delivered by the Fortinet team was smooth and eye opening.  As we left their facility at the end of the day, the general consensus was that Fortinet listened, changed their approach, and delivered with a great presentation at #MFD3.  While I agree they did a great job with their presentation and everyone was impressed, I want to point out that they didn’t really announce anything new or groundbreaking while we were there.  More on that later.

The next morning we went to NETSCOUT for their presentation.  We didn’t know it at the time, but the ENTIRE product line that they presented on was almost 2 feet out the door.  Think about this; they presented Friday morning at 9 AM PDT and when I woke up at 6:30 AM PDT on Monday they had already made the announcement.  As far as I know the ink was dry on the deal on Friday and they were just waiting for the approval on the press release wording.  Their presenter, Julio Petrovich, did a great job talking about their product line for 2 hours all while having to have some inkling, or concrete knowledge, that something was afoot.  You should go back and watch his presentation, I’ll add the links at the end, and keep in mind of what he might have known or assumed all while presenting.  Got to give the guy some props for that!  One other key piece of information is that I know that Julio will be moving with the Handheld Network Test (HNT) product line as it is “carved out” of NETSCOUT.

All of this to bring me to my point.  While I know company acquisitions and mergers and such are common place in “The Valley” (HP Enterprises bought Aruba in 2015 as a point), for me it was interesting to see such a different approach to their presentations, and possibly a lesson to be learned for whatever the HNT line that was spun out of NETSCOUT will be called.

For three years after Fortinet acquired Meru, I would say they languished in misinformation and confusion about what they were as a wireless company and what they could offer to the wireless community.  I would say that Mitch calling them out after WLPC, other feedback from the community and with the efforts of a gentleman named Christopher Hinsz, Fortinet turned their message around.  I give Chris a lot of credit because he did a nice job while he was at NETSCOUT and it was readily apparent that he had a big hand in the presentation Fortinet did at #MFD3.

To both the “TBD” handheld network testing company that we just found out about on 17 September 2018, and to the team over at Arista, you are both on the same journey, just a couple of weeks apart.  Take a page from what they did at Fortinet (just don’t take Chris, I like him at Fortinet) and learn that the wireless community is always there to help.  That was what we tried to do at #MFD3, and the community at large is always willing to chime in (some better than others) on the good and the bad that a company is doing.

Lastly, and I hate to say this, this is all about your messaging.  Not marketing, we can smell that out with both eyes tied behind our back, but your actual message.  Fortinet was able to redeem themselves by presenting a concise, cohesive message to the wireless community, and that means something.

Be honest.  Admit when things are still in the works but not ready yet.  As a community we are all used to things not going our way (ever heard of client drivers?) and are generally forgiving, especially early on.  Listen to the feedback we give on social media and at conferences, it will go a long way as you try to navigate the world of people that MIGHT have been exposed to just a little more radiation than is really recommended.

You can find all the videos from #MFD3 on YouTube

Cost Of Perfection

When asked, pretty much ANYBODY in IT can tell you what ROI means.  It’s the return on your investment.  What you get for what you put in.  When talking money, it’s really easy to calculate and most people are on board with that.  I am too.

What I want to discuss is the same idea but on a more general, abstract scale.  Last week at #MFD3 I spent some time with some pretty smart Wi-Fi folks and the topic turned to antennas and feedline, Polyphasers (a term was never used by the way) and other assorted instruments used in the day to day use of outdoor radio gear.  Sorry to rain on your parade, but all of this stuff was in use long before 802.11 was even a thing so the topic was applicable to all outdoor gear that is connected to a radio; whether the radio is outside or not wasn’t point.  What this is used for is when the antenna is mounted outdoors.

Disclaimer – I only can think of one scenario where the radio might be mounted outside and the antenna mounted inside, and none of my Wi-Fi peeps should ever do it. It’s a bad idea for 802.11 that I won’t even mention.

We talked about outdoor specific AP’s, AP’s designated as a “P” version as opposed to just a regular “E” types and all the other nerdy stuff that came along with it.  It was during this conversation that it started to dawn on me that we were fulling the age old adage that if you put 10 Wi-Fi professionals in a room and ask for a solution you will get 12 answers, and they are all acceptable.

Don’t get me wrong, it was an evening well spent and I fully enjoyed it.

As the group broke up, it made me start to ponder.  As an evening event, it was time well spent.  Had this happened to me during the day when I actually have other things to get done, I might not have the same attitude about how my time was spent.

Wi-Fi is designed to work when things aren’t awesome, and in most cases, that is what most people end up using on a day to day basis, my customers included.  Every time I look at my system I see all the faults in it, things I want to change and wish I could fix.  Other devices that now have AP’s imbedded in them, things that used to never have them in the past that are now causing problems.  What I discovered was that in my pursuit of perfection, I was chasing a dream that I was never going to reach, and driving me insane in the process.

I think at times, as a wireless professional, we chase perfection without any concern about being able to accept the good.  We worry about 1 dB loss here and a 5 degree difference there, but at what cost?  Does that 1 dB really make that big of a difference?  It might, and in some cases it could be the difference in making a solution work, but is that the only time you chase that 1 dB?  It might be time to pay attention to the work you do to try and achieve that last 1 dB to get from MCS 8 to MCS 9 when in the end, the only thing the client really needs to operate is MCS 5.

Remember, Wi-Fi is designed to work when conditions aren’t optimum, and sometimes, good is all we need to get there.

Long Live the Controller!

Lately, it appears that every time I turn around, I read somewhere where everything, and I mean EVERYTHING is moving to the cloud.  Maybe I am an “old geezer” in this respect, but I believe that not everything belongs in “The Cloud.”

In this particular post I want to focus on the heart of WLAN infrastructure, the venerable WLC.  Now granted, there are situations and the always present “It Depends” that can call for a controller in the cloud, or offsite controller, or controller-less, or mesh, or whatever the vendor is calling it this week, but sometimes, in some situations, having a physical, on-site good old fashioned controller just can’t be beat.

In my current employment, I work at a facility that covers 53 square miles.  Granted, not all of that space if covered in buildings and facilities that have Wi-Fi, or network connectivity (although we have received that request more than once) but we do have facilities that are pretty well spread out.  While I don’t want to spell out all the details, we also have a massive fiber infrastructure that allows us to do some pretty cool things all in house, and we don’t rely on leased lines, or ISP’s, for anything other than our internet connectivity.

Hopefully, at this point, you get the idea of where I am coming from when I say that in an environment like mine, having a centralized, on-premises, good old fashioned chunk of metal and electronics programmed to be a Wireless LAN Controller is a great thing!

Look, I get it.  Not every customer is going to be.  Not every customer can provide their own dedicated fiber between buildings miles apart to get sub-millisecond latency between hardware, but I can.  Not every customer benefits from centralized forwarding, and that’s fine.  I’m not saying that all of the other solutions are not warranted, and don’t have their advantages; they really do.  I can think of a myriad of customers and/or situations where either fully cloud based or a hybrid solution is definitely the way to go.  Companies that have a large central office with branch offices spread across the country immediately springs to mind of a situation where either a full cloud based or hybrid solution would be, and should be, the solution of choice.

Everybody can agree that when it comes to RF coverage, AP placement and AP count, that it all depends on the requirements of the space.  The same thing applies to selecting how the WLAN will be managed and controlled and which type of solution is eventually installed.  Requirements should be the first decision, then cost.  Whether or not your chosen vendor has just rolled out a new shiny cloud based solution should NEVER factor into that decision making process.  I get that sometimes cost will over-ride everything, I’ve been on that side of the fence before, but please don’t immediately jump there, give hardware a chance!

Let me give you some examples in my argument for centralized forwarding to an on-site controller.  Sorry, I can’t bring myself to call in “on prem” or “on premises” or whatever marketing calls it this year.

  1. Configuration of my access layer switch ports has been standardized to a single configuration.  Since I only need an access port with a single VLAN, the wired network team now knows how to configure a switch port where an AP is being installed without the wireless “team” getting involved.  You would be surprised how confusing WLAN technology can be to wired guys who have never dealt with it in the past.  If I need to do a flex connect type scenario, it’s rare enough that I don’t mind dealing with it personally.
  2. VLAN segmentation is much, MUCH easier.  I currently have 28 active VLAN’s off of my WLC’s, and only having to deal with them on a couple of switches relieves a lot of stress, questions and mis-configurations from the wired team.
  3. Security is easier to implement.  I run a Cisco WLAN, so there is an encapsulated (not encrypted) CAPWAP tunnel between the AP and the WLC.  In my environment we added an additional routing “feature” around the CAPWAP to keep it locked down.  That was a one-time configuration challenge that we haven’t had to go back and touch, no matter how many VLAN’s I have added to the WLC.
  4. Using the CAPWAP functionality allows me to “get around” network segmentation on the logical network.  In certain circumstances, it can be very advantageous to have 2 devices 10 miles apart but on the same subnet since they both terminate at the same location.  Yes, concentrators can be used to achieve the same thing but if I have to add hardware onsite, why add just that?  A concentrator will add complexity and another point of failure to deal with, so now I need to add in redundancy.
  5. I have full control over when and how my upgrades are done.  Yes, in theory this shouldn’t be an argument since it is your cloud instance, but how many times have you had a service in the cloud have an update or reboot done simply by accident?  As the engineer/architect on record, I am always the first one blamed.  This leads to the next point.
  6. Troubleshooting during outages is frustrating.  Even when things are in the cloud we are blamed for outages, and in our group alone we have spent countless hours trying to show that issues with reaching an offsite service is an ISP problem, not ours or the cloud data center’s fault.  What ends up happening is we point the finger at the cloud provider, the cloud provider points the finger at us.  Eventually we point a finger at an ISP.  Ever try to get two different ISP’s working together to solve a problem?  It’s bad enough when you are paying them for service and you need them to work for you, let alone work with a different ISP to figure out routing problems between themselves.  It’s a nightmare, and as the customers technical people we are always left holding the bag.

I could go on, but I think you get the point.  Keep in mind, I am not here to say that cloud based controller solutions are the devil or should go away.  On the contrary, I think in the correct situation, cloud based is 100% the way to go, and all vendors should be able to support that model.  I am just here to argue that in that same vein of thought, in the correct situation, physical, on-site, metal chassis based controllers are still very pertinent and needs to be considered as a viable, if not the correct, solution for some solutions. And just like with cloud based controllers, all vendors should be able to support that model.  If not, in my mind, they will always be a second tier vendor since they can’t support ALL possible solutions needed for any given customer.

As Lee Badman reminded us in the #WIFIQ for 8/21/18, try and take emotions out of the discussion.  Emotion should never be part of the conversation when designing the correct WLAN solution for any customer.  Define the requirements and design the solution based on those requirements.  The solution will change based on other factors but to say that I won’t recommend a physical controller no matter what just isn’t fair, and isn’t in keeping with the spirit of designing the best Wi-Fi for any given scenario.

Let me know your thoughts on the subject, sometimes 288 characters just isn’t enough to make your argument.

P.S. – I also don’t think 2.4 GHz is dead and will argue that one until the end of time!  Maybe I am the old geezer who won’t change!


Setting Up Cisco wIDS/wIPS, Part 3

In Part 1 of my Cisco wIDS/wIPS adventure, found here, I covered trying to scale and scope what it is you need before trying to purchase a Cisco wIDS/wIPS.  If you haven’t read that one, you should go do that now and then move on to Part 2, the link is at the end of Part 1.

In Part 2, I covered how the information flows within the Holy Trinity and lessons learned that I think better explain what’s going on so when you tackle this mess you don’t waste as much time as me.  Assuming you got to Part 3 by reading Part 1 and skipping Part 2 somehow, it’s located here and I suggest you give that a glance before continuing.

OK – on to Part 3!  By this point you should have a WLAN that is up and running and passing traffic (admit it, videos of cats and people doing dumb stuff is much more fun than this so I hope got that working already) as well as a somewhat functioning Cisco Prime Infrastructure (here on out referred to as CPI, or “Sub-Prime” by a certain individual that will remain anonymous) and now a nice shiny new MSE with some expensive licenses and and not much else going on.  Feeling a little angry and cheated?  Good, that’s how I felt as well.  Now that your MSE is up and running, go ahead and forget about it for a while, it’s job is done.

We now get to work on CPI and the myriad of things it does.  I’m going to dangle the carrot here a second and show you what you are hoping happens if all goes well.  On CPI navigate to the menu and go to “Dashboard –> Wireless –> Security.”  What you want to see is something that looks like this:dashboard

Now, what shows up on your page will either be nothing (because you haven’t finished configuring it) or hopefully something similar  to the image on the right, because you haven’t tuned your system to match what I did.  Different environments should have different alert profiles.  Ideally, what you will see here is a whole host of alerts that then allow you to fine tune the alerts down based on the fine security policy that you created in Part 1.  If not, I will get to that in a minute.  If you don’t see any alerts, let’s get to work on CPI to get you something pretty to show management, and more importantly the project manager so they will leave you alone!

Navigate to the menu on CPI and go to “Services –> Mobility Services –> Mobility Services Engines.”  On this screen you should see your new shiny MSE listed with a name, device type, IP address, version, reachability and then the services running on it.  If no servers are listed, look in the upper right corner and using the drop down select “Add Mobility Services Engine” and select go.  After walking through the steps, and assuming you wrote down the information from the installation correctly, it should add the MSE to this page and give you a reachability status of “Reachable.”  If it’s not reachable check your firewall configurations (remember from Part 2?) or MSE configuration.  Remember, these 2 servers were designed to talk to each other and want to communicate so it’s not like you are linking an Apple product to your Windows laptop.  Also, from this screen, you can click on the device name (highlighted with blue letters) and it will take you to the MSE GUI log-in screen.  This helps when others want to access the MSE GUI later.  Don’t ask me why, my boss is nosy and yours might be as well.

Next step is to synchronize the MSE with the WLC’s.  Not sure why you do it from CPI, but it’s where you do it.  In Part 2 I showed you what it looks like when the synchronization is working, this is how you do it as well as what to do when it doesn’t work.  Top tip, it won’t show as working the first time so don’t panic.

On CPI navigate to “Services –> Mobility Services –> Synchronize Services.”  The first page that will appear will be the maps and buildings that you may or may not have built in CPI.  These are used for your other MSE (or CMX) and aren’t applicable here.  Don’t panic just look in the grey box on the left hand side and pick the second entry down, “Controllers.”  This will pull up a list of all network devices that CPI has identified on your wired network that report themselves as controllers.  I personally have some WLC’s that I think of as controllers(3504, 8510) and then some 3850 and 4507 catalyst switches listed as well.  If you have some weird issues with your WLAN infrastructure, looking here might give you some places to start figuring out why you have a Catalyst 3850 that still thinks it’s a WLC.  Luckily I have a couple of wired networking guys that help me with this, you might not be as lucky.

Anyways, select the controller(s) that you want to actually use by checking the box on the left side of it’s name and at the bottom of the list, select “Change MSE Assignment.”  This process might need to be repeated in the future and is handy when troubleshooting so it’s a good page to remember how to get to.  Once you select that button it will give you a list of functioning MSE’s and the service you want to assign.  Find your new shiny wIPS MSE in the name list and check the box to the right of it under “wIPS” and then select “Synchronize.”  It will take you back to the list of controllers and on the right hand side will show you the service, which MSE that service is running on and if it is synchronized.  In my experience, it won’t show you a good status at this point.  My suggestion is to go to lunch at this point, let it sit and come back in a little bit to check things.  It’s painful, but eating a good lunch is much better than banging your head on a table.

After lunch come back to the screen from before, refresh it and see if the status has changed.  If it’s working, great!  If not, select the blue entry next to the WLC and MSE that isn’t working named “NMSP Status.”  This is the only way I know to get to this screen, and it’s probably the best and most helpful place to look at your new MSE.  There is even a place to troubleshoot why the MSE and WLC aren’t synced.  For that look at the NMSP Status in the summary and click on the status (it looks like a doctors stethoscope.)

Keep in mind, this is what tells the WLC where to send the wIPS NMSP traffic in the first place, so it’s a critical step.  No data in, no data out.  In this scenario; no NMSP in, no SNMP traps out.  I will also point out that it might take a couple of attempts to get the sync to work.  Go back and select your WLC’s and change the MSE assignment to nothing (un-select the boxes) and sync again.  Go back and try synchronizing them again.  Another tip, NTP can be an issue here so verify you have a good time source.  In my experience, everything can be correct and it will take a couple of attempts to get them to talk.  Keep with it, it will happen.  After it’s synced up, move onto the profiles


This is a big enough step I wanted to highlight it as it’s own section.  It’s where you make alerts show up, and maybe don’t show up.  It takes some time, focus, and a good written policy before beginning.  It’s also going to take longer than you think, so be prepared.  While the steps are pretty straight forward, it’s just a bear to do.

On CPI, navigate to “Services –> Mobility Services –> wIPS profiles.”  You can use the default profile to play around with, but my suggestion is to build a profile based on the deployment, how many locations and WLC’s it will be deployed to.  It’s OK, you can base the new profile on a couple of different pre-built profiles, the default being one of them.  Using the drop down on the right hand side select “Add Profile” and select go.  Name your profile and select a profile to copy from.  If your type of environment is listed, or is close to one that is listed, I suggest choosing that here; it could save you a lot of time in the following steps.  Select “Save and Edit” and it gets you going.

On this next step it’s best to understand what this really is.  I went through it a couple of times before I read the note in the guide that says this really only applies to an overlay system.  If you selected an integrated system way back when, this section really doesn’t apply so you can skip it.  The AP’s reporting the wIPS stuff already know about what SSID’s them and their neighbors have so you don’t need to define it here.  This really only comes into play if you are managing an overlay system or multiple systems.  If you are doing a true overlay, take the time to read the manual on this step and build in the SSID’s you are really concerned about.  If this is for an integrated system, simply ignore everything and click next.

Now for the fun, and time consuming part.  You are looking at a 2 “window” screen at this point.  The window on the left is all of the available alerts (roughly 260) and the window on the right will change based on which alert is selected.  The box next to the alert name indicates if that particular alert/alarm is included in the profile.  This is the step that tells the MSE what to care about and what should trigger an SNMP Trap to be sent to the trap receiver(s).  My only suggestion is bring a snack and a drink and expect to spend some time here.  Click on each alert name and read the corresponding window on the right.  The upper section will allow you to adjust certain parameters of the alert while the bottom section will explain the alert.  It’s fun to read through all of that only to get to a statement that says something like “Cisco WLAN knows about this and can deal with it so this alert isn’t needed.”  Make sure those alerts are un-selected on the left hand side and move to the next one.

Going to the way back machine, remember when I stated that having a written policy is crucial before doing this?  If you don’t have a written policy before this step, you will fully understand what I meant.  While technically some alerts make sense (like a dictionary attack,) most fall into the category of technically it doesn’t matter one way or the other, but what do others expect to see out of this system.  Not having a policy makes selecting alerts difficult, if not impossible.  Don’t worry, it’s been my experience that you will visit this list many times before everyone is happy and then sporadically in the future as alerts are found to be false positives or new alerts emerge.  Either way, expect to spend some time here and be patient, it’s at the heart of the system.

Once at the bottom of the list, click the “Save” button at the top.  It’s a no brainer, but it has bitten me in the past.  Click save before hitting next, don’t count on or expect the system to prompt you, though it might.  After clicking “Next” we are almost home!  Select the WLC’s that you want your new policy to apply to by checking the box  to the left of the WLC name you want included in this and then hit “Apply” at the bottom of that window.  You’ll see an alert/warning that’s it’s applying the profile, and you are done!  Pat yourself on the back, get up and stretch your legs and go get a drink!

Also, it will be at this point you realize that you can configure, and apply, different policies to different WLC’s based on what they do.  Going back and creating a new policy, defining the alerts and then applying the new policy to the different WLC is all it takes to care about Mi-Fi’s in one environment but not in a different one.  Unfortunately, I don’t know how to apply different policies to the same WLC but different AP’s.  I don’t think you can but I am open to being corrected.

Now, when you get back, navigate to “Dashboard –> Wireless –> Security” and you will now see your own list of crazy things happening in your environment.  Selecting the count to the right of the alert will give you a detailing listing of that alert.  It’s about this time you will realize that while this will suffice for just about any auditor, functionally it’s lacking.  This is where adding a second SNMP trap receiver, like Splunk, comes in handy.  You can either do that from the MSE GUI itself or from CPI on the page where the NMSP status link got you.  Adding it is simple and straight forward.  Realizing later that there isn’t a published MIB for the SNMP traps sent by the MSE makes you want to pull your hair out.

Side note here – if anyone figures that part out I would appreciate a link to a blog post.  I wasted too much time on figuring that out and had to move on without great results.

That’s it!  Your Cisco wIPS/wIDS is now functioning, and it’s just a matter of setting either more or less AP’s into wIPS mode and/or changing the alerts you are seeing.

My last input is this.  If you don’t have a written policy by this point, all of this was for naught.  These systems need to be cared for and updated.  There needs to be constant babysitting of the alerts generated and guidance on what to do about the alerts.  A lack of written, enforceable policy means the system you just built will exist only for the auditors, and that’s just depressing.

If you want to go back and read Part 1, click here.

If you want to go back and read Part 2, click here.

A link to official guide can be found here.

If you have any input or additional tricks, tips, or steps I forgot, please leave me a comment and I will make changes as needed.  Remember, this isn’t meant to replace the official Cisco guide (found here), just to tell my story and tips and tricks I learned along the way.  Thanks for hanging with me and getting to the end!

Until next time!

Setting Up Cisco wIDS/wIPS, Part 2

In Part 1 of my Cisco wIDS/wIPS adventure, found here, I covered trying to scale and scope what it is you need before trying to design and purchase a Cisco wIDS/wIPS solution.  If you haven’t read that one, you should go do that now and then come back here.

In Part 2, we are going to cover deploying your new and shiny Cisco wIPS solution, and then watch the magic start to pour in in Part 3!  I’m not going to get into how to install Cisco Prime Infrastructure or the MSE 8.0 with the wIPS licenses, the guides are pretty straight forward and self explanatory, that’s not what this is about.  The one lesson learned for this section, and I think it’s a lesson most of us have learned at one point in our career, is understanding where the enterprise firewall’s live, logically, in the intended architecture.  This will come in handy later when certain things just aren’t happening, or when it’s time to submit those request to get the firewall changed.

Before going much further, I want to highlight a drawing from the guide that I want to explain in detail, since no one really did this for me and it cost me some time trying to figure it out.  I know the quality is poor but I suspect it’s been copied and pasted more than once!Screenshot from 2018-08-19 15-13-49

With this layout I think you can see where the term “Holy Trinity” came from, and in a little bit you will really understand it.  As you can see, the WLAN hardware in the lower left centers on the WLC as this is a converged deployment.  I think you could do this in a different mode but at scale it might prove tough to manage.  Anyways, as you can see by the arrows, there are different communications protocols that happen within this trinity, and they all come into play when setting up and running a Cisco wIPS/wIDS system.

If there is one thing I need to impress at this point is while the MSE plays a significant role in the functionality of the system, other than the basic setup and configuration to get it to “talk” on your network, you actually don’t DO anything on the MSE server.  There is a couple of set up tasks that you can go back and add later (licensing and SNMP traps) but in a functioning solution, there isn’t any to see or do here.  Let me show you my live system to show you what I mean.  The main screen looks like this:

MSE Status

Not much to see, but it shows that something is happening.  The other screen to check is called the NMSP status page, and this gives you just that.  By referring to the trinity picture above, you can see that NMSP is the communication between the WLC and MSE, and it shows that information is indeed flowing INTO the MSE, which is step 1.  To give you an idea, this is what that looks like:

NMSP Status

As far as the MSE server goes, that’s about it.  There are a couple of other screens, but they are self explanatory (licensing and SNMP trap destinations) so I won’t cover them.  That and SNMP trap destinations can be done on Prime Infrastructure.  Biggest thing I learned over time, that no one pointed out to me and the one thing I want you to take away from this, almost ALL of the work is done on Cisco Prime Infrastructure (here on out referred to as CPI.)  Let’s break that down.

I know the picture above shows all three devices playing a part, and they do, but do you notice which server is at the top of the pyramid?  It’s CPI.  Assuming the WLC and Wi-Fi infrastructure is already up and running and you added CPI to give you some historical visibility into the system (and then they tried to tell you to do all the management of the system with CPI and you said no, just like me) the MSE is the last piece of this trinity.  It probably does the most work in a wIPS solution with the least amount of credit.

Using either a configuration template from CPI or manually selecting the option on the AP via the WLC, you are going to have wIPS information being sent from an AP configured in WIPS mode to the WLC via the CAPWAP tunnel and then from the WLC to the MSE server using the NMSP protocol.  The MSE is going to see that information as having come from the WLC (hence the NMSP status pages I just pointed out) but what it does with that information is controlled by CPI.  CPI is where the profile being used (basically what alarms you want to see) is configured and then pushed to the MSE via the SOAP/XML link link between the two.

Now when the NMSP traffic from the environment hits the MSE, it analyzes that traffic looking for different patterns that indicate known events happening.  When it sees a pattern that matches a known event, it triggers an SNMP trap from the MSE server to CPI.  During the configuration that is done for this solution on CPI, the IP address for CPI is inserted as the primary, and only SNMP trap destination in the wIPS MSE.  This is important since there isn’t anywhere on the MSE that you can actually tell that an alert has been triggered.  I spent a lot of time searching and there is no other way I can find that will show you this information.  It’s not in the syslog of MSE nor can you find anything great in the trap logs on the WLC.

Now, if you have turned on the mitigation part of wIPS there will be a coordinated effort from CPI to the WLC to deal with the offender, but only in certain circumstances have I ever heard of that functionality being enabled.  This is where the policy from Part 1 comes into play.  Accidentally doing a DDOS attack against a legitimate, neighbor AP can get you in trouble with the FCC so you better be sure your management has a written, approved policy that makes sense before turning that on.

To sum all this up, after initial configuration, it works like this:

  1. WIPS traffic from the WLAN is sent to the wIPS MSE using NMSP.
  2. WIPS MSE analyzes the data looking for specific patterns that match known “issues.”
  3. WIPS MSE identifies a known pattern and checks to see if it’s an issue that has been configred to alert on.
  4. If it is an alert-able pattern, MSE will send a SNMP trap to the destinations listed in it’s configuration (should be CPI by default, others can be added.)
  5. SNMP trap is logged on CPI, and others if configured, and depending on configuration can attempt to mitigate the offending device.

Like I said, the MSE does almost all of the work, but for all of that there isn’t much to actually see or do on the MSE.  If NMSP traffic is flowing and alerts are showing up on CPI and other trap receivers, then it’s working.

Coming up in Part 3 – Configuring and looking at these mythical wIPS/wIDS alerts.

If you missed it, Part 1 is here