Hi Clifford,
I’m still recovering from my time in Higher Ed, I jumped to local government after 20 years in Higher Ed. Full Disclosure, we weren’t running Rapid7 where I worked previously, so all things aren’t applicable.
Decentralized IT Environments:
We were centrally decentralized… by that I mean we had a central IT department that handled security, purchasing, and infrastructure (including servers). We also set standards for IT replacement/standardization of workstations so that we wouldn’t have some faculty members getting a new machine every year and others running a 10-year-old beater that was nothing but problems. Most faculty saw this as a benefit; for Central IT, it made our lives easier since we didn’t have a hodgepodge of systems to manage compliance. Central IT also “owned” the MDM and patch management solutions.
Departmental IT provided end-user support for applications, workstations, and other related systems.
Remediation Workflows: We skipped automation and instead trained our Central IT staff to enter tickets once they received notices from the Security Team. The method to the madness here:
We didn’t want to create work orders for items that would be taken care of in the regular patching cycles. If a vulnerability is remediated by a Microsoft patch that will be pushed out within a week, we keep it as a “watch item” within Central IT. We only spawned work orders for the Colleges / Departments if machines missed patches or were out of sync. Other tickets were generated and priority was set based on the severity of the vulnerability and risk involved (this came from security staff who also have to be trained that not everything is the end of the world).
For prioritization, SLAs were our best friends. Different departments had different requirements (PCI, HIPAA, CJIS, etc.). Rather than trying to set up dozens of different timelines (and fight faculty who balked at anyone’s authority other than their own), we tied the priorities back to frameworks and NIST best practices. We did receive some assistance from the State Government when our legislature decided that everyone had to adhere to a set of guidelines they published.
We had meetings, upon meetings, with the various colleges and departments across campus to explain why things had to change and to answer questions. The first meeting with each group was typically a bloodbath, after they got to vent a bit, we could get down to ironing things out. The key here was opening and maintaining lines of communication and explaining the “why”.
As mentioned earlier, I’d break each area down into a site and put all the tags in use. From there, set priority based on the impact.
Communication and Accountability:
Everyone likes communication and accountability, until they’re being held accountable… At the start, we held “town hall” type meetings with each College/Department. We kicked off with a brief intro, explained the “why”, and then opened the floor. Each one was similar: someone was upset their printer had been broken for two days, someone had issues that had never been reported, and someone was upset their “control” was being taken away. We took notes and followed up on each issue raised. Essentially, we started by holding ourselves accountable.
From there, we established SLAs, had them signed off on by the Dean/Director and the head of departmental IT. SLAs were reviewed (with metrics) annually. We also held quarterly updates with the Deans/Department heads and departmental IT (separately), so we could keep everyone up to date and ensure there were no surprises.
The best advice I can give here is what I learned from our college president: personalization and being personable is good business. Creating a trust-based/comfortable environment will take you a whole lot further than focusing solely on the business. That meant spending a bit of time talking about personal interests, asking about how vacations or conferences went, asking about research projects, etc. Meetings were always in person. This way, you can read body language and other unspoken communication. You’re able to determine better if there’s confusion, additional questions, and how the message is being received.
The personal approach led to much better communication. Instead of people asking for forgiveness, they started to view us as a trusted business partner and would reach out at the beginning of projects. When there were issues, we had rapport that kept the situations from going rabid.
Accountability also increased. With clear and better communication established, we adopted a three-step approach (when appropriate). We started with the person responsible for the issue (eg: workstation support tech). If the issue wasn’t addressed in an established timeline, it went to the supervisor and the person responsible. After that, if the issue persisted, we would go to the departmental IT head or the Dean. The big thing here, no surprises. These steps were laid out in the SLAs.
Tracking and reporting:
We modified our processes to match the tools at hand, utilizing the available dashboards and reports within the systems we were using. That’s the same approach we’re using here with remediation projects and the reports in our ticketing solution. I have things broken down by site, and individuals are given access to view the scan results for their respective areas.
Lessons Learned:
In addition to buy-in from Senior Leadership, you need to get buy-in from the “unofficial” power brokers. Maybe it’s a departmental admin assistant, the “tech-savvy” faculty member, or the “old guard” that’s been around for decades. These individuals need to be identified, and then they need to become supporters of the project as well.
Standardization across the board will be a huge help. Asset groups, tags, sites, etc. Before we rolled out Rapid7 here, I spent months reading the documentation, finding best practices, and then figuring out the changes we needed to make to our processes to best work with Rapid7.
Keep changes manageable… I’ve found it isn’t that people don’t like change, it’s that people have a limited capacity for change. Communication is key here: Determining the phases of roll-outs needs to be timed with other changes (not just within IT). Being flexible (to a degree) goes a long way, and knowing deadlines/changes impacting the other departments before your deadlines are set is a huge benefit.
Know the current environment, establish a baseline, and communicate it before reports start going out. It’s easy for individuals to get overwhelmed when they’re exposed to new data sets. If they haven’t seen a vulnerability report and are presented with one that states there are 4,011 vulnerabilities in their area, the likelihood of them shutting down or adopting a “we haven’t been hit yet” mindset is high. If they’re given the same information, but it shows there are 200 machines with 50 vulnerabilities that will be resolved with the next series of patches from Microsoft, it’s a lot more palatable.
Reward accomplishments and failures. If it’s new, mistakes are going to be made. Maybe you have to redo the sites, groups, etc. Explain why it’s changing, acknowledge the efforts that went into the initial setup that wasn’t optimal, but focus on what was learned. Get a stack of stationery and handwrite thank-you notes. Keep track of who receives one and why, so you can spread the love.