The Danger of Metrics

I got an email a few weeks ago about my article Operational Metrics: The Next Step in the Evolution of Defensive Cyberspace Operations1 from a researcher studying the same problem. We talked about my article, some of the other writing I have published on this topic, and and then something I briefly touched on in my SOC Metrics series: the dangers of metrics.

Read more

Artificial Intelligence Reading List

Thanks in no small part to OpenAI’s ChatGPT, the past year has seen an explosion in interest in artificial intelligence in general, and in large language models in particular. That democratization of access turned this niche research area into a common topic of conversation, and has led to a lot of fascinating writing on the subject. Although certainly not comprehensive, this article collects some of my favorite articles, papers, and resources into a single reading list.

Read more

You Don't Need AI, You Need an Algorithm

A few weeks ago, I participated in a working group tasked with identifying areas where artificial intelligence and machine learning could improve network utilization. What, specifically, “improve” and “utilization” meant were some of the first questions we addressed. During that conversation, though, one of the participants made an insightful observation that the group promptly ignored. He told a brief story about how someone had asked him for a networking device that could aggregate multiple uplinks and then “use AI to choose the best one.” When he offered a simple WAN failover solution, a technology that has existed for decades and comes built-in to open source firewalls like pfSense, commercial appliances from the likes of Cisco, and every modern mobile phone, the customer told him, “No.” Like our working group, that customer person had a solution (artificial intelligence) driving their requirements (an AI-based WAN failover), not the other way around.

Read more

Introducing Colors

Over a year ago I read Matthew Strom’s article How to pick the least wrong colors, where he explored the fascinating intricacies of designing good color schemes. Here, “good” meant not just whether the colors looked nice, but also how distinguishable they were from each other and whether or not they accomodated individuals with conditions like protanopia, for example, who cannot see red light. He came up with a great solution and posted the NodeJS code to GitHub. Since I do most of my work in Python, though, I had few chances to take advantage of his research — until now. My latest project, Colors, helps to generate large sets of attractive and accessible color schemes. The current version of this project is, for the most part, a direct NodeJS-to-Python port of Matthew’s code from 2022 with a few minor changes, but I look forward to building on it and keeping pace with Matthew. Colors is currently available as a Jupyter notebook and a regular Python script.

Read more

SANS Recommendations for Defensive Cyber Analysts

I keep fielding this question in private, so I finally decided to answer it in public: “What SANS courses should I take?” Although I have a much longer answer to give about training in general, this article answers that specific question based on my own personal experience, having taken several SANS courses over the years.

Read more

ChatGPT Wrote my Shell Prompt

I have, for years now, wanted a cool shell prompt — something worth showing off on r/linux or even r/unixporn. Not enough to dig into the weeds and figure out how to write one myself, but enough that it came up every once in a while. Then, the other day, I had a great idea: why not just let ChatGPT make it for me?

Read more

The Baseline Fallacy in Defensive Cyber Operations

Both civilian cybersecurity practitioners and military defensive cyber operations personnel continue to push a similar idea, that to identify malicious activity one must “just get a baseline and then flag anything different.” If only! I call this the Baseline Fallacy:

Read more

Collection, Transport, and Presentation: The Three Wicked Problems Inhibiting Data-Driven Decision-Making in the Army

Military leaders have sought hard data to drive their decisions for decades, perhaps most famously beginning with Secretary of Defense Robert McNamara’s so-called Whiz Kids’’ in the 1960s. As retrospective analysis of those decisions made clear, however, data alone does not good decisions make. Errors in collection, transmission, and presentation decimated the efficacy of this initiative.1 The Vietnam War is a cautionary tale in data-driven decision-making gone wrong, an important reminder that modernity’s insatiable need for more data is no more a silver bullet today than it was sixty years ago.

Read more

Git is My Hammer; Everything is a Nail

A few days ago, I mentioned that I use Git to synchronize work between several different devices and across several different platforms. This setup has served me well, and so today I want to talk about it in detail.

Read more

Changing History with git-filter-repo

I have used git to synchronize work between several different devices and across several different platforms for quite some time now. Over the years, though, inconsistencies have crept in. The name attached to my commit messages, for example, might be “Zac” on one device, “Zachary Szewczyk” on another, and “Zac Szewczyk” when committing via GitLab’s web editor. The same applied to the email address attached to those messages. Elijah Newren’s git-filter-repo project made it a breeze to fix this.

Read more

Cybersecurity Tools & Resources

My book Handbook for Defensive Cyberspace Operations contains an extensive chapter on tools and resources for cybersecurity analysts. As a closed project, however, this knowledge has had little impact outside of my niche community. This post contains a version of that chapter suitable for distribution to the public.

Read more

Passing CISSP

Like college degrees before them, professional certifications seem to be waning in popularity in the cybersecurity industry. Perhaps as a way to mitigate the well-documented workforce shortage, some companies have gradually begun to account for competence and experience, too, rather than onerous credentials alone — and in some cases, base their hiring decisions solely on those criteria. This is a localized phenomenon, though, exclusively limited to the private sector, and even there primarily limited to smaller firms. In the public sector, at many large firms, and particularly at large firms that work with the government, formal credentials such as a degree and professional certifications remain not only an important factor, but in many cases the only factor. This led me to take (ISC)<sup>2</sup>’s Certified Information Systems Security Professional (CISSP) exam earlier this summer. Even as a Cyber Operations Officer, credentials like this one carry great weight. I passed on my first attempt, so in this article I want to share my preparation strategy.

Read more

Individual versus Program SOC Metrics

The other day, a member of SANS’s GIAC Advisory Board asked a question about Mean Time to Detection, or MTTD. In the ensuing discussion, another member cautioned against confusing key performance indicators used to assess individual efficiency, to then serve as the basis for penalties and rewards, with measures of program efficiency. Holding individuals accountable for MTTD, for example, made little sense when that detection relied on factors outside of their control such as effective data collection and efficient aggregation into a centralized platform, each of which were likely to be managed by a separate department. I alluded to the challenge of implementing metrics in part one, but did not deal with this issue specifically. This bonus article provides specific recommendations for applying SOC metrics to individuals versus an entire program.

Read more

There is no iceberg

From David Heinemeier Hansson:

It’s human nature to assume there’s a good reason for why things are the way they are. And that this reason is either benign, based on careful deliberation, or malignant, derived from malice or incompetence. But this is a false dichotomy that often steers us away from the simpler answer: Nobody thought about this at all.

Given an absence of information, many tend to fill those gaps with generous assumptions. At the beginning of Russia’s botched invasion of Ukraine, national security pundits considered it some sort of 4D chess game; time proved otherwise. No one knows what they’re doing, “It’s OK though — that’s not where your problems are coming from. Rather, your problems are coming from the fact that you think other people know what they’re doing.”

Read more

Amateurs Talk Detection. Professionals Talk Collection.

I went to a presentation from a large cybersecurity firm the other day. The salesmen — and they were all salesmen, as even a few post-presentation questions made clear — focused on the intelligence their company produced, but knew little about the rest of their company’s intelligence cycle. As a consumer of their products, though, I consider knowledge of the process that created them critical. After an hour on finished intelligence, and in particular a slide that touted a “globally deployed sensor grid”, I wanted to know more about that production process in general, and about their collection specifically. Unfortunately, when asked, the salesmen offered a handful of handwavy “We have millions of sensors across the globe”, and, “We ingest billions of events per day” statements, but little beyond that.

Read more

Operational Metrics: The Next Step in the Evolution of Defensive Cyberspace Operations

As a senior First Lieutenant, I volunteered to teach new Second Lieutenants about defensive cyberspace operations. Over the course of a year I spent more than twelve hours with three different Cyber Basic Officer Leaders Course (BOLC) classes; each time, they asked the same question: what did your missions ac- complish? Each time, I struggled for an answer.

Read more

SOC Metrics, Part III: Measures of Effectiveness

In part one of the SOC Metrics series, I introduced the idea that success requires the right person doing the right things in the right ways. That article also described several foundational metrics, ways to measure the SOC’s ability to produce meaningful results. Part two then focused on measures of performance (MOPs), which assess whether or not (and to what degree) the SOC is doing the right things. This article delves into measures of effectiveness, the last step in defining useful SOC metrics.

Read more

SOC Metrics, Part II: Measures of Performance

In part one of the SOC Metrics series, I introduced the idea that success requires the right person doing the right things in the right ways. Measures of performance (MOPs) assess whether or not (and to what degree) the SOC is doing the right things, and measures of effectiveness (MOEs) assess whether or not (and to what degree) the SOC is doing them in the right ways. MOPs and MOEs rely on foundational metrics to produce meaningful results, though, and so I started with them in part one. This article delves into measures of performance, the next key step in defining useful SOC metrics.

Read more

SOC Metrics, Part I: Foundational Metrics

I prefer a hands-off approach to management. Describe a goal, give me the freedom and resources to achieve it, and I will. The success of that approach, which the Army calls “mission command”, ultimately depends on the right person doing the right things in the right ways. Unfortunately, many focus on that first criterion but neglect the second and the third. In many cases, that leads to failure. In the final days of my last job, in a security operations center (SOC), I realized that this explained many of our systemic problems. We had the right people, they just did the right things in the wrong ways or — in some cases — did the wrong things altogether. This was not an inherent consequence of that hands-off approach, of mission command, but rather a necessary consequence of its partial implementation: we lacked measures of performance (MOPs), to assess whether or not (and to what degree) we were doing the right things, and measures of effectiveness (MOEs), to assess whether or not (and to what degree) we were doing them in the right ways. In their absence the right things and the right ways became inconsistent and subjective, and so did our success. I wrote this series to fix that.

Read more

XeTeX on Windows

I switched from Ubuntu to Windows 10 a few days ago. After eight years of macOS and then a year of Linux, I wanted to circle back to Windows.1 Since my latest writing project relies on XeTeX, a lesser-known typesetting engine within the already obscure TeX world, I expected some trouble recreating my workflow in Windows; I was not disappointed.2 This post explains the steps I went through to create a functional TeX writing environment in Windows 10.

Read more

Command versus Leadership

The Army has a strange relationship with words. On the one hand, precision means everything: the term “seize” means something very different to a ground forces commander than the term “secure”. On the other hand, “leader” has become an umbrella term for both the entire force in general and the select few in command positions specifically. This has little impact on non-commissioned and warrant officers who — for the most part — will never command a unit; “leadership”, to them, means leadership: influence to accomplish a mission. For officers, however, the impact of this imprecision is significant.

Read more

Bookmarks

I frequently encounter people eager to learn or interested in understanding current events but that do not know where to start. Many lack the experience to know what they should study. Others have become so disillusioned by the hyper-partisan twenty-four-hour news cycle that they just ignore current events altogether. While I have addressed those challenges in Personal Development and the two-part series Keeping Up with Current Events and My Evening Reads, respectively, those aggressively curated lists leave out many useful sources that did not make the cut. I just don’t have time to keep up with everything. This article highlights those sources as well as several other valuable resources. While I may not visit these websites often, these are my first stops when I must seek out trustworthy information on a range of topics. This article also links to several useful tools both as a way for me to keep track of them and as a way to highlight them to others.

Read more

Threat Intelligence & Threat Research

Network defenders often struggle to obtain timely, accurate, relevant, predictive, and tailored intelligence about the enemy and other aspects of the operational environment. This document attempts to address that difficulty by detailing a single, curated list of reputable, open sources of threat data, information, and intelligence. This list combines traditional sources such as cybersecurity organizations that publish detailed reports (think finished intelligence) with non-traditional ones like social media that provide feeds more akin to information or raw data. While many others exist, I tried to stick to those with a high signal to noise ratio. That ruled out some decent sources, but I consider that a worthy trade off. Keeping up with this entire list would require a full-time job as-is, so while you may choose differently, choose wisely.

Read more

Advice to New Lieutenants

As I close in on my promotion to captain, mentorship has become an increasingly important part of my job. Many new lieutenants find the Basic Officer Leader Course just that — basic — and so they report eager to develop personally, professionally, and militarily. My article Personal Development addresses that first domain in terms of improving one’s personal, professional, and military knowledge, but neglects other areas. After several conversations with new lieutenants eager for professional development as officers and military development as warfighters, I decided to turn some of those discussion points into a post here. As a Cyberspace Operations Officer, most of this advice will target other 17-series soldiers, although some of it may help new officers in other branches. This advice is also not specific to officers, and in some cases may apply to enlisted soldiers as well.

Read more

Your First Week as a New Lieutenant

Aside from maintaining my article on personal development, I have shared little about this profession of arms. As I close in on my promotion to captain, though, mentorship has become an increasingly important part of my job. After several similar conversations with new officers, I decided to answer a common question here: “What do I do as a new lieutenant?” While the answer to that question will change from branch to branch and from unit to unit, I decided to write about some advice that should generalize well. If you follow this advice during your first week, you will put yourself in a good spot at the beginning of your assignment and set you up for success in the long run.

Read more

Build Systems, and Habits

If I like an article, I look into its author. Most good writers write well and often. This helps me find independent websites, like Ben Kuhn’s blog. In his 2018 piece, Stop Trying, Ben talks about steps he took to better his life: tracking time with RescueTime, and waking up with automatic lights. His productivity went up, and he gets out of bed on time now — because these changes required “literally zero maintenance”. He tried to make productivity and wakefulness a habit, but did not succeed until he built systems around those two goals. From Stop Trying:

Read more

Staying Focused with a Simple Method

Leo Babauta, in Staying Focused with a Simple Method:

“When you notice yourself avoiding something hard or uncertain ... the method is to turn towards it. Turn towards what you’re avoiding. Open to the discomfort, embrace it as training and growth. Bring curiosity. Do it even when you don’t feel like it. This is the training. The simple method makes it easier. Take it on, and see what happens.”

As I have said before, and will keep on saying, there is no shortcut, life hack, or trick; grind.

Read more

"Be yourself" is terrible advice

One of my favorite mentors used to tell me this: “Be yourself, unless yourself is a shitbag; then be someone else.”

Read more

Attribution

Not long after I found an old piece on attribution from Thinkst Applied Research, Cisco’s Talos Intelligence group posted an article on the same topic. Attribution is fraught, as both pieces explain. IP geolocation is woefully inaccurate, many of the tactics, techniques, and procedures tied to specific actors are either so common that they apply to several groups or so general that they might as well, and the proliferation of viable tool sets has made even this metric unreliable. At best, we can attribute activity with a degree of certainty — but never complete confidence, and you should be circumspect of anyone who does.

Read more

Showing Up Even When You're Not Feeling It

Leo Babauta, in Showing Up Even When You’re Not Feeling It:

“Most of us have an expectation that we should feel in the mood to do something. We should be excited, rested, focused. And when we do it, it should be easy, comfortable, fun, pleasurable. Something like that. That results, predictably, in running from the things that feel hard, overwhelming, uncomfortable.”

You will not always feel like taking the hard road, but achieving success means defining a goal, making a plan to accomplish it, and then applying unmitigated daily discipline until you get there. There is no shortcut, life hack, or trick; grind.

Read more

Reshoring Supply Chains: A Practical Policy Agenda

Outsourcing has put America in a precarious position. It quietly became one of the greatest threats to national security over the course of several decades, unrealized until the Coronavirus pan(dem)ic revealed the fragility of our situation. David Adler and Dan Breznitz, in the American Affairs Journal, discuss the issue and offer some concrete suggestions for fixing it. I encourage you to give it a read. See also: On National and Enterprise Outsourcing, and Out-Sourced Profits: The Cornerstone of Successful Subcontracting.

Read more

Personal Development

A smart person knows everything about a single topic. An intelligent person knows as much as possible about more than one topic. Both have great value, and depending on the industry, some organizations value one more than the other. In general, though, smart people fill entry-level jobs, who then become intelligent people to move up the ladder1. A robust personal development strategy will help you go from the former to the latter.

Read more