A Tutorial On Redaction

This image has been redacted for your convenience.

Writing software is fun. Redacting software is torturous. It gets worse when the software was never intended for redaction, but you need to redact it anyway.

I’ve worked on multiple projects in the past decade and all of them have involved redaction at some level. The source code, the documentation, the bug fix requests; I’ve redacted every type of digital thing you can think of. Sadly, even after all this time, I’ve yet to find a way to remove the vampiric qualities of redaction which consume the souls of those who perform it. However, I have learned a few ways to make redaction more effective with less time-consuming rework, and that’s what this post is all about.

Redaction, if you are uninitiated into the cult, is a form of evil magic where you remove sections of important information from documents or source code all while somehow retaining most of your sanity. Sometimes, the information is removed because you aren’t licensed to give away someone else’s work, but you need to deliver something that contains the work. Other times, you want to protect your own inventions but still be able to sell portions of your work. You might be allowed to make redaction obvious, printing black boxes where text should be. Or maybe you’ll need to completely hide the fact you messed with the content. This latter approach is the one I’ll assume, because I don’t think there’s a whole lot of value in the former.

What’s the Point?

Stop redacting for a second. It’s easy to jump into redaction work and go through some easy, repeatable steps to get your job done and end up missing the whole point of redaction. Remember, the reason you are redacting is because whoever is receiving the information you have can’t have specific stuff it contains.

Are you redacting to remove terms? Maybe the names of intellectual properties? If that’s all, you might think you can search and replace the contents of the files you need to redact. Replacing terms is easy enough. You can probably finish your redaction work in a few minutes. But what value is there in removing terms? If the people who are being provided the redacted material have any idea what the material was used for, they’ll know which terms were redacted. You haven’t done anything. And if they have no idea what it was used for, why do they care about it?

I’ve found that redaction is time-consuming and tedious, but also an inconsistent process. You can’t write a program to perform redaction for you, because a program can’t interpret every conceivable spelling error, phrasing (especially poor English), or acronym. Searching for terms with software is really helpful, but it only catches the most obvious stuff.

Consider this paragraph:

“The software uses a proprietary component called This Sure Is Awesome Technology. This technology is used to generate output in a comma-separated list, but in columns instead of rows. This is protected by patent Des. 555,555. Our tool uses this tool to turn pictures of ducks into pictures of chickens. Chickens are better than ducks.”

Suppose you can’t transfer This Sure Is Awesome Technology, because it is licensed. And suppose you can’t transfer the patent information because of international law. A search for the relevant terms would get you what you want, but try just removing them:

“The software uses a proprietary component called. This technology is used to generate output in a comma-separated list, but in columns instead of rows. This is protected by patent. Our tool uses this tool to turn pictures of ducks into pictures of chickens. Chickens are better than ducks.”

You’ve left in the nature of the technology in question and the content of the patent. It doesn’t take much effort for someone to replace your redacted text. So what was the point? You need to do this by hand:

“The software uses a proprietary component which translates files from one type to another. Our tool uses this tool to turn pictures of ducks into pictures of chickens. Chickens are better than ducks.”

Here, the meaning is preserved, but not the method, which is the focus of the redaction. Redaction is almost always an effort to protect methods and concepts, so why apply a method that can’t protect anything?

Search, however, is limited. Consider a different writing of the same text:

“The software uses a proprietary component called TSiAT. This technology is used makes row-based CSVs. This is protected by PD 555,556. Our tool uses this tool to turn pictures of ducks into pictures of chickens. Chickens are better than ducks.”

Your search won’t catch every possible spelling error. It won’t catch different forms of the same phrases, especially if those forms have poor grammar. It won’t catch acronyms you don’t expect. If you understand the point of redaction, you won’t consider your job complete just because a search doesn’t return terms from a list of “bad” words.

The Redaction Balance

It’s easy not to go far enough in your redaction and leave too much content behind. On the other hand, redacting massive sections of documents removes any value from them. At that point, why even give the documents away?

You’ve probably seen documents redacted by the government. You know, those poorly scanned pages that have a handful of words floating in a sea of black ink. You might find pieces of information here and there but you might not. Why did they release the document at all if there’s nothing in it?

A better approach, as described above, is to search for terms and concepts  yourself. You, a human being and not a computer, can understand practically everything that might show up in the information you are redacting. It’s tedious, it’s terrible, and it might be evil, but redaction is something you can’t describe in logical terms any more than you can describe writing a book in logical terms. You can’t delegate this terrible work to a computer, no matter how much you want to. And if you are doing redaction the right way, you’ll really, really want to.

Your goal is to remove terms and ideas in a careful way that doesn’t make it obvious that the terms and ideas ever existed. For instance, if you need to remove the section in brackets, do it like this:

“The tool is capable of [feature A], which does X, Y, and [Z] in order, as well as feature B, which does X and Y only.”


“The tool is capable of feature B, which does X and Y in order.”

A hard redaction of this, replacing [feature A] with [redacted feature] and [Z] with [redacted function] would read like this:

“The tool is capable of redacted feature, which does X, Y, and redacted function in order, as well as feature B, which does X and Y only.”

This gives away the fact a feature exists as well as 66% of what it does. If you want someone to know about “Feature B” and not “Feature A”, this is a terrible way to do it.

Acronyms Are Your Enemy

If a term you are replacing is an acronym, things get much worse.

Imagine you have an acronym like RED. That term might appear thousands of times in unrelated words: hundred, redaction, credibility, bred, not to mention the word “red” itself. If this term is just replaced forcibly with something like “Supplier Technology”, you end up with ridiculous sentences like:

There are two-hundSupplier Technology tests, each of which appear in black if they passed and Supplier Technology if they failed, establishing the cSupplier Technologyibility of the claim that the software was tested.”

You’d need to go through these results by hand, which is no faster than searching and reading in the first place. If you left this in place, anyone reading the redacted document would realize that “Supplier Technology” is clearly what all instances of “RED” become. Again, search-and-replace has accomplished nothing.

And this isn’t the end of the pain you will suffer at the hands of linguistic shortcuts. Laziness compels people to turn all sorts of things into acronyms where you may not expect them. And even if you try and expect them, they probably won’t use the same letters you would. This doesn’t even take spelling errors into account. A misspelled acronym is like a land mine of important information you can’t sweep for. It’s just waiting for the recipient of the redacted material to trip on, blowing up in your face. Acronyms are just another reason you should be performing redaction by hand.

Some Precautions

You may have no idea if the documents or software you are writing will be subject to redaction in the future. But if you somehow do know, there are a few things you should keep in mind.

If you want to make redaction trivial, don’t mix different proprietary information. If you are working with three companies, try to keep the IP of each of them separate, restricting interaction to as few documents as possible. You’ll find that this won’t be possible, at least completely. The closer you get to it, however, the easier the redaction.

If you want to make redaction impossibly difficult, use extremely short proprietary terms. For instance, two-character terms like “A1” will show up in binary data in guids, maybe millions of times. Even if an engineer can look through things at a rate of 10/second (which is practically begging for human error), that’s still four full terrible days to look for a term which might legitimately appear twice in its proprietary context. Inconsistent acronyms, lack of spelling and grammatical checks, and images all help multiply the length of time you will need to perform redaction. Avoid these things as much as possible in any material that might be redacted in the future.

Life Organization Part 3 – Yearly Goals

Note: This is in a series of posts, and as the others are written, I’ll update a table of contents with links to the whole series here.

Part 1: What is Life Organization? Why do it?
Part 2: Life Goals
Part 3: Yearly Goals

Having some long-term goals is important, but without a plan to achieve them, they always remain in the distance without moving closer. In the previous article in this series, we looked at the method I’ve used to lay out these larger goals. Now, we’ll break them down into more manageable chunks.

Why Yearly?

A lot of people come up with New Years resolutions for themselves, finding that changing over to a new year and coming back to work after some major holidays is a great point to set up new habits. It’s convenient, but not always effective. Many people who seem to have a lot of success with their resolutions in January have completely forgotten them by May.

Years are still a great length of time to plan. They are finite and fixed, but still long enough that you can get a lot done towards whatever longer term goals you have.

I recommend not waiting until New Years Day to decide what you want to do for the upcoming year, though it doesn’t hurt to read through your plans that day.

Creating Yearly Goals

There’s some overlap here with lifelong goals, as there probably ought to be. After all, if you are going to be accomplishing something, it needs to be accomplished at a particular point in time. To make part of a larger goal into a yearly goal isn’t to say you have all year to achieve it (though you may). It simply means that it will be accomplished at some point during the year. I’ll have some thoughts about how to help make this more likely in the next post on planning a year. For now, the focus is on the goals themselves.

The first step I take is to look at my lifelong goals. Are there any I can do this year? Are there any parts of a goal I could contribute to?

Once I have those things identified, I try to balance each of the seven categories (see the previous article) of goals for the year. After that, I break down the goals even further, trying to figure out what major steps need to be accomplished to get the whole thing completed. Sometimes this isn’t necessary. Other times it can’t be helped.

For example, one of my perennial goals is to be on time to and have a good attitude about all of my commitments with a good attitude. I made them, after all. There’s no point in breaking this down; it’s pretty straightforward and uncomplicated, even if difficult at times.

On the other hand, one of my lifelong goals is to learn new things. This is ambiguous, so I’ve broken it down even at a long-term level into categories like “learn Latin” and “learn Statistics”. I also want to consistently read (and truly understand) a lot of books. At a yearly level, this breaks down into a list of the books I actually want to read. This means the yearly goal might fluctuate a little bit as I discover new books or decide not to read a book in the end, but this is infrequent.

The Next Step

After you have all of your yearly goals listed out, you should find yourself with a list that looks a lot like your lifelong goals, but less grand in scale and less comprehensive. In the next article, we’ll look at what to do with this concrete set of goals and how to plan your year out to actually get them done.

Life Organization Part 2 – Life Goals

Note: This is in a series of posts, and as the others are written, I’ll update a table of contents with links to the whole series here.

Part 1: What is Life Organization? Why do it?
Part 2: Life Goals
Part 3: Yearly Goals

In the first part of this series, we covered the meaning of life organization – as far as I use the term – and looked at what constitutes a goal. This time, we’ll focus on the sorts of lifelong goals everyone should be thinking about.

The Seven Types of Goals

You’ve probably heard of Dave Ramsey and recognize him for the popular financial advice he gives. While his work in finance is great on its own, I mention him here because I think he’s created an eminently useful breakdown of the sorts of lifelong goals everyone should have.

Dave list seven categories which broadly cover anything you can think of, and which he labels the “Wheel of Life”:

  1. Career
  2. Financial
  3. Spiritual
  4. Physical
  5. Intellectual
  6. Family
  7. Social

There are two things to keep in mind with these categories. First, it is important to have goals in every category. These may change over time, but you should still have a long-term plan to grow in every one of the seven respects.

Second, there must be a healthy balance between each of these focuses. This balance is not merely spending the appropriate amounts of time on each category, although it is not less than that either. Balance also includes how we prioritize them, and which we are willing to postpone and which we are not.

You need goals in each of the categories. Even better, you need a short but unambiguous description of where you want to be in a few decades (depending on your age). Do you want to be married with at least three children, all of whom you spend time with and for whom you’ve built a home? That’s both short and unambiguous. You can’t fudge the number of kids you have or the presence or absence of a home.

In 7 Habits of Highly Effective People, Stephen Covey talks about the need in everyone’s life for a “personal constitution”. A good format for that is to create the aforementioned summaries in the seven categories and write them down.

The purpose of doing all of this is to figure out the destination you are trying to arrive at. It isn’t a prerequisite to living a great life or doing important things, but I think for most people, it increases the chances of doing either of those things.

How to Find Life Goals

None of the life management gurus I’ve read has ever said this – probably in some cases to preserve an audience – but I don’t think everyone is naturally prepared to figure out what their own life goals should be. Until you figure out what matters most in life and what the purpose of life is in the first place, it’s all a waste of time.

If you want to figure out what your life goals ought to be, you need to ask some more fundamental questions first: Why are we here? Where do we go when we die? How do I know the difference between right and wrong? What does it mean to live well?

Providing thorough answers to these questions is not the topic of this post (but it may be for future posts). The short answers are:

  • We are here to worship God in fellowship with Him, and this can be done only through Jesus.
  • Where we go depends on whether we trust Jesus for our salvation. Whether we trust Jesus for our salvation depends in part on whether we think we need salvation (we do).
  • We can know right from wrong through both special and general revelation; from both Scripture and through nature and natural law.
  • To live well is to live in accordance with our nature as being created in the image of God. To live well is to pursue God in all things, from eating, exercise, and reading to worship and charity.

Only by having correct answers to these questions can we begin to know what sort of life goals we should create. That doesn’t mean it becomes easy to figure out what you should pursue in the seven categories. It means it is possible to make the right choices.

It will take time.

Examples of Life Goals

I’ve spent about a decade trying to refine my life goals, and I can offer up a few of them as examples, as well as describe the way I keep track of them.

I use Microsoft’s OneNote for a lot, but one of the original things I did with it was keep track of goals. It’s still the primary reason I use it. I’ve created a notebook called “Goals” where I keep track of my lifelong goals. I keep track of yearly goals, notes and articles related to the process of setting good goals, and checklists that help me plan my days, weeks, months, and years, but those are all for a later article in this series.

Part of my section for “intellectual goals” looks a bit like this:

Intellectual Success

  • Learn New Things
    • Learn how to write classical genres of music
      • Learn how to write fugues
      • Learn how to write chorales
    • Learn Latin
      • Finish reading through and doing the homework in my Latin textbook
  • Learn Piano
    • Practice multiple times a week
    • Learn individual pieces
      • Learn Kansas’ “Point of Know Return”

The section is larger than this, but it’s all the same format. I have the category (“Intellectual success”) broken down into specific goals. Each of those is broken down further as necessary.

For me, it was easier to list more things I might want to do than I’d ever have time for and then prune the things that weren’t as important to me. You’ll have to find a way that works for you.

Once you’ve got your goals listed at this high level, you’re ready for the next step: figuring out how to contribute to each one this year. But that’s for next time!


Life Organization Part 1 – What is it? Why do it?

Note: This is the first in a series of posts, and as the others are written, I’ll update a table of contents with links to the whole series here.

Part 1: What is Life Organization? Why do it?
Part 2: Life Goals
Part 3: Yearly Goals

For the first 20 years of my life, I didn’t make much of an effort to keep myself organized. My parents did most of the work and were better at it, so why even bother?

As I completed college, however, I realized that I would just drift through life if I didn’t sit down and figure out what my goals were and how to achieve them. This began as it probably does for a lot of people: I had really lofty dreams and I wanted to make them reality. Over several years, I read hundreds of articles and several books on the topic of life management and organization, and I spent a lot of time trying to figure out the ideal system.

After getting married, and especially as my wife and I began to have children, I changed my approach completely. This post is the first in an intended series of what my new method is, and why I think it works better.

What is “Life Organization”?

Planners and calendars are more common than people in the Western world. Everyone who has a smartphone has at least one (I’m not among this group just yet, but I’m told by my wife that I am an endangered species). But a planner or calendar alone is not the whole crux of what this term means.

Admittedly, “life organization” sounds too simple and too simplistic. Life, in truth, cannot be organized comprehensively. Nor should it be.

For our purposes, the term “life management” means the tools, methods, and behaviors a person uses to keep track of and accomplish their long-term goals. It involves figuring out what your own goals are, creating a plan to achieve them, and learning to stick by your plan even when you don’t want to.

Dreams vs Goals

If you’ve ever attended a high school graduation, you will have heard teachers and students alike speak of the virtues of “pursuing your dreams”.

But what if your dreams are bad? What if they are truly impossible? What if the risk of failure isn’t worth the pursuit?

These are the sorts of questions you are not allowed to ask, but which are important to answer. They can change the whole course of a life.

Goals are a bit different than dreams. Instead of being esoteric and ambiguous, goals must be concrete. This is best demonstrated by way of example:

Dream: I want to be important.
Goal: I want to own my own company.

Even in this example, the goal could use a lot of definition. What sort of company? What products or services will be produced? Will you simply have a managerial role in the company? Is this a company you will start on your own or acquire from someone else? By what date does this need to be accomplished?

These questions may seem to suck the fun out of making goals, but in truth, they infuse the goals with purpose, realistic constraint, and urgency. Without asking and answering such questions, you’ll never achieve the goals except by accident. I wouldn’t want to plan on my greatest accomplishments being achieved by accident. I don’t think you’d want to do that either.

An even better example is this:

Dream: I want to be a famous musician.
Goal: Within ten years, I want to play piano with the proficiency of a concert pianist and I want to be performing in city-level orchestras.

This goal is lofty, but achievable. It has a time limit to give it some urgency and to help you plan year-by-year what you need to do to achieve it. It has concrete objectives (playing in a concert is a real and tangible thing; having a level of proficiency can be further expanded to listing pieces of music that, when performed without error, prove it).

Why spend the time doing it?

You don’t have an infinite amount of time in this life. If you want to make the most of it, you need to know what “making the most of it” means. What are the greatest things you can do and how can you do them? This is the heart of life organization.

Planning too much is just as bad as not planning enough, and most of us are guilty of doing one or the other. Everyone needs to plan and organize their time effectively, but not waste time in the process. There’s no secrete formula here, and it will be a lot of hard, irritating, headache-inducing work.

In the coming weeks, I plan to continue this series by discussing what sort of life goals you ought to establish, how to break those goals up yearly, how to use a planner and calendar to accomplish the tasks required, how to cultivate the behaviors and rituals you will need to succeed at them, and anything else I can think of that’s on-topic.