t-prot
 by jeff covey, in Project Reviews - Sun, Jun 27th 2004 00:00 UTC

It's bad enough having to put up with our own bad habits. What can we do about others who can't or won't change their annoying ways? One solution is to learn patience. This takes a long time and is not at all fun. Another is to quietly reorder the world the way we like it when they're not looking. We secretly reseat the toilet paper roll so the flap goes over instead of under. We turn all the coat hangers to face the same way. We put the spices on the rack the way they belong. t-prot is another tool in this arsenal. It makes the ragtag email we receive look like the good and proper email we send.

The Internet has grown large enough that there's little hope of forcing all its users to love, honor, and obey guidelines such as these. Every new mail user agent seems to default to quoting the entire message to which the author is replying (which usually quotes the previous one and the previous one and the previous one). In part, this is a blessing, since otherwise nothing would ever be quoted, and we'd never have any idea what "me" was "tooing". It also means, however, that a large portion of the messages that hit our inboxes contain 273 quoted lines with a couple of original ones at the top or bottom.

Many people have no idea they're doing this and will never be computer-literate enough to delete the quoted text or prune it to just the relevant parts. Others are fully aware of what they're doing but simply don't care. What can we do? My solution for people who send HTML messages, MS Word files, and the like was not to transform everyone in the world into a reasonable person, but to write scripts that convert their messages into plain text ones that look like every other message. t-prot lets me do the same for quoting.

Let's see an example:

(I apologize to the members of the lilypond-user mailing list that the examples here come from their posts. Unfortunately, as I watched my mail for messages that would be good for this article, the best ones all came from there. :)

Now, what can we make of this message? It's a screenful of a quotation of a message we have presumably already read. We're seeing 28% of it. Is there something new at the bottom? In the middle, interspersed among more quoting? Or did the author just hit "Reply" and "Send" by mistake, without adding anything? One way to find out is to hit pagedown, pagedown, pagedown. Another is to let t-prot do its magic:

Voilà! The whole message fits on one screen, and we can see all we need to see as soon as we open it.

Here's how it works:

t-prot is simply a filter; pipe a message through it, and more palatable results are produced. It can:

1. Remove mailing list footers.
2. Shrink big quotations.
3. Squeeze a sequence of blank lines to just two. (Useful for dealing with the cute people who hide the punch line of a joke with "Are you ready?...", [blank page, blank page, blank page], "Here it comes...", [blank page, blank page, blank page], ...)
4. Change a series of four or more periods or exclamation points into just three. ("Hey there!!!!!!!!!!!!!!!!!" becomes "Hey there!!!".)
5. Move PGP/GPG verification output to the bottom of the message (optionally only if verification shows a good signature and the signature could be verified as authentic).
6. Hide overlong signatures.
7. Hide TOFU.

"TOFU"? The manual page explains it this way:

TOFU is an abbreviation which mixes German and English words; it expands to "text oben, full-quote unten" which means "text above - full quote below" and describes the style of so many users who let their mailer or newsreader quote everything of the previous message and just add some text at the top; obviously they think that quoted text must not be changed at all.

So "t-prot" provides "TOFU Protection".

Some more examples:

This time, we have some new text at the top with a huge quote below. That's probably all there is to the message, right? But can you be sure? What if he's put tomorrow's winning lottery numbers at the bottom? You'll never be satisfied until you've scrolled down (and been disappointed).

But if you were using t-prot, you'd see this:

Here's another six-line message...

... from which t-prot strips a 67-line quote and a 15-line (!) signature:

One more. Is there anything here?:

Yes, indeed. t-prot squeezes out the extras on both sides and brings us the gold hidden in the middle:

t-prot is well-documented and very customizable. You can control which annoyances you want it to handle, how much quoting is too much, etc.

If you haven't already, imagine how much more quickly you can go through the day's mail from a list if only the important parts of a message are shown. Run on over to Jochen Striepe's site, give t-prot a try, and give him your thanks.

By the way...

You may have guessed that the MUA in the above screenshots is mutt. t-prot is simply a filter and can be used with any email or Usenet software, but it was originally written with mutt in mind and has some features which make it especially nice when combined with mutt. So, as long as we're talking about mutt, let me share a few favorite mutt tips:

1. Putting this in your ~/.muttrc:

       
         set tilde
       
     

   will put the vi-like "~"s at the bottom of each message, as in the screenshots. This lets you be absolutely sure you're seeing the very bottom of the message.

2. Adding this:

       
         set smart_wrap
         set wrapmargin=10
         unset markers
       
     

   will wrap long lines to fit the screen. This is a real dream-come-true for reading messages from MUAs like Outlook which sometimes put each paragraph of a message on a single line (*boggle*).

3. Adding this:

       alternative_order text/plain text/enriched text
     

   sets your preference for viewing messages with the Content-Type "multipart/alternative". These are often messages which, for reasons best known to MUA authors, contain the same text twice, once in plain text and once in HTML. You've probably configured mutt to pipe HTML messages through an HTML-to-text filter, but by setting this, you get the plain text version right away, and don't have to wait for the HTML to be processed. An added bonus of this is that a lot of spam puts the advertisement in the HTML attachment and nonsense in the plain text one, so when you open a message and see 300 random words, you can delete it right away without having waited for the Viagrra message to be converted.

4. The t-prot sample muttrc file adds keybindings to turn t-prot filtering on and off. You can also hit "v" to view a list of the MIME attachments to a message and see any of them in their raw state.
5. Pressing "T" will hide quoted text in a message. This will often compress what seems to be a long message into a single screenful.
6. t-prot can remove mailing list footers, but this requires saving them in a directory and doesn't help if their content changes each time. I prefer piping my mail through Phil!'s display_filter.sh, then piping the results to t-prot. display_filter.sh is a simple sed script which has example regular expressions for deleting Yahoo! Groups advertisements, the extremely long signatures of certain people and lists, etc. Please bug Phil! for a copy or to put it on his homepage.

If you have any other mutt goodness to share, please add it to the comments below.

Author's bio:

jeff covey no longer has a cat on his shoulder.

T-Shirts and Fame!

We're eager to find people interested in writing articles on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an article gets a t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.

[Comments are disabled]

Referenced categories Topic :: Communications :: Email :: Filters

Referenced projects Mutt - A small but very powerful text-based mail client. SLRN - An NNTP based newsreader for Unix, VMS, and OS/2 systems t-prot - A filter which improves the readability of email messages and Usenet posts.

 Comments

[»] Old ways...
by David BL - Sep 18th 2004 18:59:45

I like programms like t-prot, it's a shame theres ain't more of em out there. In fact they should be added to some mail/news clients as a default snip'er.

If we look at the old ways on how to behave on a maillist or on a news server etc then theres a lot of old habbits that have been lost in the evolution heh. I mean iv'e always been userminded and manual removed long reply text and added a --snip-- or something like that. It's just being friendly for the other users who might find your text interresting.

I don't use t-prot myself, must admit that ;) But im also used to the old ways and it's something i can't get rid off that easy. Most of the things t-prot does, I do it without blinking. Ofcourse i can't do it as professionel as t-prot or as fast 8) That's why iv'e decided to try it out.

Thanks for a nice article.

--
z/OS System Programmer David BL Denmark -- ------------------------ ---- ---- -- -

[reply] [top]

[»] mutt-display-filter
by Phil! Gregory - Jun 28th 2004 16:47:56

My display filter may be found at http://aperiodic.net/phil/configs/bin/mutt-display-filter.

Though my mutt colors are done in a very different style than jeff's, you can see my setup at http://aperiodic.net/phil/configs/mutt/colors.

--
355/113 -- Not the famous irrational number PI, but an incredible simulation!

[reply] [top]

[»] Re: mutt-display-filter
by jeff covey - Jul 5th 2004 12:32:16


> My display filter may be found at
> http://aperiodic.net/phil/configs/bin/mutt-display-filter.

Thanks for sharing, Phil!! :-)

--
vs lbh pna ernq guvf, lbh'er n trrx.

[reply] [top]

---

[»] Cat on Shoulder?
by BozMo - Jul 23rd 2004 02:46:48

Could someone kindly explain about the cat? BozMo

--
BozMo

[reply] [top]

---

[»] Re: Cat on Shoulder?
by jeff covey - Jul 23rd 2004 04:52:34


> Could someone kindly explain about the cat?

She got tired, and hopped down.

--
vs lbh pna ernq guvf, lbh'er n trrx.

[reply] [top]

---

[»] Re: mutt-display-filter
by Aristotle Pagaltzis - Jul 25th 2004 15:53:25


> My display filter may be found at

> http://aperiodic.net/phil/configs/bin/mutt-display-filter.


Just FYI, since it's just a sed script, you can call it right from the shebang line.

So instead of

> #!/bin/sh

> sed -e '

> ...

> '


you just make it

> #!/usr/bin/sed -f

> ...


The same can be done with awk scripts, makefiles, and every other tool that uses the hash as a comment marker.

[reply] [top]

---

[»] Nice tool.
by Neil Mansilla - Jun 27th 2004 12:28:22

Google Mail has a nice way of stripping down quoted segments and just getting to the meat of replies. I haven't tried to tear it apart to see how they've engineered what constitutes as meat, modified meat, or duplicated meat, but it definitely reminds me of something I'm seeing here with t-prot. I look forward to trying it. Thanks.

[reply] [top]

[»] Mutt colors?
by random - Jun 27th 2004 07:50:37

I like your mutt colors setup. Would you care to give us your .muttrc settings?

[reply] [top]

[»] Re: Mutt colors?
by jeff covey - Jul 5th 2004 12:29:17


> I like your mutt colors setup. Would you
> care to give us your .muttrc settings?

Here are the color parts:

http://jeffcovey.net/tmp/muttcolors.html

Sincerely,
Jeff

--
vs lbh pna ernq guvf, lbh'er n trrx.

[reply] [top]

---