[maildropl] error writing to filter (bogofilter)

[Christian Ebert]

Hi,

After an upgrade to Mac OS 10.8.4 (64bit) from 10.5.8 (32bit) I
get the following error for _some_ messages:

temporary failure.
Command output: maildrop: error writing to filter.
/usr/local/bin/maildrop: Unable to filter message.

The offending rule is rather bog standard:
xfilter "/usr/local/bin/bogofilter -u -e -p"

Trying the bogofilter suspect on an offending message however
seems to cause no problems:

$ bogofilter -u -e -p < testbogomsg 1>/dev/null
$ echo $?
0

Of course I recompiled all involved programs, like maildrop,
bogofilter, tokyocabinet.

What could be the problem?

[Sam Varshavchik]

Post by Christian Ebert
Hi,
After an upgrade to Mac OS 10.8.4 (64bit) from 10.5.8 (32bit) I
temporary failure.
Command output: maildrop: error writing to filter.
/usr/local/bin/maildrop: Unable to filter message.
xfilter "/usr/local/bin/bogofilter -u -e -p"
Trying the bogofilter suspect on an offending message however
$ bogofilter -u -e -p < testbogomsg 1>/dev/null
$ echo $?
0
Of course I recompiled all involved programs, like maildrop,
bogofilter, tokyocabinet.
What could be the problem?

bogofilter does not read the entire contents of the message from its
standard input. It terminates before it is read in its entirety. xfilter
pipes the message to the external filter program that it runs. If xfilter
detects that the pipe is broken, before it's finished writing the entire
message to the pipe, it concludes that the external filter has crashed.

Given the buffering involved with pipes, you get non-predictable behavior
that, sometimes, maildrop succesfully finishes writing the end of the
message, which gets buffered in the pipe before the external command
terminates without reading in its entirety. There's no way to detect that,
so, depending on random factors, sometimes an error gets detected, sometimes
not; and this depends in part on the underlying operating system platform
and implementation. It's not surprising that different behavior is observed
after upgrading.

[Christian Ebert]

* Sam Varshavchik on Thursday, June 13, 2013 at 08:06:13 -0400

Post by Sam Varshavchik

Post by Christian Ebert
After an upgrade to Mac OS 10.8.4 (64bit) from 10.5.8 (32bit) I
temporary failure.
Command output: maildrop: error writing to filter.
/usr/local/bin/maildrop: Unable to filter message.
xfilter "/usr/local/bin/bogofilter -u -e -p"
Trying the bogofilter suspect on an offending message however
$ bogofilter -u -e -p < testbogomsg 1>/dev/null
$ echo $?
0
Of course I recompiled all involved programs, like maildrop,
bogofilter, tokyocabinet.
What could be the problem?

bogofilter does not read the entire contents of the message from its
standard input. It terminates before it is read in its entirety.
xfilter pipes the message to the external filter program that it runs.
If xfilter detects that the pipe is broken, before it's finished
writing the entire message to the pipe, it concludes that the external
filter has crashed.
Given the buffering involved with pipes, you get non-predictable
behavior that, sometimes, maildrop succesfully finishes writing the
end of the message, which gets buffered in the pipe before the
external command terminates without reading in its entirety. There's
no way to detect that, so, depending on random factors, sometimes an
error gets detected, sometimes not; and this depends in part on the
underlying operating system platform and implementation. It's not
surprising that different behavior is observed after upgrading.

Thanks for the explanation. However, it's not completely random,
it is replicable with certain messages.

What shall I do? Take it to the bogofilter list? Switch back to
procmail - it has no trouble with exactly the same message(s)?

[Christian Ebert]

* Christian Ebert on Thursday, June 13, 2013 at 14:04:24 +0100

Post by Christian Ebert
* Sam Varshavchik on Thursday, June 13, 2013 at 08:06:13 -0400

Post by Sam Varshavchik

Post by Christian Ebert
After an upgrade to Mac OS 10.8.4 (64bit) from 10.5.8 (32bit) I
temporary failure.
Command output: maildrop: error writing to filter.
/usr/local/bin/maildrop: Unable to filter message.
xfilter "/usr/local/bin/bogofilter -u -e -p"
Trying the bogofilter suspect on an offending message however
$ bogofilter -u -e -p < testbogomsg 1>/dev/null
$ echo $?
0
Of course I recompiled all involved programs, like maildrop,
bogofilter, tokyocabinet.
What could be the problem?

bogofilter does not read the entire contents of the message from its
standard input. It terminates before it is read in its entirety.
xfilter pipes the message to the external filter program that it runs.
If xfilter detects that the pipe is broken, before it's finished
writing the entire message to the pipe, it concludes that the external
filter has crashed.
Given the buffering involved with pipes, you get non-predictable
behavior that, sometimes, maildrop succesfully finishes writing the
end of the message, which gets buffered in the pipe before the
external command terminates without reading in its entirety. There's
no way to detect that, so, depending on random factors, sometimes an
error gets detected, sometimes not; and this depends in part on the
underlying operating system platform and implementation. It's not
surprising that different behavior is observed after upgrading.

Thanks for the explanation. However, it's not completely random,
it is replicable with certain messages.
What shall I do? Take it to the bogofilter list? Switch back to
procmail - it has no trouble with exactly the same message(s)?

Sorry for having sounded a bit harsh, but I'm a sorta desperate
here - I had taken my entire postfix config apart before I nailed
it down to this rather trivial cause.

Are you suggesting that the only way around this in practice is:
1. inspect the postfix queue
2. in case there is this problem, disable bogofilter by
temporarily changing the mailfilter
3. sendmail -q
4. reinstate bogofilter in the mailfilter

At least for me this is not even a workaround.

Would it help if I sent you one of the "offending" messages
via pm for inspection?

[Sam Varshavchik]

Post by Christian Ebert
* Christian Ebert on Thursday, June 13, 2013 at 14:04:24 +0100

Post by Christian Ebert
What shall I do? Take it to the bogofilter list? Switch back to
procmail - it has no trouble with exactly the same message(s)?

Sorry for having sounded a bit harsh, but I'm a sorta desperate
here - I had taken my entire postfix config apart before I nailed
it down to this rather trivial cause.
1. inspect the postfix queue
2. in case there is this problem, disable bogofilter by
temporarily changing the mailfilter
3. sendmail -q
4. reinstate bogofilter in the mailfilter
At least for me this is not even a workaround.
Would it help if I sent you one of the "offending" messages
via pm for inspection?

Looking at the message won't be very useful. There's nothing wrong with the
message itself, but rather with what bogofilter appears to be doing; namely
failing to read the message and terminating before doing that. It's unlikely
that one can tell why, just by looking at the message itself.

The only thing I know is that the whole purpose of the "xfilter" command is
to pass the contents of the message to "something external", which is
expected to read it, and create a new message that maildrop uses as a
replacement message for further filtering or delivery. Although I'm not
familiar with bogofilter, looking at its man page the -p option seems to be
doing just that.

But if that "something external" clearly terminates before it reads the
entire message, then something obviously went wrong. If whatever's
procmail's equivalent to "xfilter" is, ignores this, and blissfully carries
on, that's procmail's call, but I don't think that's a smart decision.

You should look at what bogofilter is printing, for the offending messages.
It can't be the original message, if bogofilter isn't even reading it.
Perhaps looking at bogofilter's output, for the message, will provide more
clues.

[Dimitri Maziuk]

Post by Sam Varshavchik
You should look at what bogofilter is printing, for the offending
messages. It can't be the original message, if bogofilter isn't even
reading it. Perhaps looking at bogofilter's output, for the message,
will provide more clues.

Well, in theory bogofilter could drop the pipe before reading the last
\n so its output would look fine...

Dima

[Sam Varshavchik]

Post by Dimitri Maziuk

Post by Sam Varshavchik
You should look at what bogofilter is printing, for the offending
messages. It can't be the original message, if bogofilter isn't even
reading it. Perhaps looking at bogofilter's output, for the message,
will provide more clues.

Well, in theory bogofilter could drop the pipe before reading the last
\n so its output would look fine...

bogofilter has no way of knowing it's the last \n until it reads everything,
and gets an EOF.

However, if the purpose of the -p option, as documented, would be to read
the entire message, do whatever bogofilter wants to do with it, which seems
to be just add some headers to it, and write it out, then it's reasonable to
expect that this is what it should do.

[Christian Ebert]

[ keeping you guys up to speed ]

See also:
http://www.bogofilter.org/pipermail/bogofilter/2013-June/009951.html
ff.

* Sam Varshavchik on Saturday, June 15, 2013 at 11:25:31 -0400

Post by Dimitri Maziuk

Post by Sam Varshavchik
You should look at what bogofilter is printing, for the offending
messages. It can't be the original message, if bogofilter isn't even
reading it. Perhaps looking at bogofilter's output, for the message,
will provide more clues.

~$ cat testbogomsg | bogofilter -u -e -p 1>/dev/null
~$ echo $?
0
~$ cat testbogomsg | bogofilter -u -e -p >| bogoout
~$ diff testbogomsg bogoout
94a95

X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3

$ maildrop < testbogomsg
maildrop: error writing to filter.
maildrop: Unable to filter message.

Post by Dimitri Maziuk
Well, in theory bogofilter could drop the pipe before reading the last
\n so its output would look fine...

bogofilter has no way of knowing it's the last \n until it reads
everything, and gets an EOF.
However, if the purpose of the -p option, as documented, would be to
read the entire message, do whatever bogofilter wants to do with it,
which seems to be just add some headers to it, and write it out, then
it's reasonable to expect that this is what it should do.

It seems to concern multipart only, and I'm starting to wonder
whether some temp file writing problem is involved.

[Dimitri Maziuk]

Post by Christian Ebert
[ keeping you guys up to speed ]

...

Post by Christian Ebert
It seems to concern multipart only, and I'm starting to wonder
whether some temp file writing problem is involved.

Now that I think about it this looks like what I've seen with
spamassassin and whatever version of maildrop shipped with Fedora 9. The
fix was to install an up to date version of maildrop.

[Christian Ebert]

* Dimitri Maziuk on Saturday, June 15, 2013 at 15:07:27 -0500

Post by Dimitri Maziuk

Post by Christian Ebert
[ keeping you guys up to speed ]

...

Post by Christian Ebert
It seems to concern multipart only, and I'm starting to wonder
whether some temp file writing problem is involved.

Now that I think about it this looks like what I've seen with
spamassassin and whatever version of maildrop shipped with Fedora 9. The
fix was to install an up to date version of maildrop.

maildrop 2.6.0 here, built from source. I believe that is up to
date.

Mac OS has "strange" tmp file locations under /var/folder/ but
that wasn't a problem in 10.5.8 - and other programs, including
my own scripts don't have a problem now with 10.8.4 either.

[Christian Ebert]

* Christian Ebert on Saturday, June 15, 2013 at 18:41:30 +0100

Post by Christian Ebert
It seems to concern multipart only, and I'm starting to wonder
whether some temp file writing problem is involved.

Replicable with all messages containing base64 encoded
attachments.

[Sam Varshavchik]

Post by Christian Ebert
* Christian Ebert on Saturday, June 15, 2013 at 18:41:30 +0100

Post by Christian Ebert
It seems to concern multipart only, and I'm starting to wonder
whether some temp file writing problem is involved.

Replicable with all messages containing base64 encoded
attachments.

In addition to the environment issues that were mentioned on the bogofilter
list, another difference between invoking bogofilter from the command line,
and invoking it via xfilter, is the process environment variables.

This is described more detail in the "Environment" section of the
maildropfilter man page, but, basically, the environment variables that are
inherited by processes started by maildrop, like the xfilter command, may or
may not include the environment variables that maildrop itself inherited –
depending on whether maildrop itself was invoked to deliver mail by the mail
server, or invoked from the command line; but since you appear to have a
reproducible test case, this does not appear to be a factor. However, in all
case maildrop resets some environment variables, and also any variables
defined in the maildrop recipe get exported as environment variables to the
maildrop-started process, as described in the man page.

I'll just guess that bogofilter does not decode base64-encoded content on
the fly, but tries to save them in a temporary file, and then reads it back;
and the temporary files locations are specified by environment variables;
which should be defined, but are not; or even the other way around;
bogofilter reads some environment variable that's not normally defined, and
then uses some fallback setting, but this environment variable is set here,
and bogofilter tries to use it.

[Christian Ebert]

* Sam Varshavchik on Saturday, June 15, 2013 at 16:55:04 -0400

Post by Sam Varshavchik
I'll just guess that bogofilter does not decode base64-encoded content
on the fly, but tries to save them in a temporary file, and then reads
it back; and the temporary files locations are specified by
environment variables; which should be defined, but are not; or even
the other way around; bogofilter reads some environment variable
that's not normally defined, and then uses some fallback setting, but
this environment variable is set here, and bogofilter tries to use it.

But then it would fail on its own or via procmail, wouldn't it?

[Sam Varshavchik]

Post by Christian Ebert
* Sam Varshavchik on Saturday, June 15, 2013 at 16:55:04 -0400

Post by Sam Varshavchik
I'll just guess that bogofilter does not decode base64-encoded content
on the fly, but tries to save them in a temporary file, and then reads
it back; and the temporary files locations are specified by
environment variables; which should be defined, but are not; or even
the other way around; bogofilter reads some environment variable
that's not normally defined, and then uses some fallback setting, but
this environment variable is set here, and bogofilter tries to use it.

But then it would fail on its own or via procmail, wouldn't it?

If the conditions would be the same, as I described previously.

[Christian Ebert]

* Sam Varshavchik on Saturday, June 15, 2013 at 19:08:06 -0400

Post by Sam Varshavchik

Post by Christian Ebert
* Sam Varshavchik on Saturday, June 15, 2013 at 16:55:04 -0400

Post by Sam Varshavchik
I'll just guess that bogofilter does not decode base64-encoded content
on the fly, but tries to save them in a temporary file, and then reads
it back; and the temporary files locations are specified by
environment variables; which should be defined, but are not; or even
the other way around; bogofilter reads some environment variable
that's not normally defined, and then uses some fallback setting, but
this environment variable is set here, and bogofilter tries to use it.

But then it would fail on its own or via procmail, wouldn't it?

If the conditions would be the same, as I described previously.

Here the conditions looks pretty much the same:

$ maildrop < testbogomsg
maildrop: error writing to filter.
maildrop: Unable to filter message.
$ procmail < testbogomsg
[ is delivered with bogofilter adding its header ]

[Sam Varshavchik]

Post by Christian Ebert
* Sam Varshavchik on Saturday, June 15, 2013 at 19:08:06 -0400

Post by Sam Varshavchik

Post by Christian Ebert
* Sam Varshavchik on Saturday, June 15, 2013 at 16:55:04 -0400

Post by Sam Varshavchik
I'll just guess that bogofilter does not decode base64-encoded content
on the fly, but tries to save them in a temporary file, and then reads
it back; and the temporary files locations are specified by
environment variables; which should be defined, but are not; or even
the other way around; bogofilter reads some environment variable
that's not normally defined, and then uses some fallback setting, but
this environment variable is set here, and bogofilter tries to use it.

But then it would fail on its own or via procmail, wouldn't it?

If the conditions would be the same, as I described previously.

$ maildrop < testbogomsg
maildrop: error writing to filter.
maildrop: Unable to filter message.
$ procmail < testbogomsg
[ is delivered with bogofilter adding its header ]

I already explained how conditions are not the same, namely the environment
variables that are set or cleared, by maildrop, for started processes. As I
wrote before, see "Environment" in the maildropfilter man page, and
replicate how bogofilter gets invoked, by setting or clearing the same
environment variables, and piping the message to its standard input.

[Christian Ebert]

* Sam Varshavchik on Saturday, June 15, 2013 at 20:14:23 -0400

Post by Sam Varshavchik
I already explained how conditions are not the same, namely the
environment variables that are set or cleared, by maildrop, for
started processes. As I wrote before, see "Environment" in the
maildropfilter man page, and replicate how bogofilter gets invoked, by
setting or clearing the same environment variables, and piping the
message to its standard input.

Then please be so kind and tell me how exactly I can achieve
this. I am not aware of having fiddled with any vars - especially
not compared to my previous machine where the exact same setup
worked. I just compared the output of env with the old machine
and could not spot any difference.

thx

[Sam Varshavchik]

Post by Christian Ebert
* Sam Varshavchik on Saturday, June 15, 2013 at 20:14:23 -0400

Post by Sam Varshavchik
I already explained how conditions are not the same, namely the
environment variables that are set or cleared, by maildrop, for
started processes. As I wrote before, see "Environment" in the
maildropfilter man page, and replicate how bogofilter gets invoked, by
setting or clearing the same environment variables, and piping the
message to its standard input.

Then please be so kind and tell me how exactly I can achieve
this. I am not aware of having fiddled with any vars - especially
not compared to my previous machine where the exact same setup
worked. I just compared the output of env with the old machine
and could not spot any difference.

As the man page explains, it's not about manually fiddling with environment
variables, it's the environment variables that maildrop sets when starting
an external process, as described.

Ask yourself, if you don't know what the environment variables are, how do
you find out? Obviously, run the "env" command. So, if you can't figure out,
after reading the man page, what kind of environment variables an external
process inherits, you just run the env command, and save its output into a
file, then look at it.

Then you would read the rest of maildropfilter, looking for commands that
execute a new process. You know about one of them, xfilter. However,
obviously it wouldn't work here. But, reading the man page, you would come
across the description of the backtick operator, which seems perfectly
suitable for the job. So, you would have

`env >environ.save`

There you have, now you can look at it, and see what the environment
variables are. Set your environment variables to be the same, pipe the
suspect message to bogofilter, and see what happens. This would be the most
accurate reproduction of the overall environment that bogofilter would run
in, from the xfilter command.

[Christian Ebert]

* Sam Varshavchik on Saturday, June 15, 2013 at 21:38:39 -0400

Post by Sam Varshavchik
As the man page explains, it's not about manually fiddling with
environment variables, it's the environment variables that maildrop
sets when starting an external process, as described.
Ask yourself, if you don't know what the environment variables are,
how do you find out? Obviously, run the "env" command. So, if you
can't figure out, after reading the man page, what kind of environment
variables an external process inherits, you just run the env command,
and save its output into a file, then look at it.
Then you would read the rest of maildropfilter, looking for commands
that execute a new process. You know about one of them, xfilter.
However, obviously it wouldn't work here. But, reading the man page,
you would come across the description of the backtick operator, which
seems perfectly suitable for the job. So, you would have
`env >environ.save`
There you have, now you can look at it, and see what the environment
variables are. Set your environment variables to be the same, pipe the
suspect message to bogofilter, and see what happens. This would be the
most accurate reproduction of the overall environment that bogofilter
would run in, from the xfilter command.

You meant "pipe to maildrop", right?

Oh, right, you meant something easy ;-) I'd already done that -
put the env backticks right before the xfilter with bogofilter -
and compared to the output on my old machine where it works - it
is exactly the same:

SED=/usr/bin/sed
HOSTNAME=krille.blacktrash.org
SHELL=/bin/bash
UMASK=077
MAILDIRQUOTA=
SIZE=333285
LOCKEXT=.lock
LC_ALL=en_US.UTF-8
DEFAULT=/Users/chris/Maildir
MAILFILTER=.mailfilter
PATH=/bin:/usr/bin:/usr/local/bin
LOCKTIMEOUT=60
PWD=/Users/chris
REFORMAIL=/usr/local/bin/reformail
LANG=en_US.UTF-8
MAILDIRMAKE=/usr/local/bin/maildirmake
LINES=4455
SHLVL=1
HOME=/Users/chris
LOCKSLEEP=5
LOCKREFRESH=15
FROM=chris
LOGNAME=chris
MATCH=Message-ID:
SENDMAIL=/usr/sbin/sendmail -oi
_=/usr/bin/env

I have 3 rules and program var settings before the exfilter,
setting the backticks as very first rule yields this:

HOSTNAME=krille.blacktrash.org
SHELL=/bin/bash
UMASK=077
MAILDIRQUOTA=
SIZE=333309
LOCKEXT=.lock
LC_ALL=en_US.UTF-8
DEFAULT=/Users/chris/Maildir
MAILFILTER=.mailfilter
PATH=/bin:/usr/bin:/usr/local/bin
LOCKTIMEOUT=60
PWD=/Users/chris
LANG=en_US.UTF-8
LINES=4458
SHLVL=1
HOME=/Users/chris
LOCKSLEEP=5
LOCKREFRESH=15
FROM=***@xxxx
LOGNAME=chris
SENDMAIL=/usr/sbin/sendmail -oi
_=/usr/bin/env

[Matthias Andree]

Christian,

Sorting and comparing your ENV dumps with comm -3 yields
(left-justified: only in first part; indented: only in second part;
omitted: common to both parts):

FROM=chris
FROM=***@xxxx
LINES=4455
LINES=4458
MAILDIRMAKE=/usr/local/bin/maildirmake
MATCH=Message-ID:
REFORMAIL=/usr/local/bin/reformail
SED=/usr/bin/sed
SIZE=333285
SIZE=333309

Looks harmless from bogofilter's perspective, it would not use any of those.

I'd now suggest to add a few bogofilter debug options, and redirect its
stdout and stderr to some file, such as (replace DEBUG flags by a
concatenation of *lower-case* characters from the list below), this goes
on one line:

xfilter "bogofilter -u -e -p -vvv -x DEBUG 2>stderr.txt | tee stdout.txt"

#define BIT_ALGORITHM MASK_BIT('A')
#define BIT_READER MASK_BIT('B')
#define BIT_CONFIG MASK_BIT('C')
#define BIT_DATABASE MASK_BIT('D')
#define BIT_GENERAL MASK_BIT('G')
#define BIT_HTML MASK_BIT('H')
#define BIT_ICONV MASK_BIT('I')
#define BIT_LEXER MASK_BIT('L')
#define BIT_MIME MASK_BIT('M')
#define BIT_MULTI MASK_BIT('U')
#define BIT_REGISTER MASK_BIT('R')
#define BIT_SPAMICITY MASK_BIT('S')
#define BIT_TEXT MASK_BIT('T')
#define BIT_WORDLIST MASK_BIT('W')
#define BIT_MEMORY MASK_BIT('Y')
#define BIT_TEST MASK_BIT('Z')

I think that -x bcdgmty and possibly an added l (letter ell) should be
all you need. stderr.txt will then contain a trace of what bogofilter is
attempting to do and how far it has gotten. Note it may contain binary
data, so don't dump it to a console directly, but use cat -v or
something else that escapes control characters.

Report those stdout.txt and stderr.txt files to David Relson and myself
by email OFF-LISTS (but I'll probably not be able to look at this before
30 hours from now).

Setting Reply-To: to bogofilter list, to which you are also subscribed.


It may also help to show a trace of

cat offending-message.msg | maildrop -V2
(note that this does *not* reproduce how Postfix calls maildrop, because
your postfix uses maildrop -d chris, but -d is incompatible with -V
according to my maildrop manpage).


As well as the truss.txt you get from:

cat offending-message.msg | truss -aef -o truss.txt maildrop -d chris


(This is assuming "chris" for your login and that Darwin kept FreeBSD's
truss(1). Replace accordingly.)

[Matthias Andree]

(now moving to the maildrop list)


Christian,

can you send me the dtruss files you sent to David, off-list? They are
stripped from the bogofilter lists, so I don't have them, and I would
like to have a look at them.


Sam,

we appear to see that maildrop-on-new-MacOS does not get all data
through xfilter. David Relson traced in- and outputs of bogofilter, and
it looks as though the stdin that bogofilter is getting is truncated at
exactly 40 kBytes. Hints as to the problem below.

Is there a way to log more details when writing to the xfilter
socketpair or pipe (whatever it's using, I haven't read your code),
especially like name, decoded errno, and thereabouts?
(I hope I see more in dtruss output.)


This is from the bogofilter list archives:

1. David's test setup:

http://www.bogofilter.org/pipermail/bogofilter/2013-June/009970.html

I suppose that David had an additional pipe symbol in his bogofilter.sh
here, and personally, I would replace $* by "$@" to retain word
separators and avoid a second word-splitting phase on the shell:

#### BEGIN bogofilter.sh ####
tee /tmp/stdin.out | bogofilter $* -x amlt 2> /tmp/stderr.out \
| tee /tmp/stdout.out
#### END bogofilter.sh ####


2. resulting sizes on David's Linux computer in
http://www.bogofilter.org/pipermail/bogofilter/2013-June/009971.html
showing the test message to be > 320 kB.

3. Christian's resulting stdin/out/err.out sizes on MacOS X, showing
bogofilter's stdin being exactly 40960 bytes:
http://www.bogofilter.org/pipermail/bogofilter/2013-June/009972.html


4. David's conclusion at
http://www.bogofilter.org/pipermail/bogofilter/2013-June/009973.html
"Since 40k is far short of the 333,309 bytes in the problem message, I
conclude this is a MacOS specific problem -- possibly a maildrop issue,
but more likely a pipe issue. [...]"


The thread index is at
http://www.bogofilter.org/pipermail/bogofilter/2013-June/thread.html
under the same subject.

Best regards
Matthias Andree

[Sam Varshavchik]

Post by Matthias Andree
Sam,
we appear to see that maildrop-on-new-MacOS does not get all data
through xfilter. David Relson traced in- and outputs of bogofilter, and
it looks as though the stdin that bogofilter is getting is truncated at
exactly 40 kBytes. Hints as to the problem below.
Is there a way to log more details when writing to the xfilter
socketpair or pipe (whatever it's using, I haven't read your code),
especially like name, decoded errno, and thereabouts?
(I hope I see more in dtruss output.)

Nope, the loop that writes the message to the pipe does not do any logging
of its own. If write() returns an error, and it's not EINTR, then you get
"maildrop: error writing to filter." It's fairly simple.

[Christian Ebert]

* Sam Varshavchik on Thursday, June 20, 2013 at 07:01:45 -0400

Post by Sam Varshavchik

Post by Matthias Andree
we appear to see that maildrop-on-new-MacOS does not get all data
through xfilter. David Relson traced in- and outputs of bogofilter, and
it looks as though the stdin that bogofilter is getting is truncated at
exactly 40 kBytes. Hints as to the problem below.
Is there a way to log more details when writing to the xfilter
socketpair or pipe (whatever it's using, I haven't read your code),
especially like name, decoded errno, and thereabouts?
(I hope I see more in dtruss output.)

Nope, the loop that writes the message to the pipe does not do any
logging of its own. If write() returns an error, and it's not EINTR,
then you get "maildrop: error writing to filter." It's fairly simple.

My layman's summary:

1) Not related to bogofilter, can be replicated with this minimal
config:

xfilter "/bin/cat"
to "$DEFAULT"

2) Happens with any kind of message > 40kb

I suspect an issue regarding temp files and/or with clang and/or
header files, but have C or C++ experience whatsoever.

[Sam Varshavchik]

Post by Christian Ebert
* Sam Varshavchik on Thursday, June 20, 2013 at 07:01:45 -0400

Post by Sam Varshavchik

Post by Matthias Andree
we appear to see that maildrop-on-new-MacOS does not get all data
through xfilter. David Relson traced in- and outputs of bogofilter, and
it looks as though the stdin that bogofilter is getting is truncated at
exactly 40 kBytes. Hints as to the problem below.
Is there a way to log more details when writing to the xfilter
socketpair or pipe (whatever it's using, I haven't read your code),
especially like name, decoded errno, and thereabouts?
(I hope I see more in dtruss output.)

Nope, the loop that writes the message to the pipe does not do any
logging of its own. If write() returns an error, and it's not EINTR,
then you get "maildrop: error writing to filter." It's fairly simple.

1) Not related to bogofilter, can be replicated with this minimal
xfilter "/bin/cat"
to "$DEFAULT"
2) Happens with any kind of message > 40kb
I suspect an issue regarding temp files and/or with clang and/or
header files, but have C or C++ experience whatsoever.

I'm inclined to agree, except that it's probably has to do with the behavior
of the select(3) system call.

You need a Mac-OS-based developer here to investigate what's going on,
there. This should be trivially fixable, just need someone on Mac-OS to roll
their sleeves up, and start digging.

[Christian Ebert]

* Sam Varshavchik on Wednesday, July 03, 2013 at 08:10:08 -0400

Post by Sam Varshavchik

Post by Christian Ebert
1) Not related to bogofilter, can be replicated with this minimal
xfilter "/bin/cat"
to "$DEFAULT"
2) Happens with any kind of message > 40kb
I suspect an issue regarding temp files and/or with clang and/or
header files, but have C or C++ experience whatsoever.

I'm inclined to agree, except that it's probably has to do with the
behavior of the select(3) system call.
You need a Mac-OS-based developer here to investigate what's going on,
there. This should be trivially fixable, just need someone on Mac-OS
to roll their sleeves up, and start digging.

Seems chances are rather low for that to happen. This must have
been a problem at least since Mac OS 10.7 (switch to clang).

I've looked a bit what packagers do regarding patching:
- fink does not offer maildrop anymore
- macports offers maildrop, but without any significant patching
afaics; looks like no-one actually uses maildrop, or only
simple configs without xfilter

Duh. What a pity.

[Christian Ebert]

guys,

A quick - hopefully not final - summary from a layman's
perpective.

* Christian Ebert on Thursday, June 13, 2013 at 12:08:55 +0100

Post by Christian Ebert
After an upgrade to Mac OS 10.8.4 (64bit) from 10.5.8 (32bit) I
temporary failure.
Command output: maildrop: error writing to filter.
/usr/local/bin/maildrop: Unable to filter message.
xfilter "/usr/local/bin/bogofilter -u -e -p"
Trying the bogofilter suspect on an offending message however
$ bogofilter -u -e -p < testbogomsg 1>/dev/null
$ echo $?
0
Of course I recompiled all involved programs, like maildrop,
bogofilter, tokyocabinet.
What could be the problem?

With the friendly help of Sam, Matthias and David (from
bogofilter) it looks like this:

maildrop refuses to write more than ca 40kb via xfilter - that's
basically it.
If it is not only my machine, probably Mac OS >= 10.7 is affected
(64bit, clang).

I can offer to try patches - but only off and on during the next
10 days, if at all. Real work and life begs, and I brought up my
old procmail config reasonably up to speed - peppered with the
appropriate { EXITCODE=75 HOST } - to keep my email going. Still
interested in pursuing the maildrop hobby though.

Thanks.

Christian

[Dimitri Maziuk]

Post by Christian Ebert
guys,
A quick - hopefully not final - summary from a layman's
perpective.
maildrop refuses to write more than ca 40kb via xfilter - that's
basically it.
If it is not only my machine, probably Mac OS >= 10.7 is affected
(64bit, clang).

Interesting. My .mailfilter starts with

xfilter "/usr/bin/spamc -f ${SENDER}"

and I'm getting multi-megabyte e-mails w/o problems. That's centos 6
x86_64, "standalone" maildrop.