[Pkg-exim4-users] rspamd spam scanning fails after upgrade to 4.88 bpo on jessie

[Thomas Hager]

Hi list,

I updated Exim on my jessie box to 4.88-4~bpo8+1 a few days ago and
discovered about now that the update broke spam scanning with rspamd.

From the logs:

2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: spamd: failed
to connect to any address for 127.0.0.1: Broken pipe
2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: all spamd
servers failed

The relevant configuration:

spamd_address      = 127.0.0.1 11333 variant=rspamd

rspamd is up and running, but isn't registering any connection attempt
by Exim. Running Exim with debugging didn't show more than the logged
broken pipes from above.

After downgrading to 4.87-3~bpo8+1 spam scanning works as expected
again. So I traced both versions, which showed a difference in
connecting to rspamd:

4.87:
[...]
26092 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
26092 alarm(5)                          = 0
26092 connect(6, {sa_family=AF_INET, sin_port=htons(11333),
sin_addr=inet_addr("127.0.0.1")}, 16) = 0
26092 alarm(0)                          = 5
26092 close(-1)                         = -1 EBADF (Bad file
descriptor)
26092 fcntl(6, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
26092 sendto(6, "CHECK RSPAMC/1.3\r\nContent-length"..., 210, 0, NULL,
0) = 210
[...]

4.88:
[...]
25605 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
25605 alarm(5)
25605 sendto(6, NULL, 0, MSG_FASTOPEN, {sa_family=AF_INET,
sin_port=htons(11333), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EPIPE
(Broken pipe)
[...]

Before I take this to the upstream list I'd like to verify if this is
an issue with the 4.88 bpo package on jessie.

Tia,
Tom.

[Andreas Metzler]

Post by Thomas Hager
I updated Exim on my jessie box to 4.88-4~bpo8+1 a few days ago and
discovered about now that the update broke spam scanning with rspamd.
2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: spamd: failed
to connect to any address for 127.0.0.1: Broken pipe
2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: all spamd
servers failed

[...]

Post by Thomas Hager
Before I take this to the upstream list I'd like to verify if this is
an issue with the 4.88 bpo package on jessie.

Hello,

I would be surprised if this was Debian specific. There was a similar
report in December.
http://lists.alioth.debian.org/pipermail/pkg-exim4-users/2016-December/002368.html

cu Andreas

[Thomas Hager]

Post by Andreas Metzler
I would be surprised if this was Debian specific. There was a similar
report in December.
http://lists.alioth.debian.org/pipermail/pkg-exim4-users/2016-December/002368.html

Yeah, Tim already pointed that out. Kinda missed that thread while searching, sorry.

I'll see to it that I can recompile and test Exim tonight with the mentioned patch.

Cheers,
Tom.

[Thomas Hager]

Post by Andreas Metzler

Post by Thomas Hager
I updated Exim on my jessie box to 4.88-4~bpo8+1 a few days ago and
discovered about now that the update broke spam scanning with rspamd.
2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: spamd: failed
to connect to any address for 127.0.0.1: Broken pipe
2017-01-22 08:59:55 1cVD46-0004AB-QI spam acl condition: all spamd
servers failed

[...]

Post by Thomas Hager
Before I take this to the upstream list I'd like to verify if this is
an issue with the 4.88 bpo package on jessie.

Hello,
I would be surprised if this was Debian specific. There was a similar
report in December.
http://lists.alioth.debian.org/pipermail/pkg-exim4-users/2016-Decembe
r/002368.html

Hi Andreas,

I cc the Exim users list now, because I found the code, which causes
the described connection failures.

To begin with, reverting the patch described in
https://bugs.exim.org/show_bug.cgi?id=1802 didn't fix the error.

So I dug deeper and found commit fb05276a ("TCP Fast Open"). 

If TCP_FASTOPEN is defined in netinet/tcp.h, Exim 4.88 uses
MSG_FASTOPEN for STREAM sockets. If TFO is not available, the code in
ip.c catches an EOPNOTSUPP error in the call to sendto() and falls back
to connect(). On jessie, sendto() doesn't raise an EOPNOTSUPP error
when Exim tries to connect to rspamd with MSG_FASTOPEN, but raises an
EPIPE error instead.

This error is not caught and Exim is unable to connect to the rspamd
process running on my box.

A small patch fixed this error (see below), but I don't know if this is
the best approach. Maybe Exim should verify if TFO is available and
enabled before using TFO at all on a Linux server.

Cheers,
Tom.

diff --git a/src/src/ip.c b/src/src/ip.c
index 8bb6bed..9d82e49 100644
--- a/src/src/ip.c
+++ b/src/src/ip.c
@@ -236,7 +236,7 @@ connect in FASTOPEN mode but with zero data.
 if (fastopen)
   {
   if (  (rc = sendto(sock, NULL, 0, MSG_FASTOPEN, s_ptr, s_len)) < 0
-     && errno == EOPNOTSUPP
+     && (errno == EOPNOTSUPP || errno == EPIPE )
      )
     {
     DEBUG(D_transport)

[Andreas Metzler]

Post by Thomas Hager
I updated Exim on my jessie box to 4.88-4~bpo8+1 a few days ago and
discovered about now that the update broke spam scanning with rspamd.

[...]

So I dug deeper and found commit fb05276a ("TCP Fast Open"). 
If TCP_FASTOPEN is defined in netinet/tcp.h, Exim 4.88 uses
MSG_FASTOPEN for STREAM sockets. If TFO is not available, the code in
ip.c catches an EOPNOTSUPP error in the call to sendto() and falls back
to connect(). On jessie, sendto() doesn't raise an EOPNOTSUPP error
when Exim tries to connect to rspamd with MSG_FASTOPEN, but raises an
EPIPE error instead.
This error is not caught and Exim is unable to connect to the rspamd
process running on my box.

[...]

Thank you for chasing this down.

For the list archives:
Jeremy Harris has opened a bug report on
https://bugs.exim.org/show_bug.cgi?id=2027 to keep track of the issue.

cu Andreas