Paramiko sftp hanging with connections between machines on the same interface of a filtering pfsense box

Odd problem; I had the following set up:

[[machine with paramiko 10.100.x.x]] –|

                                                              | —–(int X) 10.100.x.x [[pfsense]] (int Y) 10.2.x.x —– | —— [[10.2.x.x machine B]]

[[machine A 10.100.x.x]] ——————|

 

I had a script on the paramiko machine connecting via ssh and sftp to machines A and B.  Connections to machine B had no problem whatsoever.  Connections to machine A, however, would work 5% of the time, and drop the rest of the time either when setting up the channel to execute a command over ssh, or when invoking the sftp subsystem on the remote machine.  Normal ssh and sftp connections (not using paramiko) had no problems whatsoever.  Also, when pfSense filtering was turned off, there were also no problems.

It turned out that pfsense was dropping a lot of packets sent by paramiko due to fragmentation (logs show TCP:PA, TCP:RA and TCP:A).  Unfortunately, tweaking pfsense settings didn’t help here (some people have reported that setting Firewall Optimization Options (under Advanced > Firewall/NAT) to conservative worked – that didn’t help me unfortunately – or disabling firewall scrub worked – which I couldn’t do as it’s required by NAT).

I haven’t been able to figure out exactly what the problem is.  The packets received by machine B and machine A (with filtering off) look exactly the same.  I’m tempted to think this is a pfsense problem, although I have no specific proof (I’ve tested with multiple machines in position of machine A by the way, compared ssh settings, ensured there were no other connectivity problems in the way).

In the end, I’ve set up another network (virtual one, since these are VMs – 10.100.x.x machines plus pfsense on one physical host, and 10.2.x.x on another) connecting these VMs directly to eachother, to bypass pfsense for these connections.

Leave a Reply

Your email address will not be published. Required fields are marked *