Tag Archives: pfsense

Paramiko sftp hanging with connections between machines on the same interface of a filtering pfsense box

Odd problem; I had the following set up:

[[machine with paramiko 10.100.x.x]] –|

                                                              | —–(int X) 10.100.x.x [[pfsense]] (int Y) 10.2.x.x —– | —— [[10.2.x.x machine B]]

[[machine A 10.100.x.x]] ——————|

 

I had a script on the paramiko machine connecting via ssh and sftp to machines A and B.  Connections to machine B had no problem whatsoever.  Connections to machine A, however, would work 5% of the time, and drop the rest of the time either when setting up the channel to execute a command over ssh, or when invoking the sftp subsystem on the remote machine.  Normal ssh and sftp connections (not using paramiko) had no problems whatsoever.  Also, when pfSense filtering was turned off, there were also no problems.

It turned out that pfsense was dropping a lot of packets sent by paramiko due to fragmentation (logs show TCP:PA, TCP:RA and TCP:A).  Unfortunately, tweaking pfsense settings didn’t help here (some people have reported that setting Firewall Optimization Options (under Advanced > Firewall/NAT) to conservative worked – that didn’t help me unfortunately – or disabling firewall scrub worked – which I couldn’t do as it’s required by NAT).

I haven’t been able to figure out exactly what the problem is.  The packets received by machine B and machine A (with filtering off) look exactly the same.  I’m tempted to think this is a pfsense problem, although I have no specific proof (I’ve tested with multiple machines in position of machine A by the way, compared ssh settings, ensured there were no other connectivity problems in the way).

In the end, I’ve set up another network (virtual one, since these are VMs – 10.100.x.x machines plus pfsense on one physical host, and 10.2.x.x on another) connecting these VMs directly to eachother, to bypass pfsense for these connections.

Configuring a network-to-network NAT in pfSense

In this case, I’m NATing 10.8.0.0/16 (interface name = vlan8) to 10.120.0.0/16 (interface name = int8), so a packet to 10.8.150.3 will be NATed to 10.120.150.3.

Go to Firewall -> Nat
Create a new 1:1 mapping, and put the settings as follows:
Interface: vlan8
External subnet IP: 10.8.0.0
Internal IP: int8 subnet
Destination: any (you might be able to use int8 subnet here, but it wouldn’t work with my VPN configuration as VPN IPs are on a separate subnet)
NAT reflection: use system default

And save, now to Firewall -> virtual IPs
Create a new virtual IP
I’ve used CARP, but when I get the chance I’ll try Proxy ARP, which would be better for those who have an entire subnet behind the pfsense (I don’t, so I need to put in each address to NAT individually)

And then the settings on your host behind the pfsense:
IP: 10.120.150.3/16 (whatever IP you want)
gw: 10.120.x.x (IP of pfsense’s int8 interface)
(to set the gateway in Ubuntu, using /etc/network/interfaces didn’t seem to want to work for me, so I used “route add default gw 10.120.x.x” instead)