Sigh, I swear I find all the weird edge cases with every product I touch, hence this blog. The latest being pmacct on CentOS 6 with a flow-fanout
The latest relates to those shiny new ASRs I mentioned earlier and netflow.
So we use Netflow on our border routers to account traffic leaving the network and let people have free traffic within the DC.
We have a few netflow boxes (stats, analytics and accounting) so to spare the load on the routers we use flow-fanout.
In our network ig goes something like this
10.30.30.0/24 on the routers -> 10.30.30.10 Fanout eth1 -> 10.30.31.0/24 eth2 -> Various boxes, one of which is pmacct
Becuase of the way we use pmacct, it needs the SNMP interface index, so we pass the 10.30.30.0/24 address to pmacct
Now this is all well and good on CentOS 5, but were trying to upgrade from that, and when I spun it up on Cent OS 6, i couldn’t get it to work for the life of me. I thought it was ESXi (all our netflow is on one box to save moving data around the network 4 times)
I’ll spare you the long story but its to do with the network stack on CentOS 6, even with the interface in promiscious mode, its rp_filter.
rp filter is responsible for a reverse path check. Because the flow-fanout box rewrite part of the packet header, it looks like a spoofed packet and although you can see the traffic on the interface, it doesn’t make it to pmacct.
The fix is to put it into a nice mode with:
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
And viola! Netflow data, sweet sweet netflow data
