Building a fault-tolerant firewall system with virtual machines: expectation: part 2: expectation


Recovering helper

The feature of fault-tolerant firewall is the ability to recover connections. But if it loses the helper, the connection recovering fails. When conntrackd injects a conntrack whose inherent helper into the kernel conntrack table, the netlink subsystem creates helper for it. Unfortunately, the work of NAT later took the helper away (don't use automatic helper assignment, now the safe way is to explicitly define helper using the firewall rule ct helper set). To fix it, we add the following code to the __nf_ct_try_assign_helper() function:

/* tmpl has no helper but injected conntrack may have */
if (help && helper == NULL)
    helper = rcu_dereference(help->helper);

This bug has been reported to

An expectation is a nf_conntrack_expect structure. Notable fields are:
     • lnode: A hlist_node to insert the expectation into the list in the master conntrack.
     • hnode: A hlist_node to insert the expectation into the expectation hash table (expect table).
     • tuple: Contains layer 3 and layer 4 protocol information to match the expected conntrack.
     • mask: Tuple mask to compare expectations.
     • helper: The helper pointer to assign to the expected conntrack.
     • master: conntrack master.
     • timeout: When the expectation expires, the system destroys the expectation and broadcasts the IPEXP_DESTROY message.
     • use: When use is equal to 1 the function nf_ct_expect_put() will destroy expectation.
     • flags: If the flag is not set the expectation is used only once, it is destroyed immediately after that.
     • class: The class of expectation. FTP uses only one class, 0.

When the user is at the ftp command prompt at the client and runs a command such as the 'ls' command, client/server negotiation about the destination port number begins. The client requests a port in the origin direction and the server returns the port number in the reply direction. The negotiation uses the master connection, i.e. uses the same source port and destination port 21 so it reuses the master conntrack, this conntrack is old so the negotiation is not seen by the user, ie no more events are seen.
The conntrack's internal status is changeless, it is still "established". However the external state, ctinfo is updated in the direction of the package. When the server returns the destination port number for the expected connection, i.e. in the reply direction, now ctinfo equals IP_CT_ESTABLISHED_REPLY and the helper's help() function begins to create expectation (in the postrouting stage).
It first takes the TCP header in the packet to calculate the data offset and gain the data. There is an important piece of information in the TCP header, ie the sequence number of the first byte in the data. The help() function always updates the sequence numbers so it knows that this first byte is behind a newline ('\n') in the data series. And so it analyzes the port pattern to find the destination port. Layer 3 protocol information including source address, destination address, ip protocol is already available in master conntrack, class is 0, layer 4 protocol is TCP, along with the destination port number just found is enough to create expectation for the expected connection.
The expected connection is exactly data connection, it happens right after the connection creating expectation. It is a new connection and a new conntrack is created. During initialization, conntrack looks for expectation, obviously it finds out the expextation just created for it. So it sets the IPS_EXPECTED bit to status, and so ctinfo is IP_CT_RELATED. The data connection is related to the master connection and is allowed to go through the firewall.
During the finding process, once found the expectation is destroyed because there is no flag by default. That shows that the expect table is always empty and the commands conntrackd -i exp, conntrackd -e exp and conntrack -L exp always appear nothing. Thus the user never sees expectations in the table. Furthermore, conntrackd's expectation synchronization function is meaningless.

To observe the expectation we need to modify the kernel a bit. First set the expectation flag to NF_CT_EXPECT_PERMANENT, so it will persist in the expect table and in the master conntrack's list until it expires.
But just setting the flag is not very good because every time a data connection is made a new expectation is created even though the old expectation has not expired yet. To reuse the old expectation we add the following code to the help() function:

/* We only update sequence number and create expectation
 * for old dest port. Old one is then renewed shortly
 * after that in the hash table. */
hlist_for_each_entry(exp, &help->expectations, lnode) {
    if (exp->tuple.dst.u.all == cmd.u.tcp.port)
    ret = NF_ACCEPT;
    goto out_update_nl;

Something is not right here! Expected connections cannot use other expectation. We need to define a port mask to allow a port number. In this practice we allow 4 ports. A mask is passed to the nf_ct_find_expectation() function:

__be16 port_mask = htons(0xfffc);

And the expectation finding condition is modified:

int found = 0;
for (h = 0; h < nf_ct_expect_hsize; h++) {
    hlist_for_each_entry(i, &nf_ct_expect_hash[h], hnode) {
        if (!(i->flags & NF_CT_EXPECT_INACTIVE) &&
            net_eq(net, nf_ct_net(i->master)) &&
            nf_ct_zone_equal_any(i->master, zone) &&
            nf_ct_tuple_src_mask_cmp(tuple, &i->tuple, &i->mask) &&
            nf_inet_addr_cmp(&tuple->dst.u3, &i->tuple.dst.u3) &&
            tuple->dst.protonum == IPPROTO_TCP &&
            !((tuple->dst.u.all ^ i->tuple.dst.u.all) & port_mask)) {
            exp = i;
            found = 1;
    if (found)

Details are in the patch file linux-5.10.81-exp-keep-development-purpose-only-1.patch.
Note: The above patch file is only used in practice to observe expectations, do not apply in fact!

Now we will see expectations

Once an expectation is created, it will last until it expires. In a special case, if the new expecting connection has the same destination port as an existing expectation, a new expectation is added. These two expectations will be in the same bucket in the hash table and have the same tuple so the old expectation is destroyed and the new expectation replaces it

In the first message, after a 'ls' command, an expectation is created for port 30003. After the second 'ls' command, the port is the same as before, 30003 so a new expectation is created and the old expectation in the same bucket is destroyed despite the time limit, 19 seconds left. In the fourth message, that new expectation is destroyed because of its expiration (a newly created expectation has a life time of 30 seconds). Now the expect table becomes empty so a new expectation is created for port 30002 by the next 'ls' command, in the fifth message. It ended up being destroyed also because of expiration.

To see more, watch the video below

Sync expectations

Now let's switch to the backup firewall fw-2. Its expect table is initially empty. The conntrackd -e exp command shows that fw-2 has obtained an expectation from the active firewall fw-1 and cached it externally. We use the conntrackd -c command to inject that expectation into the kernel expect table. So fw-2 now has an entry in the expect table

Currently unrated


There are currently no comments

New Comment


required (not published)



What is 8 × 1?