Add First Packet Classification capability

[IvanNardi]

In some contexts/deployments (load-balancing or QoS involved, routing on different interfaces based on classification...) is quite important to have a flow classification on the very FIRST packet of the flow, even if that classification is not as precise as it could be if more packets are analyzed.

Note that, for TCP, "first packet" means (usually) the SYN packet...

[mmanoj]

@IvanNardi

This is good to have,I like to implement this feature, if you can guide me.

[IvanNardi]

@mmanoj , very sorry for the delay.
FPC is a complex topic, but the most easy (and obvious) ways to achieve that are:

• via IP match, improving the IP lists. nDPI already has some ip lists of the most important services (see /util and utils/asn_update.sh. We might add some other protocols/lists
• using the DNS information. If clientA triggers the DNS transaction hostnameXXX/PROTOCOL_Y -> IP_XXX, then it is likely that a future flow from clientA to IP_XXX is related to PROTOCOL_Y

If you are still interested, we can discuss about the details

[mmanoj]

@IvanNardi Thx for the reply, Yes I'm interested and done some initial work with nDPIsimple integration as well.Can we discuss in detail with some actionable steps, so we can focus on the step by step implementation.

[IvanNardi]

1. Extend the ip lists. Take a look at the existing lists (in /util and utils/asn_update.sh) and try adding other ones: new important/used applications/services, new streaming services, new game networks, missing CDN providers.... There are already examples about integrating a list from a csv or txt file, or from BGP info: the hard part is to find an official/reliable source for those ips...

2. Create a new LRU cache where the key is the pair "Client_IP - Resolved_IP" and the value is a protocol id:

• for each DNS A/AAAA pair create a new entry, where the protocol is the classification associated to the required hostname
• for each flow (at the very first packet) lookup into the cache using "SRC IP - DST IP" as key and, if a match is found, save the protocol id into ndpi_flow_struct structure as a "fpc result"

[mmanoj]

@IvanNardi

Thanks for the update,I will work on below plan.
Step #1. Start with action item #2 with existing IP list to implement PoC with nDPI-Simple integration example.

Please clarify below point: Is current nDPI DNS dissectors support to capture DNS answer (A/AAAA) record and IP address associated with it or need to develop that part? anyway I will lookup the code as well.

Step #2. I will work on enhancing the new IP Lists.

Please advice is above approach is fine or need any changes.

[IvanNardi]

Please clarify below point: Is current nDPI DNS dissectors support to capture DNS answer (A/AAAA) record and IP address associated with it or need to develop that part? anyway I will lookup the code as well.

It already does it.

The plan sounds good.

Please, take a close look at #2488: once it is merged, it provides basic common code (and output...) that you might find useful...

[mmanoj]

@IvanNardi

Thank you very much for the FPC base code,I will go through the fpc brach for details.

this is how I'm going to capture the first packet in nDPI simple Integration code::

tcp = (struct ndpi_tcphdr *)l4_ptr;
flow.is_flow_first_pkt = (tcp->syn == 1 ? 1 : 0);

#Then:
/* Check first Pkt logic */
if(flow.is_flow_first_pkt ==1 && flow.flow_ack_seen ==0)
{
printf("\n This is First-Pkt........\n");
#Step 1: check in LRU cache for the service/protocol based on "Client_IP - Resolved_IP"
#Step 2: if not found in LRU cache then classify and update the LRU cache. To classify I will use below method from FPC base code:
flow->fpc.protocol,
}

Is above logic and understanding if fine ? Please advice.

[mmanoj]

@IvanNardi

Can you please advice what is the best way to retrieve the category of the FPC protocol, I'm going to add that also to LRU cache as it's useful for blocking certain categories than individual protocols.

I try below under fpc_update() method but not work as expected. ndpi_proto->category = ndpi_get_flow_category(flow);

[mmanoj]

@IvanNardi

I done the base work for Action #2 LRU cache, can I send the file(s) to review?

[IvanNardi]

@IvanNardi

I done the base work for Action #2 LRU cache, can I send the file(s) to review?

Yes, please! These are busy days.. I'll take a look at it (and at your previous messages...) at the end of the week

[mmanoj]

@IvanNardi

Thanks for feedback and appreciate your support.I will attached the files instead pull request lets review the overall flow and structure of my FPC LRU cache implementation.Once review will proceed with DNS FPC.

[mmanoj]

ndpi_fpc_lru_cache.zip
@IvanNardi
Please find initial FPC_LRU cache related changes attached for your review.

Note:During testing I noticed ndpi_str->fpc_cache not initialized.Need more debugging for that.

[IvanNardi]

@mmanoj , could you push your code as a Pull Request, please? It will drastically improve the review process... Thanks

[mmanoj]

@IvanNardi

Thanks for the feedback, I will arrange it by today.

[mmanoj]

@IvanNardi

Please have a look pull request #2497

[mmaatuq]

Can we make FPC feature configurable, maybe through conditional compilation
The reason for suggesting this, In some scenarios to support FPC we might need additional heavy processing

• examples:

  • ftp control, ftp data: when you have to parse the ftp control connection for commands like PORT, PSV to extract the address,port for the data connection that will start.
  • in some other application protocols: there will a specific pattern you have to follow that will till you a new connection will start for some new activities.
  • I don't have a strong opinion in this, as the requirements for FPC might not need to go for such heavy processing and we just will go with things like dns, ips, ports.

[IvanNardi]

@mmaatuq , you can already enable/disable FPC via "fpc" configuration parameter. You can take a look a the tests under tests/fpc_disabled/ for an example.

[mmaatuq]

@mmaatuq , you can already enable/disable FPC via "fpc" configuration parameter. You can take a look a the tests under tests/fpc_disabled/ for an example.

thanks for this