Eric Dumazet
b0af0ab193
tcp: TCP Small Queues
...
This introduce TSQ (TCP Small Queues)
TSQ goal is to reduce number of TCP packets in xmit queues (qdisc &
device queues), to reduce RTT and cwnd bias, part of the bufferbloat
problem.
sk->sk_wmem_alloc not allowed to grow above a given limit,
allowing no more than ~128KB [1] per tcp socket in qdisc/dev layers at a
given time.
TSO packets are sized/capped to half the limit, so that we have two
TSO packets in flight, allowing better bandwidth use.
As a side effect, setting the limit to 40000 automatically reduces the
standard gso max limit (65536) to 40000/2 : It can help to reduce
latencies of high prio packets, having smaller TSO packets.
This means we divert sock_wfree() to a tcp_wfree() handler, to
queue/send following frames when skb_orphan() [2] is called for the
already queued skbs.
Results on my dev machines (tg3/ixgbe nics) are really impressive,
using standard pfifo_fast, and with or without TSO/GSO.
Without reduction of nominal bandwidth, we have reduction of buffering
per bulk sender :
< 1ms on Gbit (instead of 50ms with TSO)
< 8ms on 100Mbit (instead of 132 ms)
I no longer have 4 MBytes backlogged in qdisc by a single netperf
session, and both side socket autotuning no longer use 4 Mbytes.
As skb destructor cannot restart xmit itself ( as qdisc lock might be
taken at this point ), we delegate the work to a tasklet. We use one
tasklest per cpu for performance reasons.
If tasklet finds a socket owned by the user, it sets TSQ_OWNED flag.
This flag is tested in a new protocol method called from release_sock(),
to eventually send new segments.
[1] New /proc/sys/net/ipv4/tcp_limit_output_bytes tunable
[2] skb_orphan() is usually called at TX completion time,
but some drivers call it in their start_xmit() handler.
These drivers should at least use BQL, or else a single TCP
session can still fill the whole NIC TX ring, since TSQ will
have no effect.
Signed-off-by: Eric Dumazet <edumazet@google.com >
Cc: Dave Taht <dave.taht@bufferbloat.net >
Cc: Tom Herbert <therbert@google.com >
Cc: Matt Mathis <mattmathis@google.com >
Cc: Yuchung Cheng <ycheng@google.com >
Cc: Nandita Dukkipati <nanditad@google.com >
Signed-off-by: David S. Miller <davem@davemloft.net >
Change-Id: I37d5e4d7c9ced1846385b6a04ae3ad134763a949
2020-12-12 21:36:55 +01:00
..
2020-12-09 16:01:42 +01:00
2013-10-14 10:15:17 -07:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:22:45 +02:00
2019-08-02 15:14:10 +02:00
2019-08-08 15:00:19 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-09 09:14:19 +02:00
2019-08-09 09:13:49 +02:00
2019-08-09 09:14:12 +02:00
2019-08-02 15:14:10 +02:00
2013-07-02 14:15:23 -07:00
2019-08-06 11:41:21 +02:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2020-09-23 17:28:13 +02:00
2020-09-23 18:05:04 +02:00
2008-02-06 10:41:02 -08:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 11:41:21 +02:00
2013-06-20 10:46:21 -07:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2012-03-24 10:41:37 -07:00
2019-08-05 14:22:29 +02:00
2019-08-05 14:20:47 +02:00
2010-12-06 15:58:43 -05:00
2014-10-30 22:27:18 -07:00
2011-10-31 00:33:36 +08:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2012-03-04 17:54:34 -05:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-23 13:20:00 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2013-07-17 15:53:24 -07:00
2019-08-08 15:09:54 +02:00
2020-09-23 18:05:05 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2013-11-26 11:26:44 -07:00
2019-08-05 14:21:54 +02:00
2013-09-06 12:37:23 +08:00
2013-06-18 10:24:51 -07:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2012-02-28 16:02:54 +01:00
2010-08-10 13:47:42 -07:00
2019-08-05 14:20:47 +02:00
2008-01-30 13:32:42 +01:00
2013-09-16 12:49:17 -07:00
2013-06-07 13:49:53 -07:00
2013-09-16 12:49:19 -07:00
2019-08-05 14:22:44 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2013-11-26 11:26:45 -07:00
2019-08-05 14:20:47 +02:00
2020-09-23 17:28:13 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 10:44:47 +02:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2011-11-23 18:49:22 +01:00
2011-10-03 09:57:10 -07:00
2014-03-11 17:27:09 -06:00
2019-08-05 14:20:47 +02:00
2020-08-08 14:52:26 +02:00
2015-08-31 11:08:11 -07:00
2014-08-18 10:34:41 -07:00
2019-08-08 12:19:52 +02:00
2009-04-15 12:22:37 +02:00
2010-07-28 09:59:01 -04:00
2012-01-24 12:48:54 -08:00
2011-11-29 12:46:19 -05:00
2011-06-27 09:11:02 -04:00
2019-08-05 14:20:47 +02:00
2012-03-04 17:54:34 -05:00
2019-08-05 14:20:47 +02:00
2013-07-12 10:50:32 -07:00
2011-03-22 17:44:11 -07:00
2019-08-02 15:14:10 +02:00
2012-05-07 19:21:29 -04:00
2012-04-06 02:43:12 -04:00
2019-08-08 15:09:54 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 12:31:21 +02:00
2011-01-12 20:16:43 -05:00
2019-08-02 15:14:10 +02:00
2020-09-23 17:37:32 +02:00
2019-08-08 15:08:49 +02:00
2019-08-05 14:21:58 +02:00
2014-07-10 14:09:27 -07:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2013-07-19 17:42:04 -07:00
2005-04-16 15:20:36 -07:00
2019-08-02 15:14:10 +02:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2020-09-23 17:37:30 +02:00
2019-08-06 12:26:38 +02:00
2011-07-08 00:21:27 -05:00
2019-08-08 15:28:37 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 10:44:15 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2013-06-14 13:34:03 -07:00
2019-08-08 12:42:04 +02:00
2014-01-31 11:31:36 -08:00
2019-08-08 16:37:24 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 12:26:30 +02:00
2019-08-05 14:20:47 +02:00
2012-03-16 13:36:04 -07:00
2007-01-30 08:26:45 -08:00
2019-08-05 14:20:47 +02:00
2012-03-28 15:58:21 -07:00
2019-08-05 14:20:47 +02:00
2020-08-08 16:42:50 +02:00
2019-08-05 14:20:47 +02:00
2011-12-05 18:27:56 -05:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 09:12:32 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 11:41:21 +02:00
2020-08-08 16:42:41 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2011-01-31 21:17:41 -08:00
2020-03-07 17:53:58 +01:00
2019-08-02 15:14:10 +02:00
2014-05-04 17:58:19 -07:00
2010-11-25 10:42:59 +09:00
2020-08-08 16:42:41 +02:00
2015-09-15 15:39:48 +05:30
2019-08-05 14:20:47 +02:00
2020-09-23 17:28:13 +02:00
2009-04-21 13:41:48 -07:00
2020-08-08 16:42:40 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2020-09-23 17:38:55 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 12:31:22 +02:00
2020-09-23 17:28:13 +02:00
2019-08-05 14:20:47 +02:00
2007-05-17 05:23:04 -07:00
2012-03-05 15:49:42 -08:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2011-11-21 12:32:23 -08:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 10:41:58 +02:00
2019-08-05 14:20:47 +02:00
2007-07-17 10:23:03 -07:00
2019-08-08 12:42:04 +02:00
2011-11-04 17:22:14 -07:00
2011-05-24 10:01:37 +02:00
2015-06-30 12:52:32 +05:30
2019-08-05 14:22:45 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 10:44:38 +02:00
2012-03-16 10:38:24 -04:00
2019-08-02 15:14:10 +02:00
2011-01-10 19:04:08 +01:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2020-09-23 18:05:05 +02:00
2020-09-23 17:38:52 +02:00
2013-04-11 11:29:51 -07:00
2013-03-14 09:32:58 -07:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2014-09-24 12:31:25 +05:30
2013-07-17 12:21:23 -07:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2020-09-23 18:05:05 +02:00
2010-08-09 20:44:58 -07:00
2019-08-05 14:20:47 +02:00
2020-09-23 18:05:05 +02:00
2019-08-06 11:41:21 +02:00
2020-11-16 18:32:29 +01:00
2012-03-26 12:50:52 +10:30
2011-07-24 22:06:04 +09:30
2019-08-05 14:20:47 +02:00
2019-08-06 10:44:40 +02:00
2009-04-01 07:38:54 -04:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2013-11-22 22:38:45 -07:00
2013-12-06 17:17:51 -08:00
2013-08-12 11:19:00 -07:00
2014-01-21 16:47:06 -08:00
2019-08-02 15:14:10 +02:00
2015-09-13 06:52:56 -07:00
2015-09-08 17:05:31 +05:30
2019-08-02 15:14:10 +02:00
2014-05-26 14:03:44 +03:00
2016-01-05 17:20:05 +05:30
2013-12-10 16:29:37 -07:00
2014-11-11 14:48:15 +05:30
2014-08-13 14:35:04 +05:30
2014-05-04 20:17:00 -07:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-06 10:44:34 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2012-02-24 01:42:07 -08:00
2019-08-05 14:20:47 +02:00
2019-08-09 09:14:03 +02:00
2019-08-09 09:13:53 +02:00
2019-08-09 09:14:03 +02:00
2019-08-09 09:13:49 +02:00
2020-09-23 17:28:13 +02:00
2019-08-05 14:20:47 +02:00
2020-09-23 17:28:13 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-08 15:08:48 +02:00
2019-08-06 11:41:21 +02:00
2011-12-21 14:48:43 -08:00
2013-07-12 14:09:22 -07:00
2012-03-05 08:09:09 -07:00
2019-08-05 14:20:47 +02:00
2019-08-08 12:09:23 +02:00
2011-12-03 09:35:17 -08:00
2020-09-23 18:05:05 +02:00
2012-01-10 16:30:42 -08:00
2019-08-06 08:16:29 +02:00
2019-08-02 15:14:10 +02:00
2013-06-28 12:24:00 -07:00
2019-08-05 09:12:33 +02:00
2008-10-13 10:10:37 -07:00
2011-12-13 15:07:49 +00:00
2011-10-31 19:32:32 -04:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2020-03-07 14:01:44 +01:00
2014-06-22 16:43:52 +03:00
2015-09-11 14:44:24 +05:30
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 11:41:21 +02:00
2019-08-06 11:41:21 +02:00
2019-08-06 11:41:21 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2020-09-23 17:32:09 +02:00
2020-09-23 17:32:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2006-10-03 23:01:26 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-08 15:08:07 +02:00
2019-08-05 14:20:47 +02:00
2011-03-31 11:26:23 -03:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 09:13:24 +02:00
2019-08-05 14:20:47 +02:00
2011-05-23 13:01:00 -07:00
2019-08-05 14:22:31 +02:00
2014-07-10 14:09:27 -07:00
2015-06-08 03:45:59 -07:00
2014-03-18 16:33:15 -07:00
2014-01-16 18:16:12 -08:00
2014-03-25 13:24:56 -07:00
2008-09-02 19:21:38 -07:00
2020-09-23 17:28:13 +02:00
2020-09-23 17:32:07 +02:00
2019-08-08 15:09:49 +02:00
2010-02-10 17:47:17 -08:00
2019-08-05 14:20:47 +02:00
2019-08-08 12:42:04 +02:00
2014-07-21 21:46:54 +05:30
2014-07-21 21:46:54 +05:30
2014-07-21 21:44:54 +05:30
2014-01-31 12:28:43 -08:00
2019-08-02 15:14:10 +02:00
2012-02-22 15:50:28 -05:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2015-09-15 15:39:48 +05:30
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-08 15:08:49 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 11:41:21 +02:00
2020-09-23 18:05:05 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:22:41 +02:00
2019-08-02 15:14:10 +02:00
2009-10-30 08:27:25 +11:00
2020-12-10 21:34:01 +01:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2013-12-23 17:51:05 -08:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2010-10-31 10:40:39 -04:00
2020-09-23 17:37:32 +02:00
2019-08-08 12:09:25 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 09:13:25 +02:00
2020-05-20 09:44:20 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2014-02-25 13:46:47 +05:30
2013-07-11 01:35:06 -07:00
2019-08-09 12:04:49 +02:00
2019-08-05 09:12:31 +02:00
2019-08-05 14:20:47 +02:00
2014-07-22 14:33:44 -06:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2008-05-26 16:15:32 +02:00
2020-09-23 17:28:13 +02:00
2012-02-24 10:05:59 +01:00
2019-08-08 12:09:30 +02:00
2013-09-18 14:12:26 +08:00
2019-08-02 15:14:10 +02:00
2013-03-14 09:32:58 -07:00
2008-10-03 11:46:14 -05:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2020-09-23 18:05:05 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2014-11-13 14:53:50 +05:30
2020-12-10 21:34:01 +01:00
2013-10-07 12:28:53 +05:30
2019-08-05 14:20:47 +02:00
2012-02-20 12:48:47 -08:00
2019-08-05 14:20:47 +02:00
2019-08-06 12:24:29 +02:00
2020-12-12 21:36:55 +01:00
2013-07-02 15:12:24 +03:00
2019-08-05 14:20:47 +02:00
2014-01-14 21:47:08 -08:00
2019-08-05 14:20:47 +02:00
2019-08-06 12:26:30 +02:00
2011-04-26 14:01:42 -07:00
2011-09-19 11:35:58 -04:00
2012-03-20 21:29:40 -04:00
2019-08-05 14:20:47 +02:00
2013-11-17 11:31:01 +02:00
2013-09-20 11:53:24 -07:00
2020-12-12 21:36:54 +01:00
2019-08-08 15:08:48 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2020-03-07 14:01:44 +01:00
2015-09-11 14:44:24 +05:30
2015-09-11 14:44:24 +05:30
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-05 14:20:47 +02:00
2013-06-18 10:24:50 -07:00
2013-12-26 14:33:32 -08:00
2020-09-23 18:05:05 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-08 15:00:18 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-04 19:49:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-05 14:20:47 +02:00
2020-09-23 17:32:06 +02:00
2019-08-05 14:20:47 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-02 15:14:10 +02:00
2019-08-06 08:16:29 +02:00
2019-08-06 08:16:29 +02:00
2019-08-06 08:17:38 +02:00