From 535f59d8c07ac9f32fd55ae857fc6ae9f2d5407b Mon Sep 17 00:00:00 2001 From: zzz Date: Mon, 3 Jan 2022 07:26:45 -0500 Subject: [PATCH] New proposal 160 --- .../site/docs/applications/bittorrent.html | 36 +- i2p2www/spec/proposals/160-udp-trackers.rst | 378 ++++++++++++++++++ 2 files changed, 385 insertions(+), 29 deletions(-) create mode 100644 i2p2www/spec/proposals/160-udp-trackers.rst diff --git a/i2p2www/pages/site/docs/applications/bittorrent.html b/i2p2www/pages/site/docs/applications/bittorrent.html index cc1cb44d..1f2d4f8e 100644 --- a/i2p2www/pages/site/docs/applications/bittorrent.html +++ b/i2p2www/pages/site/docs/applications/bittorrent.html @@ -1,7 +1,7 @@ {% extends "global/layout.html" %} {% block title %}{% trans %}Bittorrent over I2P{% endtrans %}{% endblock %} -{% block lastupdated %}{% trans %}January 2017{% endtrans %}{% endblock %} -{% block accuratefor %}0.9.28{% endblock %} +{% block lastupdated %}2022-01{% endblock %} +{% block accuratefor %}0.9.52{% endblock %} {% block content %}

{% trans -%} @@ -71,7 +71,9 @@ There are no known I2P clients or trackers that currently support UDP announce/r

{% trans %}Non-Compact Tracker Responses{% endtrans %}

{% trans -%} The non-compact response is just as in standard bittorrent, with an I2P "ip". -{%- endtrans %}

+{%- endtrans %} +This is a long base64-encoded "DNS string", probably with a ".i2p" suffix. +

{% trans -%} Trackers generally include a fake port key, or use the port from the announce, for compatibility with older clients. @@ -296,32 +298,8 @@ are described below, and are subject to change. Contact the I2P developers if you wish to develop a client or tracker supporting datagram announces. {%- endtrans %}

-

{% trans -%} -A UDP tracker listens on two ports. -The "query port" is the advertised port, and is used to receive repliable (signed) datagrams, for the connect request only. -The "response port" is used to receive unsigned (raw) datagrams, and is the source port for all replies. -The response port is arbitrary. -A client sends and receives on a single port only. -It receives only unsigned (raw) datagrams. -Raw datagrams provides increased efficiency for replies since they contain tokens sent in the query, and need not be signed. -{%- endtrans %}

- -

{% trans -%} -In the announce request, the 4-byte IP is replaced by a 32-byte hash, and the port is still present, -although it may be ignored by the tracker. -In the announce response, each 4-byte IP and 2-byte port is replaced by a 32-byte hash (compact peer info), and no port is present. -The client sends the announce request and scrape request to the source port in the announce response packet. -The connect request, connect response, scrape request, scrape response, and error response are the same as in BEP 15. -{%- endtrans %}

- -

{% trans -%} -Source addresses in I2P cannot be spoofed, so it is possible to use a simplified protocol -with 2 packets instead of 4, omitting the connect request and response. -In this case, the announce request would be a repliable datagram sent to the tracker's query port, -and the tracker would not require a response port. -While this is more efficient, it would be more difficult to modify an existing tracker to support this mode. -The URL for the 4-packet-mode tracker would use standard "udp://" prefix. -The URL for a modified 2-packet-mode tracker would require a different prefix if both modes are supported in I2P. +

{% trans prop160=site_url('docs/spec/proposals/160) -%} +See Proposal 160. {%- endtrans %}

diff --git a/i2p2www/spec/proposals/160-udp-trackers.rst b/i2p2www/spec/proposals/160-udp-trackers.rst new file mode 100644 index 00000000..6404eff9 --- /dev/null +++ b/i2p2www/spec/proposals/160-udp-trackers.rst @@ -0,0 +1,378 @@ +================================ +UDP Trackers +================================ +.. meta:: + :author: zzz + :created: 2022-01-03 + :thread: http://zzz.i2p/topics/1634 + :lastupdated: 2022-01-03 + :status: Open + :target: 0.9.54 + +.. contents:: + + +Overview +======== + +This proposal is for implemention of UDP trackers in I2P. + + +Motivation +========== + +As the user base in general and the number of bittorrent users specifically continues to grow, +we need to make trackers and announces more efficient so that trackers are not overwhelemed. + +Bittorrent proposed UDP trackers in BEP 15 [BEP15]_ in 2008, and the vast majority +of trackers on clearnet are UDP-only. + +A preliminary proposal for UDP trackers in I2P was posted on our bittorrent spec page [SPEC]_ +in May 2014; this predated our formal proposal process, and it was never implemented. +This proposal simplifies the 2014 version. + +It is difficult to calculate the bandwidth savings of datagrams vs. streaming protocol. +A repliable request is about the same size as a streaming SYN, but the payload +is about 500 bytes smaller because the HTTP GET has a huge 600 byte +URL parameter string. +The raw reply is much smaller than a streaming SYN ACK, providing significant reduction +for a tracker's outbound traffic. + +Additionally, there should be implementation-specific memory reductions, +as datagrams require much less in-memory state than a streaming connection. + + + +Design +============ + +This proposal uses both repliable and raw datagrams, +as defined in [DATAGRAMS]_. + + +BEP 15 +------- + +The message flow in [BEP15]_ is as follows: + +.. raw:: html + + {% highlight %} +Client Tracker + Connect Req. -------------> + <-------------- Connect Resp. + Announce Req. -------------> + <-------------- Announce Resp. + Announce Req. -------------> + <-------------- Announce Resp. +{% endhighlight %} + +The connect phase is required to prevent IP address spoofing. +The tracker returns a connection ID that the client uses in subsequent announces. +This connection ID expires in one minute at the client, and in two minutes at the tracker. +This is not necessary in I2P because of repliable datagrams. + +We propose two mechanisms for I2P, compatibility mode and fast mode. + + +Compatibility Mode +------------------------- + +In compatibility mode, we keep the same message flow as BEP 15, +for ease of adoption in existing UDP-capable client code bases: + +.. raw:: html + + {% highlight %} +Client Tracker + Connect Req. -------------> (Repliable) + <-------------- Connect Resp. (Raw) + Announce Req. -------------> (Raw) + <-------------- Announce Resp. (Raw) + Announce Req. -------------> (Raw) + <-------------- Announce Resp. (Raw) + ... +{% endhighlight %} + +This mode is also useful if the client plans to send multiple announces +within one minute to a single tracker, as only the connect +message must be repliable. + + +I2P Fast Mode +------------------------- + +In fast mode, we omit the connect phase, as it is not required to prevent address spoofing. +for ease of adoption in existing UDP-capable client code bases: + +.. raw:: html + + {% highlight %} +Client Tracker + Announce Req. -------------> (Repliable) + <-------------- Announce Resp. (Raw) +{% endhighlight %} + +This mode omits a round-trip, but requires every announce request to be repliable. + + + +Specification +============= + +Repliable datagrams use I2CP protocol 17; raw datagrams use I2CP protocol 18. +Requests may be repliable or raw. Responses are always raw. + + +Connect Request +----------------- + +Client to tracker. +16 bytes. Must be repliable. Same as in [BEP15]_. + + +.. raw:: html + + {% highlight %} +Offset Size Name Value + 0 64-bit integer protocol_id 0x41727101980 // magic constant + 8 32-bit integer action 0 // connect + 12 32-bit integer transaction_id +{% endhighlight %} + + + +Connect Response +----------------- + +Tracker to client. +16 bytes. Must be raw. Same as in [BEP15]_. + + +.. raw:: html + + {% highlight %} +Offset Size Name Value + 0 32-bit integer action 0 // connect + 4 32-bit integer transaction_id + 8 64-bit integer connection_id +{% endhighlight %} + +The response MUST be sent to the I2CP "to port" that was received as the request "from port". + + + + +Announce Request +----------------- + +Client to tracker. +98 bytes. Same as in [BEP15]_ except as noted below. + +If preceded by a connect request/response, must be raw, +with the connection_id received in the connect response. + + +If NOT preceded by a connect request/response, must be repliable, +and the connection_id is ignored. + + +.. raw:: html + + {% highlight %} +Offset Size Name Value + 0 64-bit integer connection_id + 8 32-bit integer action 1 // announce + 12 32-bit integer transaction_id + 16 20-byte string info_hash + 36 20-byte string peer_id + 56 64-bit integer downloaded + 64 64-bit integer left + 72 64-bit integer uploaded + 80 32-bit integer event 0 // 0: none; 1: completed; 2: started; 3: stopped + 84 32-bit integer IP address 0 // default + 88 32-bit integer key + 92 32-bit integer num_want -1 // default + 96 16-bit integer port +{% endhighlight %} + +Changes from [BEP15]_: + +- connection_id is ignored if repliable +- IP address is ignored +- key is ignored +- port is probably ignored + +The response MUST be sent to the I2CP "to port" that was received as the request "from port". +Do not use the port from the announce request. + + + + + +Announce Response +----------------- + +Tracker to client. +20+ bytes. Must be raw. Same as in [BEP15]_ except as noted below. + + + +.. raw:: html + + {% highlight %} +Offset Size Name Value + 0 32-bit integer action 1 // announce + 4 32-bit integer transaction_id + 8 32-bit integer interval + 12 32-bit integer leechers + 16 32-bit integer seeders + 20 + 32 * n 32-bit integer binary hash +{% endhighlight %} + +Changes from [BEP15]_: + +- Instead of 6-byte IPv4+port or 18-byte IPv6+port, we return + a multiple of 32-byte "compact responses" with the binary peer hashes. + +The response MUST be sent to the I2CP "to port" that was received as the request "from port". +Do not use the port from the announce request. + +I2P datagrams have a very large maximum size of about 16 KB; +however, for reliable delivery, datagrams larger than 4 KB should be avoided. +For bandwidth efficiency, trackers should probably limit the maximum peers +to about 50. + + +Scrape +---------- + +Scrape request/response from [BEP15]_ is not required by this proposal, +but may be implemented if desired, no changes required. +The scrape request is always repliable (unless there is a previous connect request/response) +and the scrape response is always raw. + + +Error Response +------------------ + +Error response from [BEP15]_ is not required by this proposal, +but may be implemented if desired, no changes required. +The error response is always raw. + + +Announce URL +------------ + +As in clearnet, UDP announce URLs are of the form "udp://host:port/path". +The path is ignored and may be empty. +If the ":port" part is omitted, use an I2CP port of 0. + + + +Issues +======= + +- Repliable datagrams do not support offline signatures. + That requires a separate proposal. +- This proposal does not support blinded destinations. +- This proposal offers two modes at the client's option. + An existing clearnet tracker such as "opentracker" would require more modifications + to support the fast mode. There is no way in the announce URL to indicate + support for only one mode. +- Compatibility mode may not be necessary, pending feedback from BiglyBT and + other developers. However, it would still save a lot of bandwidth + if it is used for several announces within a minute. + Repliable announces are about 450 bytes larger than raw announces. + + +Extensions +============= + +Extension bits or a version field are not included. +Clients and trackers should not assume packets to be of a certain size. +This way, additional fields can be added without breaking compatibility. + + + +Implementation guidelines +========================== + +Clients +-------- + +For a given tracker hostname, a client should prefer UDP over HTTP URLs, +and should not announce to both. + +Clients wihout existing BEP 15 support should implement +fast mode only, as it is much simpler. +Clients with existing BEP 15 support should require only small modifications. +Evaluate both fast and compatibility modes and choose +whatever is best for the existing code base. + +If a client support DHT or other datagram protocols, it should probably +select a different port as the request "from port" so that the replies +come back to that port and are not mixed up with DHT messages. + +Clients with a default list of opentrackers should update the list to +add UDP URLs after the known opentrackers are known to support UDP. + +Clients may or may not implement retransmission of requests. +Retransmissions, if implemented, should use an initial timeout +of at least 15 seconds, and double the timeout for each retransmission +(exponential backoff). + + +Trackers +--------- + +Trackers must implement both compatibility mode and fast mode. +Trackers with existing BEP 15 support should require only small modifications. + +For an integrated application (router and client in one process, for example the ZzzOT Java plugin), +it should be straightforward to implement and route the streaming and datagram traffic separately. + +For an external tracker application that currently uses an HTTP server tunnel to receive +announce requests, the implementation could be quite difficult. +A specialized tunnel could be developed to translate datagrams to local HTTP requests/responses. +Or, a specialized tunnel that handles both HTTP requests and datagrams could be designed +that would forward the datagrams to the external process. +These design decisions will depend heavily on the specific router and tracker implementations, +and are outside the scope of this proposal. + + + + +Migration +========= + +Existing clients do not support UDP announce URLs and ignore them. + +Existing trackers do not support reception of repliable or raw datagrams, they will be dropped. + +This proposal is completely optional. Neither clients nor trackers are required to implement it at any time. + + + +Rollout +======= + +The first implementations are expected to be in ZzzOT and i2psnark. +They will be used for testing and verification of this proposal. + +Other implementations will follow as desired after the testing and verification are complete. + + + + +References +========== + +.. [BEP15] + http://www.bittorrent.org/beps/bep_0015.html + +.. [DATAGRAMS] + {{ spec_url('datagrams', True) }} + +.. [SPEC] + {{ site_url('docs/applications/bittorrent', True) }}