https://g2.doxu.org/index.php?title=Partial_File_Sharing_Protocol&feed=atom&action=historyPartial File Sharing Protocol - Revision history2024-03-28T11:47:05ZRevision history for this page on the wikiMediaWiki 1.38.4https://g2.doxu.org/index.php?title=Partial_File_Sharing_Protocol&diff=2868&oldid=prevDcat: Reverted edit of ZmkXk2, changed back to last version by Painlord2k2007-10-21T19:13:37Z<p>Reverted edit of ZmkXk2, changed back to last version by Painlord2k</p>
<p><b>New page</b></p><div>Source - [http://www.the-gdf.org/wiki/index.php?title=Partial_File_Sharing_Protocol]<br />
<br />
Partial File Sharing Protocol Version (PFSP) 1.0<br><br />
Tor Klingberg &lt;tor.klingberg@gmx.net&gt;<br><br />
August 2002<br><br />
<br />
'''Introduction'''<br />
<br />
This is a protocol for sharing partial files on the Gnutella network. A<br />
partial file is a file that a host has only downloaded parts of. Partial <br />
File Sharing allows files to spread faster over the Gnutella network.<br />
Here, the server is the host that is providing the file, and client is the<br />
host that requests the file.<br />
<br />
<br />
== Partial File Transfer ==<br />
<br />
The server allows HTTP requests for partial files, at URIs chosen by the<br />
server. They can for example be assigned a file index and shared at<br />
"/get/index/filename", or simply at "/partials/filename". If requests <br />
by URN are supported, the best way is probably to share only at <br />
"uri-res/N2R?urn:sha1:HASH_OF_COMPLETE_FILE". Servers should make sure <br />
that the URI to a partial file does not become invalid when the file is <br />
completed.<br />
<br />
Only partial requests (with a ''Range'' header) are accepted.<br />
<br />
The X-Available-Ranges header is used by the server to inform the client<br />
about what ranges are available. Note that a 2xx or 503 response without <br />
an X-Available-Ranges header means the complete file is available. The<br />
format is as follows:<br />
<br />
X-Available-Ranges: bytes 0-10,20-30<br />
<br />
The client requests the range it wants using the ''Range'' header.<br />
<br />
Range: bytes=0-<br />
means the client wants any ranges the server can provide.<br />
<br />
The server then provides the range it wants to upload using a 206 Partial<br />
Content response. This allows the server to upload different ranges to<br />
different hosts, and save bandwidth by allowing them to get the other parts<br />
from each other. The server can decide to upload any range inside the<br />
requested range. This means that the client cannot be sure that the first<br />
byte in the response is first requested byte.<br />
<br />
The 206 response contains a Content-Range header on the form<br />
<br />
Content-Range: bytes &lt;start&gt;-&lt;end&gt;/&lt;total_size&gt;<br />
<br />
Note that &lt;total_size&gt; is the size of the '''complete''' file.<br />
<br />
If the server is unable to provide any part of the requested range, it<br />
returns a "503 Requested Range Not Available" (the Reason Phrase is just my<br />
recommendation). If the client continues to request the same range, the<br />
server may send a 404 to make a PFSP unaware client stop retrying.<br />
The X-Available-Ranges header will tell a PFSP enabled client what ranges it<br />
can request.<br />
<br />
If the client provides an "Accept:" header with "multipart/byteranges" in<br />
it, the server may respond with multiple ranges at once. The client may send<br />
multiple ranges in the Range: header if it sends an ''Accept'' header with<br />
multipart/byteranges in the same header set. This is standard HTTP/1.1<br />
stuff, but I doubt that Gnutella servents will support it. If you do not<br />
want multipart support, just ignore it and everything will work fine.<br />
<br />
You should, however, be aware that there can be multiple ranges specified in<br />
one "Range:" header. Servents are then allowed to choose any range within<br />
the specified ranges, or simply read the first range only.<br />
<br />
== Tree Hashes ==<br />
<br />
Tree hashes are not absolutely required for Partial File Sharing, so you<br />
don't have to implement this part at first. TigerTree can be implemented<br />
if/when corrupt files become a problem. The reason that it is in this<br />
document is because Partial File Sharing might cause corrupt files to spread<br />
faster.<br />
<br />
TigerTree hashes are computed using a 1024 byte base size. It is then up to<br />
each vendor to decide how many sub-hashes to actually store. Storing (and<br />
advertising) the top 10 levels of the tree might be good decision. It would<br />
allow a resolution of about 2 MB on a 1 GB file, and requires only about<br />
25 kB of hash data per file.<br />
<br />
The tree is provided as specified in the '''T'''ree '''H'''ash '''EX'''change format ([http://www.open-content.net/specs/draft-jchapweske-thex-01.html THEX]).<br />
It basically says that the hash tree is provided as a long stream of binary<br />
data starting with the root hash, then the two hashes it is computed from,<br />
and so on.<br />
<br />
To inform the client about where the hash tree can be retrieved the server<br />
includes an X-Thex-URI header on this form<br />
<br />
X-Thex-URI: &lt;URI&gt; ; &lt;ROOT&gt;<br />
<br />
<URI> is any valid URI. It can be to an uri-res translator, and can even<br />
point to another host. The client can then retrieve desired parts of the<br />
hash tree by doing range requests for the specified URI.<br />
<br />
The THEX data is shared as if it was a partial file. If a client requests a<br />
subrange of the THEX data that the server does not store, and is not willing<br />
to calculate on the fly, the server uses the same routines as if it was a<br />
partial file where the requested range is not available.<br />
<br />
&lt;ROOT&gt; is the root TigerTree hash in base32 (RFC 3548) encoding.<br />
<br />
== How to find the location of partial files ==<br />
<br />
This protocol does not affect Gnutella messages in any way. The only<br />
available mean of spreading the location of a partial file is through the<br />
download mesh in X-Gnutella-Alternate-Location headers. I think this should<br />
work very well. Since those who share a partial file are also downloading<br />
the same file, they will be able to send alt-loc headers to other hosts<br />
sharing the full file.<br />
<br />
Spreading partial files in the download mesh will cause servents that do<br />
not support partial file sharing to receive addresses to partial sources. I<br />
don't think that is a problem. The worst thing that can happen is that they<br />
won't be able to use those sources.<br />
<br />
When requesting a file for download, the client must include an alt-loc <br />
header pointing to its locally shared partial file, if there is not a good <br />
reason not to do so in the particular case. If the client is firewalled<br />
(and push proxy URIs are not available) that is a reason. Clients should <br />
do this even if the download has not started yet. Since it takes a while <br />
for alt-locs to spread, the download is likely to have started when <br />
someone else get the alt-loc. If not, a few failed download requests is <br />
not a big problem.<br />
<br />
<br />
== Sample negotiation ==<br />
<br />
Here is a sample negotiation. I don't think it will look exactly like this,<br />
but it should show the headers in action. Clients might want to request a<br />
small range first, to get the list of available ranges. There are some<br />
linebreakes in long headers below.<br />
<br />
Client:<br />
<br />
<pre><br />
GET /uri-res/N2R?urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X HTTP/1.1<br />
User-Agent: FooBar/1.0<br />
Host: 192.0.2.65:6346<br />
Connection: Keep-Alive<br />
Range: bytes=73826-<br />
X-Gnutella-Content-URN: urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X<br />
X-Alt: 192.0.2.99:6348, 192.0.2.199, 192.0.2.14<br />
</pre><br />
<br />
Server:<br />
<br />
<pre><br />
HTTP/1.1 206 Partial Content<br />
Server: FooBar/1.0<br />
Content-Type: audio/mpeg<br />
Content-Range: bytes 73826-285749/533273<br />
Content-Length: 211924<br />
Connection: Keep-Alive<br />
X-Available-Ranges: bytes 0-285749,425926-488271<br />
X-Gnutella-Content-URN: urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X<br />
X-Thex-URI: <br />
/uri-res/N2X?urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X;VEKXTRSJPTZJLY2IKG5FQ2TCXK26SECFPP4DX7I<br />
X-Gnutella-Alternate-Location: &lt;list of alt-locs&gt;<br />
<br />
&lt;211924 bytes of data&gt;<br />
</pre><br />
<br />
"N2X" above is an example. Someone should comment on what should be used.<br />
Since the URI is provided in the X-Thex-URI header, each vendor can chose<br />
how to provide the THEX data.</div>Dcathttps://g2.doxu.org/index.php?title=Partial_File_Sharing_Protocol&diff=1903&oldid=prevPainlord2k at 18:09, 7 September 20052005-09-07T18:09:53Z<p></p>
<p><b>New page</b></p><div>Source - [http://www.the-gdf.org/wiki/index.php?title=Partial_File_Sharing_Protocol]<br />
<br />
Partial File Sharing Protocol Version (PFSP) 1.0<br><br />
Tor Klingberg &lt;tor.klingberg@gmx.net&gt;<br><br />
August 2002<br><br />
<br />
'''Introduction'''<br />
<br />
This is a protocol for sharing partial files on the Gnutella network. A<br />
partial file is a file that a host has only downloaded parts of. Partial <br />
File Sharing allows files to spread faster over the Gnutella network.<br />
Here, the server is the host that is providing the file, and client is the<br />
host that requests the file.<br />
<br />
<br />
== Partial File Transfer ==<br />
<br />
The server allows HTTP requests for partial files, at URIs chosen by the<br />
server. They can for example be assigned a file index and shared at<br />
"/get/index/filename", or simply at "/partials/filename". If requests <br />
by URN are supported, the best way is probably to share only at <br />
"uri-res/N2R?urn:sha1:HASH_OF_COMPLETE_FILE". Servers should make sure <br />
that the URI to a partial file does not become invalid when the file is <br />
completed.<br />
<br />
Only partial requests (with a ''Range'' header) are accepted.<br />
<br />
The X-Available-Ranges header is used by the server to inform the client<br />
about what ranges are available. Note that a 2xx or 503 response without <br />
an X-Available-Ranges header means the complete file is available. The<br />
format is as follows:<br />
<br />
X-Available-Ranges: bytes 0-10,20-30<br />
<br />
The client requests the range it wants using the ''Range'' header.<br />
<br />
Range: bytes=0-<br />
means the client wants any ranges the server can provide.<br />
<br />
The server then provides the range it wants to upload using a 206 Partial<br />
Content response. This allows the server to upload different ranges to<br />
different hosts, and save bandwidth by allowing them to get the other parts<br />
from each other. The server can decide to upload any range inside the<br />
requested range. This means that the client cannot be sure that the first<br />
byte in the response is first requested byte.<br />
<br />
The 206 response contains a Content-Range header on the form<br />
<br />
Content-Range: bytes &lt;start&gt;-&lt;end&gt;/&lt;total_size&gt;<br />
<br />
Note that &lt;total_size&gt; is the size of the '''complete''' file.<br />
<br />
If the server is unable to provide any part of the requested range, it<br />
returns a "503 Requested Range Not Available" (the Reason Phrase is just my<br />
recommendation). If the client continues to request the same range, the<br />
server may send a 404 to make a PFSP unaware client stop retrying.<br />
The X-Available-Ranges header will tell a PFSP enabled client what ranges it<br />
can request.<br />
<br />
If the client provides an "Accept:" header with "multipart/byteranges" in<br />
it, the server may respond with multiple ranges at once. The client may send<br />
multiple ranges in the Range: header if it sends an ''Accept'' header with<br />
multipart/byteranges in the same header set. This is standard HTTP/1.1<br />
stuff, but I doubt that Gnutella servents will support it. If you do not<br />
want multipart support, just ignore it and everything will work fine.<br />
<br />
You should, however, be aware that there can be multiple ranges specified in<br />
one "Range:" header. Servents are then allowed to choose any range within<br />
the specified ranges, or simply read the first range only.<br />
<br />
== Tree Hashes ==<br />
<br />
Tree hashes are not absolutely required for Partial File Sharing, so you<br />
don't have to implement this part at first. TigerTree can be implemented<br />
if/when corrupt files become a problem. The reason that it is in this<br />
document is because Partial File Sharing might cause corrupt files to spread<br />
faster.<br />
<br />
TigerTree hashes are computed using a 1024 byte base size. It is then up to<br />
each vendor to decide how many sub-hashes to actually store. Storing (and<br />
advertising) the top 10 levels of the tree might be good decision. It would<br />
allow a resolution of about 2 MB on a 1 GB file, and requires only about<br />
25 kB of hash data per file.<br />
<br />
The tree is provided as specified in the '''T'''ree '''H'''ash '''EX'''change format ([http://www.open-content.net/specs/draft-jchapweske-thex-01.html THEX]).<br />
It basically says that the hash tree is provided as a long stream of binary<br />
data starting with the root hash, then the two hashes it is computed from,<br />
and so on.<br />
<br />
To inform the client about where the hash tree can be retrieved the server<br />
includes an X-Thex-URI header on this form<br />
<br />
X-Thex-URI: &lt;URI&gt; ; &lt;ROOT&gt;<br />
<br />
<URI> is any valid URI. It can be to an uri-res translator, and can even<br />
point to another host. The client can then retrieve desired parts of the<br />
hash tree by doing range requests for the specified URI.<br />
<br />
The THEX data is shared as if it was a partial file. If a client requests a<br />
subrange of the THEX data that the server does not store, and is not willing<br />
to calculate on the fly, the server uses the same routines as if it was a<br />
partial file where the requested range is not available.<br />
<br />
&lt;ROOT&gt; is the root TigerTree hash in base32 (RFC 3548) encoding.<br />
<br />
== How to find the location of partial files ==<br />
<br />
This protocol does not affect Gnutella messages in any way. The only<br />
available mean of spreading the location of a partial file is through the<br />
download mesh in X-Gnutella-Alternate-Location headers. I think this should<br />
work very well. Since those who share a partial file are also downloading<br />
the same file, they will be able to send alt-loc headers to other hosts<br />
sharing the full file.<br />
<br />
Spreading partial files in the download mesh will cause servents that do<br />
not support partial file sharing to receive addresses to partial sources. I<br />
don't think that is a problem. The worst thing that can happen is that they<br />
won't be able to use those sources.<br />
<br />
When requesting a file for download, the client must include an alt-loc <br />
header pointing to its locally shared partial file, if there is not a good <br />
reason not to do so in the particular case. If the client is firewalled<br />
(and push proxy URIs are not available) that is a reason. Clients should <br />
do this even if the download has not started yet. Since it takes a while <br />
for alt-locs to spread, the download is likely to have started when <br />
someone else get the alt-loc. If not, a few failed download requests is <br />
not a big problem.<br />
<br />
<br />
== Sample negotiation ==<br />
<br />
Here is a sample negotiation. I don't think it will look exactly like this,<br />
but it should show the headers in action. Clients might want to request a<br />
small range first, to get the list of available ranges. There are some<br />
linebreakes in long headers below.<br />
<br />
Client:<br />
<br />
<pre><br />
GET /uri-res/N2R?urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X HTTP/1.1<br />
User-Agent: FooBar/1.0<br />
Host: 192.0.2.65:6346<br />
Connection: Keep-Alive<br />
Range: bytes=73826-<br />
X-Gnutella-Content-URN: urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X<br />
X-Alt: 192.0.2.99:6348, 192.0.2.199, 192.0.2.14<br />
</pre><br />
<br />
Server:<br />
<br />
<pre><br />
HTTP/1.1 206 Partial Content<br />
Server: FooBar/1.0<br />
Content-Type: audio/mpeg<br />
Content-Range: bytes 73826-285749/533273<br />
Content-Length: 211924<br />
Connection: Keep-Alive<br />
X-Available-Ranges: bytes 0-285749,425926-488271<br />
X-Gnutella-Content-URN: urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X<br />
X-Thex-URI: <br />
/uri-res/N2X?urn:sha1:QLFYWY2RI5WZCTEP6MJKR5CAFGP7FQ5X;VEKXTRSJPTZJLY2IKG5FQ2TCXK26SECFPP4DX7I<br />
X-Gnutella-Alternate-Location: &lt;list of alt-locs&gt;<br />
<br />
&lt;211924 bytes of data&gt;<br />
</pre><br />
<br />
"N2X" above is an example. Someone should comment on what should be used.<br />
Since the URI is provided in the X-Thex-URI header, each vendor can chose<br />
how to provide the THEX data.</div>Painlord2k