diff --git a/ChangeLog b/ChangeLog index 53f8211d7..a1fc8b1ec 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,4 +1,6 @@ + * files that are being checked will no longer stall files that don't need + checking. * changed the way libtorrent identifies support for its excentions to look for 'ext' at the end of the peer-id. * improved performance by adding a circle buffer for the send buffer diff --git a/docs/manual.html b/docs/manual.html index 5ae8bc817..119605632 100755 --- a/docs/manual.html +++ b/docs/manual.html @@ -5,7 +5,7 @@ libtorrent manual - + @@ -16,143 +16,145 @@ Author: -Arvid Norberg, c99ang@cs.umu.se +Arvid Norberg, arvid@rasterbar.com

Table of contents

@@ -227,7 +229,7 @@ Boost.Filesystem, Boost.Date_time and various other boost libraries as well as z

Fails on:

@@ -400,11 +402,30 @@ checking for main in -lboost_thread... yes directory contains spaces. Make sure you either rename the directories with spaces in their names to remove the spaces or move the libtorrent directory.

+
+

Creating a debug build

+

To tell configure to build a debug version (with debug info, asserts +and invariant checks enabled), you have to run the configure script +with the following option:

+
+./configure --enable-debug=yes
+
+
+
+

Creating a release build

+

To tell the configure to build a release version (without debug info, +asserts and invariant checks), you have to run the configure script +with the following option:

+
+./configure --enable-debug=no
+
+

The above option make use of -DNDEBUG, which is used throughout libtorrent.

+

Step 2: Building libtorrent

Once the configure script is run successfully, you just type make and libtorrent, the examples and the tests will be built.

-

When libtorrent is built it may be a good idea to run the test, you do this +

When libtorrent is built it may be a good idea to run the tests, you do this my running make check.

If you want to build a release version (without debug info, asserts and invariant checks), you have to rerun the configure script and rebuild, like this:

@@ -416,7 +437,7 @@ make
-

Building with other build systems

+

building with other build systems

If you're making your own project file, note that there are two versions of the file abstraction. There's one file_win.cpp which relies on windows file API that supports files larger than 2 Gigabytes. This does not work in @@ -431,7 +452,7 @@ options "force conformance in for loop scope", "treat wchar_t as type" and "Enable Run-Time Type Info" to Yes.

-

Build configurations

+

build configurations

By default libtorrent is built In debug mode, and will have pretty expensive invariant checks and asserts built into it. If you want to disable such checks (you want to do that in a release build) you can see the table below for which @@ -474,10 +495,21 @@ UTF-8 strings in pathnames are converted into UTF-16 before they are passed to the file operations. +LITTLE_ENDIAN +This will use the little endian version of the +sha-1 code. If defined on a big-endian system +the sha-1 hashes will be incorrect and fail. +If it is not defined and __BIG_ENDIAN__ +isn't defined either (it is defined by Apple's +GCC) both little-endian and big-endian versions +will be built and the correct code will be +chosen at run-time. + -

If you experience that libtorrent uses unreasonable amounts of cpu, it will definately help to -define NDEBUG, since it will remove the invariant checks within the library.

+

If you experience that libtorrent uses unreasonable amounts of cpu, it will +definately help to define NDEBUG, since it will remove the invariant checks +within the library.

@@ -514,7 +546,8 @@ the session, it conta class session: public boost::noncopyable { - session(const fingerprint& print = libtorrent::fingerprint("LT", 0, 1, 0, 0)); + session(const fingerprint& print + = libtorrent::fingerprint("LT", 0, 1, 0, 0)); session( const fingerprint& print @@ -888,18 +921,64 @@ can assign the value you want it to have.

entry torrent_file; // ... +// throws if this is not a dictionary entry::dictionary_type const& dict = torrent_file.dict(); entry::dictionary_type::const_iterator i; i = dict.find("announce"); if (i != dict.end()) { - std::string tracker_url= i->second.string(); + std::string tracker_url = i->second.string(); std::cout << tracker_url << "\n"; } -

To make it easier to extract information from a torren file, the class torrent_info +

The following code is equivalent, but a little bit shorter:

+
+entry torrent_file;
+// ...
+
+// throws if this is not a dictionary
+if (entry* i = torrent_file.find_key("announce"))
+{
+        std::string tracker_url = i->string();
+        std::cout << tracker_url << "\n";
+}
+
+

To make it easier to extract information from a torrent file, the class torrent_info exists.

+
+

operator[]

+
+
+entry& operator[](char const* key);
+entry& operator[](std::string const& key);
+entry const& operator[](char const* key) const;
+entry const& operator[](std::string const& key) const;
+
+
+

All of these functions requires the entry to be a dictionary, if it isn't they +will throw libtorrent::type_error.

+

The non-const versions of the operator[] will return a reference to either +the existing element at the given key or, if there is no element with the +given key, a reference to a newly inserted element at that key.

+

The const version of operator[] will only return a reference to an +existing element at the given key. If the key is not found, it will throw +libtorrent::type_error.

+
+
+

find_key()

+
+
+entry* find_key(char const* key);
+entry const* find_key(char const* key) const;
+
+
+

These functions requires the entry to be a dictionary, if it isn't they +will throw libtorrent::type_error.

+

They will look for an element at the given key in the dictionary, if the +element cannot be found, they will return 0. If an element with the given +key is found, the return a pointer to it.

+

torrent_info

@@ -922,7 +1001,8 @@ public: void add_file(boost::filesystem::path file, size_type size); typedef std::vector<file_entry>::const_iterator file_iterator; - typedef std::vector<file_entry>::const_reverse_iterator reverse_file_iterator; + typedef std::vector<file_entry>::const_reverse_iterator + reverse_file_iterator; file_iterator begin_files() const; file_iterator end_files() const; @@ -990,12 +1070,12 @@ void add_file(boost::filesystem::path file, size_type size);

These files are used when creating a torrent file. set_comment() will simply set the comment that belongs to this torrent. The comment can be retrieved with the -comment() member.

+comment() member. The string should be UTF-8 encoded.

set_piece_size() will set the size of each piece in this torrent. The piece size must be an even multiple of 2. i.e. usually something like 256 kiB, 512 kiB, 1024 kiB etc. The size is given in number of bytes.

set_creator() is an optional attribute that can be used to identify your application -that was used to create the torrent file.

+that was used to create the torrent file. The string should be UTF-8 encoded.

set_hash() writes the hash for the piece with the given piece-index. You have to call this function for every piece in the torrent. Usually the hasher is used to calculate the sha1-hash for a piece.

@@ -1037,7 +1117,8 @@ in the torrent, you can use begin rbegin_files() and rend_files(). These will give you standard vector iterators with the type file_entry.

The path is the full (relative) path of each file. i.e. if it is a multi-file -torrent, all the files starts with a directory with the same name as torrent_info::name().

+torrent, all the files starts with a directory with the same name as torrent_info::name(). +The filenames are encoded with UTF-8.

 struct file_entry
 {
@@ -1131,6 +1212,7 @@ boost::optional<boost::posix_time::ptime> creation_date() const;
 it will return an empty string. creation_date() returns a boost::posix_time::ptime
 object, representing the time when this torrent file was created. If there's no timestamp
 in the torrent file, this will return a date of january 1:st 1970.

+

Both the name and the comment is UTF-8 encoded strings.

creator() returns the creator string in the torrent. If there is no creator string it will return an empty string.

@@ -1156,7 +1238,8 @@ struct torrent_handle void force_reannounce(); void connect_peer(address const& adr) const; - void set_tracker_login(std::string const& username, std::string const& password); + void set_tracker_login(std::string const& username + , std::string const& password); std::vector<announce_entry> const& trackers() const; void replace_trackers(std::vector<announce_entry> const&); @@ -1960,7 +2043,8 @@ to encode this information into the client's peer id.

 struct fingerprint
 {
-        fingerprint(const char* id_string, int major, int minor, int revision, int tag);
+        fingerprint(const char* id_string, int major, int minor
+                , int revision, int tag);
 
         std::string to_string() const;
 
@@ -2009,6 +2093,7 @@ sure not to clash with anybody else. Here are some taken id's:

+

There's currently an informal directory of client id's here.

The major, minor, revision and tag parameters are used to identify the version of your client. All these numbers must be within the range [0, 9].

to_string() will generate the actual string put in the peer-id, and return it.

@@ -2514,14 +2599,15 @@ int main(int argc, char* argv[]) { std::ifstream in(argv[1], std::ios_base::binary); in.unsetf(std::ios_base::skipws); - entry e = bdecode(std::istream_iterator<char>(in), std::istream_iterator<char>()); + entry e = bdecode(std::istream_iterator<char>(in) + , std::istream_iterator<char>()); torrent_info t(e); // print info about torrent std::cout << "\n\n----- torrent file info -----\n\n"; std::cout << "trackers:\n"; - for (std::vector<announce_entry>::const_iterator i = t.trackers().begin(); - i != t.trackers().end(); ++i) + for (std::vector<announce_entry>::const_iterator i + = t.trackers().begin(), end(t.trackers().end); i != end; ++i) { std::cout << i->tier << ": " << i->url << "\n"; } @@ -2530,8 +2616,7 @@ int main(int argc, char* argv[]) std::cout << "piece length: " << t.piece_length() << "\n"; std::cout << "files:\n"; for (torrent_info::file_iterator i = t.begin_files(); - i != t.end_files(); - ++i) + i != t.end_files(); ++i) { std::cout << " " << std::setw(11) << i->size << " " << i->path << " " << i->filename << "\n"; @@ -2582,7 +2667,8 @@ int main(int argc, char* argv[]) std::ifstream in(argv[1], std::ios_base::binary); in.unsetf(std::ios_base::skipws); - entry e = bdecode(std::istream_iterator<char>(in), std::istream_iterator<char>()); + entry e = bdecode(std::istream_iterator<char>(in) + , std::istream_iterator<char>()); s.add_torrent(e, ""); // wait for the user to end @@ -2621,10 +2707,7 @@ int main(int argc, char* argv[]) using namespace boost::filesystem; using namespace libtorrent; -void add_files( - torrent_info& t - , path const& p - , path const& l) +void add_files(torrent_info& t, path const& p, path const& l) { path f(p / l); if (is_directory(f)) @@ -2649,8 +2732,8 @@ int main(int argc, char* argv[]) if (argc != 4) { - std::cerr << "usage: make_torrent <output torrent-file> <announce url> " - "<file or directory to create torrent from>\n"; + std::cerr << "usage: make_torrent <output torrent-file> " + "<announce url> <file or directory to create torrent from>\n"; return 1; } diff --git a/docs/manual.rst b/docs/manual.rst index 80b316f9a..2a817c051 100755 --- a/docs/manual.rst +++ b/docs/manual.rst @@ -2,7 +2,7 @@ libtorrent manual ================= -:Author: Arvid Norberg, c99ang@cs.umu.se +:Author: Arvid Norberg, arvid@rasterbar.com .. contents:: Table of contents :depth: 2 @@ -82,7 +82,7 @@ libtorrent has been successfully compiled and tested on: Fails on: - * GCC 2.95.4 (``std::ios_base`` is missing) + * GCC 2.95.4 * msvc6 sp5 libtorrent is released under the BSD-license_. @@ -318,7 +318,15 @@ libtorrent, the examples and the tests will be built. When libtorrent is built it may be a good idea to run the tests, you do this my running ``make check``. -Building with other build systems +If you want to build a release version (without debug info, asserts and +invariant checks), you have to rerun the configure script and rebuild, like this:: + + ./configure --disable-debug + make clean + make + + +building with other build systems --------------------------------- If you're making your own project file, note that there are two versions of @@ -336,7 +344,7 @@ options "force conformance in for loop scope", "treat wchar_t as built-in type" and "Enable Run-Time Type Info" to Yes. -Build configurations +build configurations -------------------- By default libtorrent is built In debug mode, and will have pretty expensive @@ -369,10 +377,20 @@ defines you can use to control the build. | | UTF-16 before they are passed to the file | | | operations. | +--------------------------------+-------------------------------------------------+ +| ``LITTLE_ENDIAN`` | This will use the little endian version of the | +| | sha-1 code. If defined on a big-endian system | +| | the sha-1 hashes will be incorrect and fail. | +| | If it is not defined and ``__BIG_ENDIAN__`` | +| | isn't defined either (it is defined by Apple's | +| | GCC) both little-endian and big-endian versions | +| | will be built and the correct code will be | +| | chosen at run-time. | ++--------------------------------+-------------------------------------------------+ -If you experience that libtorrent uses unreasonable amounts of cpu, it will definately help to -define ``NDEBUG``, since it will remove the invariant checks within the library. +If you experience that libtorrent uses unreasonable amounts of cpu, it will +definately help to define ``NDEBUG``, since it will remove the invariant checks +within the library. overview ======== @@ -406,7 +424,8 @@ The ``session`` class has the following synopsis:: class session: public boost::noncopyable { - session(const fingerprint& print = libtorrent::fingerprint("LT", 0, 1, 0, 0)); + session(const fingerprint& print + = libtorrent::fingerprint("LT", 0, 1, 0, 0)); session( const fingerprint& print @@ -808,19 +827,71 @@ The typical code to get info from a torrent file will then look like this:: entry torrent_file; // ... + // throws if this is not a dictionary entry::dictionary_type const& dict = torrent_file.dict(); entry::dictionary_type::const_iterator i; i = dict.find("announce"); if (i != dict.end()) { - std::string tracker_url= i->second.string(); + std::string tracker_url = i->second.string(); std::cout << tracker_url << "\n"; } -To make it easier to extract information from a torren file, the class ``torrent_info`` + +The following code is equivalent, but a little bit shorter:: + + entry torrent_file; + // ... + + // throws if this is not a dictionary + if (entry* i = torrent_file.find_key("announce")) + { + std::string tracker_url = i->string(); + std::cout << tracker_url << "\n"; + } + + +To make it easier to extract information from a torrent file, the class torrent_info_ exists. +operator[] +---------- + + :: + + entry& operator[](char const* key); + entry& operator[](std::string const& key); + entry const& operator[](char const* key) const; + entry const& operator[](std::string const& key) const; + +All of these functions requires the entry to be a dictionary, if it isn't they +will throw ``libtorrent::type_error``. + +The non-const versions of the ``operator[]`` will return a reference to either +the existing element at the given key or, if there is no element with the +given key, a reference to a newly inserted element at that key. + +The const version of ``operator[]`` will only return a reference to an +existing element at the given key. If the key is not found, it will throw +``libtorrent::type_error``. + + +find_key() +---------- + + :: + + entry* find_key(char const* key); + entry const* find_key(char const* key) const; + +These functions requires the entry to be a dictionary, if it isn't they +will throw ``libtorrent::type_error``. + +They will look for an element at the given key in the dictionary, if the +element cannot be found, they will return 0. If an element with the given +key is found, the return a pointer to it. + torrent_info ============ @@ -844,7 +915,8 @@ The ``torrent_info`` has the following synopsis:: void add_file(boost::filesystem::path file, size_type size); typedef std::vector::const_iterator file_iterator; - typedef std::vector::const_reverse_iterator reverse_file_iterator; + typedef std::vector::const_reverse_iterator + reverse_file_iterator; file_iterator begin_files() const; file_iterator end_files() const; @@ -914,14 +986,14 @@ set_comment() set_piece_size() set_creator() set_hash() add_tracker() add_file() These files are used when creating a torrent file. ``set_comment()`` will simply set the comment that belongs to this torrent. The comment can be retrieved with the -``comment()`` member. +``comment()`` member. The string should be UTF-8 encoded. ``set_piece_size()`` will set the size of each piece in this torrent. The piece size must be an even multiple of 2. i.e. usually something like 256 kiB, 512 kiB, 1024 kiB etc. The size is given in number of bytes. ``set_creator()`` is an optional attribute that can be used to identify your application -that was used to create the torrent file. +that was used to create the torrent file. The string should be UTF-8 encoded. ``set_hash()`` writes the hash for the piece with the given piece-index. You have to call this function for every piece in the torrent. Usually the hasher_ is used to calculate @@ -973,6 +1045,7 @@ iterators with the type ``file_entry``. The ``path`` is the full (relative) path of each file. i.e. if it is a multi-file torrent, all the files starts with a directory with the same name as ``torrent_info::name()``. +The filenames are encoded with UTF-8. :: @@ -1077,6 +1150,8 @@ it will return an empty string. ``creation_date()`` returns a `boost::posix_time object, representing the time when this torrent file was created. If there's no timestamp in the torrent file, this will return a date of january 1:st 1970. +Both the name and the comment is UTF-8 encoded strings. + ``creator()`` returns the creator string in the torrent. If there is no creator string it will return an empty string. @@ -1107,7 +1182,8 @@ Its declaration looks like this:: void force_reannounce(); void connect_peer(address const& adr) const; - void set_tracker_login(std::string const& username, std::string const& password); + void set_tracker_login(std::string const& username + , std::string const& password); std::vector const& trackers() const; void replace_trackers(std::vector const&); @@ -1981,7 +2057,8 @@ This is the class declaration:: struct fingerprint { - fingerprint(const char* id_string, int major, int minor, int revision, int tag); + fingerprint(const char* id_string, int major, int minor + , int revision, int tag); std::string to_string() const; @@ -2015,6 +2092,10 @@ sure not to clash with anybody else. Here are some taken id's: | 'XT' | Xan Torrent | +----------+-----------------------+ +There's currently an informal directory of client id's here__. + +__ http://wiki.theory.org/BitTorrentSpecification#peer_id + The ``major``, ``minor``, ``revision`` and ``tag`` parameters are used to identify the version of your client. All these numbers must be within the range [0, 9]. @@ -2607,14 +2688,15 @@ print information about it to std out:: { std::ifstream in(argv[1], std::ios_base::binary); in.unsetf(std::ios_base::skipws); - entry e = bdecode(std::istream_iterator(in), std::istream_iterator()); + entry e = bdecode(std::istream_iterator(in) + , std::istream_iterator()); torrent_info t(e); // print info about torrent std::cout << "\n\n----- torrent file info -----\n\n"; std::cout << "trackers:\n"; - for (std::vector::const_iterator i = t.trackers().begin(); - i != t.trackers().end(); ++i) + for (std::vector::const_iterator i + = t.trackers().begin(), end(t.trackers().end); i != end; ++i) { std::cout << i->tier << ": " << i->url << "\n"; } @@ -2623,8 +2705,7 @@ print information about it to std out:: std::cout << "piece length: " << t.piece_length() << "\n"; std::cout << "files:\n"; for (torrent_info::file_iterator i = t.begin_files(); - i != t.end_files(); - ++i) + i != t.end_files(); ++i) { std::cout << " " << std::setw(11) << i->size << " " << i->path << " " << i->filename << "\n"; @@ -2676,7 +2757,8 @@ This is a simple client. It doesn't have much output to keep it simple:: std::ifstream in(argv[1], std::ios_base::binary); in.unsetf(std::ios_base::skipws); - entry e = bdecode(std::istream_iterator(in), std::istream_iterator()); + entry e = bdecode(std::istream_iterator(in) + , std::istream_iterator()); s.add_torrent(e, ""); // wait for the user to end @@ -2715,10 +2797,7 @@ Shows how to create a torrent from a directory tree:: using namespace boost::filesystem; using namespace libtorrent; - void add_files( - torrent_info& t - , path const& p - , path const& l) + void add_files(torrent_info& t, path const& p, path const& l) { path f(p / l); if (is_directory(f)) @@ -2743,8 +2822,8 @@ Shows how to create a torrent from a directory tree:: if (argc != 4) { - std::cerr << "usage: make_torrent " - "\n"; + std::cerr << "usage: make_torrent " + " \n"; return 1; } diff --git a/include/libtorrent/session.hpp b/include/libtorrent/session.hpp index 948bddbd3..c92d043ff 100755 --- a/include/libtorrent/session.hpp +++ b/include/libtorrent/session.hpp @@ -54,7 +54,7 @@ POSSIBILITY OF SUCH DAMAGE. #endif #include "libtorrent/torrent_handle.hpp" -#include "libtorrent/torrent.hpp" +//#include "libtorrent/torrent.hpp" #include "libtorrent/entry.hpp" #include "libtorrent/torrent_info.hpp" #include "libtorrent/socket.hpp" @@ -78,6 +78,8 @@ POSSIBILITY OF SUCH DAMAGE. namespace libtorrent { + class torrent; + namespace detail { // workaround for microsofts @@ -128,17 +130,17 @@ namespace libtorrent // if it is not being processed, then it can be removed from // the queue without problems, otherwise the abort flag has // to be set. - volatile bool processing; + bool processing; // is filled in by storage::initialize_pieces() // and represents the progress. It should be a // value in the range [0, 1] - volatile float progress; + float progress; // abort defaults to false and is typically // filled in by torrent_handle when the user // aborts the torrent - volatile bool abort; + bool abort; }; struct checker_impl: boost::noncopyable @@ -157,7 +159,8 @@ namespace libtorrent // a list of all torrents that are currently in queue // or checking their files - std::deque m_torrents; + std::deque > m_torrents; + std::deque > m_processing; bool m_abort; }; diff --git a/include/libtorrent/storage.hpp b/include/libtorrent/storage.hpp index a46ed1a25..b9898e0be 100755 --- a/include/libtorrent/storage.hpp +++ b/include/libtorrent/storage.hpp @@ -122,11 +122,9 @@ namespace libtorrent ~piece_manager(); - void check_pieces( - boost::mutex& mutex - , detail::piece_checker_data& data - , std::vector& pieces - , bool compact_mode); + bool check_fastresume(detail::piece_checker_data& d + , std::vector& pieces, bool compact_mode); + std::pair check_files(std::vector& pieces); void release_files(); diff --git a/include/libtorrent/torrent.hpp b/include/libtorrent/torrent.hpp index 1b9383e93..2e7c5a37c 100755 --- a/include/libtorrent/torrent.hpp +++ b/include/libtorrent/torrent.hpp @@ -47,6 +47,7 @@ POSSIBILITY OF SUCH DAMAGE. #include #include #include +#include #ifdef _MSC_VER #pragma warning(pop) @@ -61,6 +62,7 @@ POSSIBILITY OF SUCH DAMAGE. #include "libtorrent/stat.hpp" #include "libtorrent/alert.hpp" #include "libtorrent/resource_request.hpp" +#include "libtorrent/piece_picker.hpp" namespace libtorrent { @@ -88,11 +90,13 @@ namespace libtorrent // for a specific download. It updates itself against // the tracker class torrent: public request_callback + , public boost::enable_shared_from_this { public: torrent( detail::session_impl& ses + , detail::checker_impl& checker , entry const& metadata , boost::filesystem::path const& save_path , address const& net_interface @@ -103,6 +107,7 @@ namespace libtorrent // (the metadata is downloaded from the peers) torrent( detail::session_impl& ses + , detail::checker_impl& checker , char const* tracker_url , sha1_hash const& info_hash , boost::filesystem::path const& save_path @@ -135,10 +140,11 @@ namespace libtorrent // this is called from the peer_connection for // each piece of metadata it receives void metadata_progress(int total_size, int received); - - void check_files( - detail::piece_checker_data& data - , boost::mutex& mutex, bool lock_session = true); + + bool check_fastresume(detail::piece_checker_data&); + std::pair check_files(); + void files_checked(std::vector const& + unfinished_pieces); stat statistics() const { return m_stat; } size_type bytes_left() const; @@ -153,8 +159,8 @@ namespace libtorrent bool is_piece_filtered(int index) const; void filtered_pieces(std::vector& bitmask) const; - //idea from Arvid and MooPolice - //todo refactoring and improving the function body + // idea from Arvid and MooPolice + // todo refactoring and improving the function body // marks the file with the given index as filtered // it will not be downloaded void filter_file(int index, bool filter); @@ -456,6 +462,7 @@ namespace libtorrent // a back reference to the session // this torrent belongs to. detail::session_impl& m_ses; + detail::checker_impl& m_checker; std::auto_ptr m_picker; diff --git a/src/policy.cpp b/src/policy.cpp index 6f461a4ef..3143f9b45 100755 --- a/src/policy.cpp +++ b/src/policy.cpp @@ -728,7 +728,7 @@ namespace libtorrent p->connection->send_choke(); } while (m_num_unchoked > m_torrent->m_uploads_quota.given); } - else + else if (m_num_unchoked > 0) { // optimistic unchoke. trade the 'worst' // unchoked peer with one of the choked diff --git a/src/session.cpp b/src/session.cpp index adf10ae7f..5a4cc320c 100755 --- a/src/session.cpp +++ b/src/session.cpp @@ -70,14 +70,6 @@ POSSIBILITY OF SUCH DAMAGE. #include "libtorrent/peer_connection.hpp" #include "libtorrent/ip_filter.hpp" -#if defined(_MSC_VER) && _MSC_VER < 1300 -namespace std -{ - using ::srand; - using ::isprint; -}; -#endif - using namespace boost::posix_time; namespace libtorrent { namespace detail @@ -100,81 +92,106 @@ namespace libtorrent { namespace detail void checker_impl::operator()() { eh_initializer(); + // if we're currently performing a full file check, + // this is the torrent being processed + boost::shared_ptr processing; + boost::shared_ptr t; for (;;) { - piece_checker_data* t = 0; - { - boost::mutex::scoped_lock l(m_mutex); - - // if the job queue is empty and - // we shouldn't abort - // wait for a signal - if (m_torrents.empty() && !m_abort) - m_cond.wait(l); - - if (m_abort) return; - - assert(!m_torrents.empty()); - - t = &m_torrents.front(); - if (t->abort) - { - m_torrents.pop_front(); - continue; - } - t->processing = true; - } - + // temporary torrent used while checking fastresume data try { - assert(t != 0); - std::string error_msg; - t->parse_resume_data(t->resume_data, t->torrent_ptr->torrent_file(), error_msg); - - // clear the resume data now that it has been used - // (the fast resume data is now parsed and stored in t) - t->resume_data = entry(); - t->torrent_ptr->check_files(*t, m_mutex); - // lock the session to add the new torrent - - boost::mutex::scoped_lock l(m_mutex); - if (t->abort) + t.reset(); { - m_torrents.pop_front(); - continue; - } - boost::mutex::scoped_lock l2(m_ses.m_mutex); + boost::mutex::scoped_lock l(m_mutex); - if (!error_msg.empty() && m_ses.m_alerts.should_post(alert::warning)) - { - m_ses.m_alerts.post_alert(fastresume_rejected_alert( - t->torrent_ptr->get_handle() - , error_msg)); - - } - - m_ses.m_torrents.insert(std::make_pair(t->info_hash, t->torrent_ptr)); - if (t->torrent_ptr->is_seed() && m_ses.m_alerts.should_post(alert::info)) - { - m_ses.m_alerts.post_alert(torrent_finished_alert( - t->torrent_ptr->get_handle() - , "torrent is complete")); + // if the job queue is empty and + // we shouldn't abort + // wait for a signal + if (m_torrents.empty() && !m_abort && !processing) + m_cond.wait(l); + + if (m_abort) return; + + if (!m_torrents.empty()) + { + t = m_torrents.front(); + if (t->abort) + { + m_torrents.pop_front(); + continue; + } + } } - peer_id id; - std::fill(id.begin(), id.end(), 0); - for (std::vector
::const_iterator i = t->peers.begin(); - i != t->peers.end(); ++i) + if (t) { - t->torrent_ptr->get_policy().peer_from_tracker(*i, id); + std::string error_msg; + t->parse_resume_data(t->resume_data, t->torrent_ptr->torrent_file(), error_msg); + + if (!error_msg.empty() && m_ses.m_alerts.should_post(alert::warning)) + { + boost::mutex::scoped_lock l2(m_ses.m_mutex); + m_ses.m_alerts.post_alert(fastresume_rejected_alert( + t->torrent_ptr->get_handle() + , error_msg)); + } + + // clear the resume data now that it has been used + // (the fast resume data is now parsed and stored in t) + t->resume_data = entry(); + bool up_to_date = t->torrent_ptr->check_fastresume(*t); + + if (up_to_date) + { + // lock the session to add the new torrent + boost::mutex::scoped_lock l(m_ses.m_mutex); + boost::mutex::scoped_lock l2(m_mutex); + + assert(m_torrents.front() == t); + + t->torrent_ptr->files_checked(t->unfinished_pieces); + m_torrents.pop_front(); + m_ses.m_torrents.insert(std::make_pair(t->info_hash, t->torrent_ptr)); + if (t->torrent_ptr->is_seed() && m_ses.m_alerts.should_post(alert::info)) + { + m_ses.m_alerts.post_alert(torrent_finished_alert( + t->torrent_ptr->get_handle() + , "torrent is complete")); + } + + peer_id id; + std::fill(id.begin(), id.end(), 0); + for (std::vector
::const_iterator i = t->peers.begin(); + i != t->peers.end(); ++i) + { + t->torrent_ptr->get_policy().peer_from_tracker(*i, id); + } + continue; + } + + // lock the checker while we move the torrent from + // m_torrents to m_processing + { + boost::mutex::scoped_lock l(m_mutex); + assert(m_torrents.front() == t); + + m_torrents.pop_front(); + m_processing.push_back(t); + if (!processing) + { + processing = t; + processing->processing = true; + } + } } - m_torrents.pop_front(); } catch(const std::exception& e) { // This will happen if the storage fails to initialize - boost::mutex::scoped_lock l(m_mutex); - boost::mutex::scoped_lock l2(m_ses.m_mutex); + boost::mutex::scoped_lock l(m_ses.m_mutex); + boost::mutex::scoped_lock l2(m_mutex); + if (m_ses.m_alerts.should_post(alert::fatal)) { m_ses.m_alerts.post_alert( @@ -182,41 +199,156 @@ namespace libtorrent { namespace detail t->torrent_ptr->get_handle() , e.what())); } + assert(!m_torrents.empty()); m_torrents.pop_front(); } catch(...) { +#ifndef NDEBUG + std::cerr << "error while checking resume data\n"; +#endif + boost::mutex::scoped_lock l(m_mutex); + assert(!m_torrents.empty()); + m_torrents.pop_front(); + assert(false); + } + + if (!processing) continue; + + try + { + assert(processing); + + float finished = false; + float progress = 0.f; + boost::tie(finished, progress) = processing->torrent_ptr->check_files(); + + { + boost::mutex::scoped_lock l(m_mutex); + processing->progress = progress; + if (processing->abort) + { + assert(!m_processing.empty()); + assert(m_processing.front() == processing); + + processing.reset(); + m_processing.pop_front(); + if (!m_processing.empty()) + { + processing = m_processing.front(); + processing->processing = true; + } + continue; + } + } + if (finished) + { + // lock the session to add the new torrent + boost::mutex::scoped_lock l(m_ses.m_mutex); + boost::mutex::scoped_lock l2(m_mutex); + + assert(!m_processing.empty()); + assert(m_processing.front() == processing); + + processing->torrent_ptr->files_checked(processing->unfinished_pieces); + m_ses.m_torrents.insert(std::make_pair( + processing->info_hash, processing->torrent_ptr)); + if (processing->torrent_ptr->is_seed() + && m_ses.m_alerts.should_post(alert::info)) + { + m_ses.m_alerts.post_alert(torrent_finished_alert( + processing->torrent_ptr->get_handle() + , "torrent is complete")); + } + + peer_id id; + std::fill(id.begin(), id.end(), 0); + for (std::vector
::const_iterator i = processing->peers.begin(); + i != processing->peers.end(); ++i) + { + processing->torrent_ptr->get_policy().peer_from_tracker(*i, id); + } + processing.reset(); + m_processing.pop_front(); + if (!m_processing.empty()) + { + processing = m_processing.front(); + processing->processing = true; + } + } + } + catch(const std::exception& e) + { + // This will happen if the storage fails to initialize + boost::mutex::scoped_lock l(m_ses.m_mutex); + boost::mutex::scoped_lock l2(m_mutex); + if (m_ses.m_alerts.should_post(alert::fatal)) + { + m_ses.m_alerts.post_alert( + file_error_alert( + processing->torrent_ptr->get_handle() + , e.what())); + } + assert(!m_processing.empty()); + + processing.reset(); + m_processing.pop_front(); + if (!m_processing.empty()) + { + processing = m_processing.front(); + processing->processing = true; + } + } + catch(...) + { #ifndef NDEBUG std::cerr << "error while checking files\n"; #endif - assert(false); boost::mutex::scoped_lock l(m_mutex); - m_torrents.pop_front(); + assert(!m_processing.empty()); + + processing.reset(); + m_processing.pop_front(); + if (!m_processing.empty()) + { + processing = m_processing.front(); + processing->processing = true; + } + + assert(false); } } } detail::piece_checker_data* checker_impl::find_torrent(sha1_hash const& info_hash) { - for (std::deque::iterator i + for (std::deque >::iterator i = m_torrents.begin(); i != m_torrents.end(); ++i) { - if (i->info_hash == info_hash) return &(*i); + if ((*i)->info_hash == info_hash) return i->get(); } + for (std::deque >::iterator i + = m_processing.begin(); i != m_processing.end(); ++i) + { + if ((*i)->info_hash == info_hash) return i->get(); + } + return 0; } void checker_impl::remove_torrent(sha1_hash const& info_hash) { - for (std::deque::iterator i + for (std::deque >::iterator i = m_torrents.begin(); i != m_torrents.end(); ++i) { - if (i->info_hash == info_hash) + if ((*i)->info_hash == info_hash) { + assert((*i)->processing == false); m_torrents.erase(i); return; } } + assert(false); } session_impl::session_impl( @@ -389,8 +521,7 @@ namespace libtorrent { namespace detail } for (std::vector >::iterator i = - writable_clients.begin(); i != writable_clients.end(); - ++i) + writable_clients.begin(); i != writable_clients.end(); ++i) { assert((*i)->is_writable()); } @@ -969,16 +1100,16 @@ namespace libtorrent std::vector session::get_torrents() { - boost::mutex::scoped_lock l(m_checker_impl.m_mutex); - boost::mutex::scoped_lock l2(m_impl.m_mutex); + boost::mutex::scoped_lock l(m_impl.m_mutex); + boost::mutex::scoped_lock l2(m_checker_impl.m_mutex); std::vector ret; - for (std::deque::iterator i + for (std::deque >::iterator i = m_checker_impl.m_torrents.begin() , end(m_checker_impl.m_torrents.end()); i != end; ++i) { - if (i->abort) continue; + if ((*i)->abort) continue; ret.push_back(torrent_handle(&m_impl, &m_checker_impl - , i->info_hash)); + , (*i)->info_hash)); } for (detail::session_impl::torrent_map::iterator i @@ -1041,14 +1172,15 @@ namespace libtorrent // the checker thread and store it before starting // the thread boost::shared_ptr torrent_ptr( - new torrent(m_impl, metadata, save_path, m_impl.m_listen_interface - , compact_mode, block_size)); + new torrent(m_impl, m_checker_impl, metadata, save_path + , m_impl.m_listen_interface, compact_mode, block_size)); - detail::piece_checker_data d; - d.torrent_ptr = torrent_ptr; - d.save_path = save_path; - d.info_hash = ti.info_hash(); - d.resume_data = resume_data; + boost::shared_ptr d( + new detail::piece_checker_data); + d->torrent_ptr = torrent_ptr; + d->save_path = save_path; + d->info_hash = ti.info_hash(); + d->resume_data = resume_data; // add the torrent to the queue to be checked m_checker_impl.m_torrents.push_back(d); @@ -1105,7 +1237,7 @@ namespace libtorrent // the checker thread and store it before starting // the thread boost::shared_ptr torrent_ptr( - new torrent(m_impl, tracker_url, info_hash, save_path + new torrent(m_impl, m_checker_impl, tracker_url, info_hash, save_path , m_impl.m_listen_interface, compact_mode, block_size)); m_impl.m_torrents.insert( @@ -1222,7 +1354,7 @@ namespace libtorrent // abort the currently checking torrent if (!m_checker_impl.m_torrents.empty()) { - m_checker_impl.m_torrents.front().abort = true; + m_checker_impl.m_torrents.front()->abort = true; } m_checker_impl.m_cond.notify_one(); } diff --git a/src/sha1.cpp b/src/sha1.cpp index 21c9f484d..afcce6345 100755 --- a/src/sha1.cpp +++ b/src/sha1.cpp @@ -18,24 +18,26 @@ changelog at the end of the file. // #include #include +using boost::uint32_t; +using boost::uint8_t; struct SHA1_CTX { - boost::uint32_t state[5]; - boost::uint32_t count[2]; - boost::uint8_t buffer[64]; + uint32_t state[5]; + uint32_t count[2]; + uint8_t buffer[64]; }; void SHA1Init(SHA1_CTX* context); -void SHA1Update(SHA1_CTX* context, boost::uint8_t const* data, boost::uint32_t len); -void SHA1Final(SHA1_CTX* context, boost::uint8_t* digest); +void SHA1Update(SHA1_CTX* context, uint8_t const* data, uint32_t len); +void SHA1Final(SHA1_CTX* context, uint8_t* digest); namespace { union CHAR64LONG16 { - boost::uint8_t c[64]; - boost::uint32_t l[16]; + uint8_t c[64]; + uint32_t l[16]; }; #define rol(value, bits) (((value) << (bits)) | ((value) >> (32 - (bits)))) @@ -44,7 +46,7 @@ namespace // I got the idea of expanding during the round function from SSLeay struct little_endian_blk0 { - static boost::uint32_t apply(CHAR64LONG16* block, int i) + static uint32_t apply(CHAR64LONG16* block, int i) { return block->l[i] = (rol(block->l[i],24)&0xFF00FF00) | (rol(block->l[i],8)&0x00FF00FF); @@ -53,7 +55,7 @@ namespace struct big_endian_blk0 { - static boost::uint32_t apply(CHAR64LONG16* block, int i) + static uint32_t apply(CHAR64LONG16* block, int i) { return block->l[i]; } @@ -72,13 +74,13 @@ namespace // Hash a single 512-bit block. This is the core of the algorithm. template - void SHA1Transform(boost::uint32_t state[5], boost::uint8_t const buffer[64]) + void SHA1Transform(uint32_t state[5], uint8_t const buffer[64]) { using namespace std; - boost::uint32_t a, b, c, d, e; + uint32_t a, b, c, d, e; CHAR64LONG16* block; - boost::uint8_t workspace[64]; + uint8_t workspace[64]; block = (CHAR64LONG16*)workspace; memcpy(block, buffer, 64); @@ -130,10 +132,10 @@ namespace } template - void internal_update(SHA1_CTX* context, boost::uint8_t const* data, boost::uint32_t len) + void internal_update(SHA1_CTX* context, uint8_t const* data, uint32_t len) { using namespace std; - boost::uint32_t i, j; // JHB + uint32_t i, j; // JHB #ifdef VERBOSE SHAPrintContext(context, "before"); @@ -163,8 +165,8 @@ namespace bool is_big_endian() { - boost::uint32_t test = 1; - return *reinterpret_cast(&test) == 0; + uint32_t test = 1; + return *reinterpret_cast(&test) == 0; } } @@ -184,7 +186,7 @@ void SHA1Init(SHA1_CTX* context) // Run your data through this. -void SHA1Update(SHA1_CTX* context, boost::uint8_t const* data, boost::uint32_t len) +void SHA1Update(SHA1_CTX* context, uint8_t const* data, uint32_t len) { #if defined __BIG_ENDIAN__ internal_update(context, data, len); @@ -203,24 +205,24 @@ void SHA1Update(SHA1_CTX* context, boost::uint8_t const* data, boost::uint32_t l // Add padding and return the message digest. -void SHA1Final(SHA1_CTX* context, boost::uint8_t* digest) +void SHA1Final(SHA1_CTX* context, uint8_t* digest) { - boost::uint8_t finalcount[8]; + uint8_t finalcount[8]; - for (boost::uint32_t i = 0; i < 8; ++i) + for (uint32_t i = 0; i < 8; ++i) { // Endian independent - finalcount[i] = static_cast( + finalcount[i] = static_cast( (context->count[(i >= 4 ? 0 : 1)] >> ((3-(i & 3)) * 8) ) & 255); } - SHA1Update(context, (boost::uint8_t const*)"\200", 1); + SHA1Update(context, (uint8_t const*)"\200", 1); while ((context->count[0] & 504) != 448) - SHA1Update(context, (boost::uint8_t const*)"\0", 1); + SHA1Update(context, (uint8_t const*)"\0", 1); SHA1Update(context, finalcount, 8); // Should cause a SHA1Transform() - for (boost::uint32_t i = 0; i < 20; ++i) + for (uint32_t i = 0; i < 20; ++i) { digest[i] = static_cast( (context->state[i>>2] >> ((3-(i & 3)) * 8) ) & 255); diff --git a/src/storage.cpp b/src/storage.cpp index f1284a259..803cde59a 100755 --- a/src/storage.cpp +++ b/src/storage.cpp @@ -790,12 +790,14 @@ namespace libtorrent const torrent_info& info , const path& path); - void check_pieces( - boost::mutex& mutex - , detail::piece_checker_data& data + bool check_fastresume( + detail::piece_checker_data& d , std::vector& pieces , bool compact_mode); + std::pair check_files( + std::vector& pieces); + void release_files(); void allocate_slots(int num_slots); @@ -836,9 +838,6 @@ namespace libtorrent private: - void post_check(boost::mutex& mutex - , detail::piece_checker_data& data); - // returns the slot currently associated with the given // piece or assigns the given piece_index to a free slot @@ -904,6 +903,29 @@ namespace libtorrent bool m_allocating; boost::mutex m_allocating_monitor; boost::condition m_allocating_condition; + + // these states are used while checking/allocating the torrent + + enum { + // the default initial state + state_none, + // the file checking is complete + state_finished, + // creating the directories + state_create_files, + // checking the files + state_full_check, + // allocating files (in non-compact mode) + state_allocating + } m_state; + int m_current_slot; + + std::vector m_piece_data; + + // this maps a piece hash to piece index. It will be + // build the first time it is used (to save time if it + // isn't needed) + std::multimap m_hash_to_piece; }; piece_manager::impl::impl( @@ -1239,31 +1261,12 @@ namespace libtorrent } } - void piece_manager::impl::post_check(boost::mutex& mutex - , detail::piece_checker_data& data) - { - if (!m_compact_mode) - { - // if we're not in compact mode, make sure the - // pieces are spread out and placed at their - // final position. - int num_slots = (int)m_unallocated_slots.size(); - - for (int i = 0; i < num_slots; ++i) - { - allocate_slots(1); - - boost::mutex::scoped_lock lock(mutex); - data.progress = (float)i / num_slots; - if (data.abort || (data.torrent_ptr && data.torrent_ptr->is_aborted())) - return; - } - } - } - - void piece_manager::impl::check_pieces( - boost::mutex& mutex - , detail::piece_checker_data& data + // check if the fastresume data is up to date + // if it is, use it and return true. If it + // isn't return false and the full check + // will be run + bool piece_manager::impl::check_fastresume( + detail::piece_checker_data& data , std::vector& pieces , bool compact_mode) { @@ -1325,235 +1328,277 @@ namespace libtorrent } } + m_unallocated_slots.reserve(int(pieces.size() - data.piece_map.size())); for (int i = (int)data.piece_map.size(); i < (int)pieces.size(); ++i) { m_unallocated_slots.push_back(i); } - post_check(mutex, data); - - return; + if (!m_compact_mode && !m_unallocated_slots.empty()) + { + m_state = state_allocating; + return false; + } + else + { + m_state = state_finished; + return true; + } } + m_state = state_create_files; + return false; + } + + // performs the full check and full allocation + // (if necessary). returns true if finished and + // false if it should be called again + // the second return value is the progress the + // file check is at. 0 is nothing done, and 1 + // is finished + std::pair piece_manager::impl::check_files( + std::vector& pieces) + { + if (m_state == state_allocating) + { + if (m_compact_mode) + { + m_state = state_finished; + return std::make_pair(true, 1.f); + } + + // if we're not in compact mode, make sure the + // pieces are spread out and placed at their + // final position. + assert(!m_unallocated_slots.empty()); + allocate_slots(1); + if (m_unallocated_slots.empty()) + { + m_state = state_finished; + return std::make_pair(true, 1.f); + } + + return std::make_pair(false, 1.f - (float)m_unallocated_slots.size() + / (float)m_slot_to_piece.size()); + } + + if (m_state == state_create_files) + { + // first, create all missing directories + path last_path; + for (torrent_info::file_iterator file_iter = m_info.begin_files(), + end_iter = m_info.end_files(); file_iter != end_iter; ++file_iter) + { + path dir = (m_save_path / file_iter->path).branch_path(); + if (dir == last_path) continue; + last_path = dir; + +#if defined(_WIN32) && defined(UNICODE) + if (!exists_win(last_path)) + create_directories_win(last_path); +#else + if (!exists(last_path)) + create_directories(last_path); +#endif + } + m_current_slot = 0; + m_state = state_full_check; + m_piece_data.resize(int(m_info.piece_length())); + return std::make_pair(false, 0.f); + } + + assert(m_state == state_full_check); + // ------------------------ // DO THE FULL CHECK // ------------------------ - - - // first, create all missing directories - path last_path; - for (torrent_info::file_iterator file_iter = m_info.begin_files(), - end_iter = m_info.end_files(); file_iter != end_iter; ++file_iter) + try { - path dir = (m_save_path / file_iter->path).branch_path(); - if (dir == last_path) continue; - last_path = dir; -#if defined(_WIN32) && defined(UNICODE) - if (!exists_win(last_path)) - create_directories_win(last_path); -#else - if (!exists(last_path)) - create_directories(last_path); -#endif - } - std::vector piece_data(static_cast(m_info.piece_length())); - - // this maps a piece hash to piece index. It will be - // build the first time it is used (to save time if it - // isn't needed) - std::multimap hash_to_piece; - // build the hash-map, that maps hashes to pieces - for (int current_slot = 0; current_slot < m_info.num_pieces(); ++current_slot) - { - try - { - - m_storage.read( - &piece_data[0] - , current_slot + m_storage.read( + &m_piece_data[0] + , m_current_slot , 0 - , static_cast(m_info.piece_size(current_slot))); + , int(m_info.piece_size(m_current_slot))); - if (hash_to_piece.empty()) + if (m_hash_to_piece.empty()) + { + for (int i = 0; i < m_info.num_pieces(); ++i) { - for (int i = 0; i < m_info.num_pieces(); ++i) - { - hash_to_piece.insert(std::make_pair(m_info.hash_for_piece(i), i)); - } + m_hash_to_piece.insert(std::make_pair(m_info.hash_for_piece(i), i)); } + } - int piece_index = identify_data( - piece_data - , current_slot + int piece_index = identify_data( + m_piece_data + , m_current_slot , pieces - , hash_to_piece); + , m_hash_to_piece); - assert(piece_index == unassigned || piece_index >= 0); - - const bool this_should_move = piece_index >= 0 && m_slot_to_piece[piece_index] != unallocated; - const bool other_should_move = m_piece_to_slot[current_slot] != has_no_slot; + assert(piece_index == unassigned || piece_index >= 0); - // check if this piece should be swapped with any other slot - // this section will ensure that the storage is correctly sorted - // libtorrent will never leave the storage in a state that - // requires this sorting, but other clients may. + const bool this_should_move = piece_index >= 0 && m_slot_to_piece[piece_index] != unallocated; + const bool other_should_move = m_piece_to_slot[m_current_slot] != has_no_slot; - // example of worst case: - // | current_slot = 5 - // V - // +---+- - - +---+- - - +---+- - - // | x | | 5 | | 3 | <- piece data in slots - // +---+- - - +---+- - - +---+- - - // 3 y 5 <- slot index + // check if this piece should be swapped with any other slot + // this section will ensure that the storage is correctly sorted + // libtorrent will never leave the storage in a state that + // requires this sorting, but other clients may. - // in this example, the data in the current_slot (5) - // is piece 3. It has to be moved into slot 3. The data - // in slot y (piece 5) should be moved into the current_slot. - // and the data in slot 3 (piece x) should be moved to slot y. + // example of worst case: + // | m_current_slot = 5 + // V + // +---+- - - +---+- - - +---+- - + // | x | | 5 | | 3 | <- piece data in slots + // +---+- - - +---+- - - +---+- - + // 3 y 5 <- slot index - // there are three possible cases. - // 1. There's another piece that should be placed into this slot - // 2. This piece should be placed into another slot. - // 3. There's another piece that should be placed into this slot - // and this piece should be placed into another slot + // in this example, the data in the m_current_slot (5) + // is piece 3. It has to be moved into slot 3. The data + // in slot y (piece 5) should be moved into the m_current_slot. + // and the data in slot 3 (piece x) should be moved to slot y. - // swap piece_index with this slot + // there are three possible cases. + // 1. There's another piece that should be placed into this slot + // 2. This piece should be placed into another slot. + // 3. There's another piece that should be placed into this slot + // and this piece should be placed into another slot - // case 1 - if (this_should_move && !other_should_move) + // swap piece_index with this slot + + // case 1 + if (this_should_move && !other_should_move) + { + assert(piece_index != m_current_slot); + + const int other_slot = piece_index; + assert(other_slot >= 0); + int other_piece = m_slot_to_piece[other_slot]; + + m_slot_to_piece[other_slot] = piece_index; + m_slot_to_piece[m_current_slot] = other_piece; + m_piece_to_slot[piece_index] = piece_index; + if (other_piece >= 0) m_piece_to_slot[other_piece] = m_current_slot; + + if (other_piece == unassigned) { - assert(piece_index != current_slot); - - const int other_slot = piece_index; - assert(other_slot >= 0); - int other_piece = m_slot_to_piece[other_slot]; - - m_slot_to_piece[other_slot] = piece_index; - m_slot_to_piece[current_slot] = other_piece; - m_piece_to_slot[piece_index] = piece_index; - if (other_piece >= 0) m_piece_to_slot[other_piece] = current_slot; - - if (other_piece == unassigned) - { - std::vector::iterator i = - std::find(m_free_slots.begin(), m_free_slots.end(), other_slot); - assert(i != m_free_slots.end()); - m_free_slots.erase(i); - m_free_slots.push_back(current_slot); - } - - const int slot1_size = static_cast(m_info.piece_size(piece_index)); - const int slot2_size = other_piece >= 0 ? static_cast(m_info.piece_size(other_piece)) : 0; - std::vector buf1(slot1_size); - m_storage.read(&buf1[0], current_slot, 0, slot1_size); - if (slot2_size > 0) - { - std::vector buf2(slot2_size); - m_storage.read(&buf2[0], piece_index, 0, slot2_size); - m_storage.write(&buf2[0], current_slot, 0, slot2_size); - } - m_storage.write(&buf1[0], piece_index, 0, slot1_size); - assert(m_slot_to_piece[current_slot] == unassigned - || m_piece_to_slot[m_slot_to_piece[current_slot]] == current_slot); + std::vector::iterator i = + std::find(m_free_slots.begin(), m_free_slots.end(), other_slot); + assert(i != m_free_slots.end()); + m_free_slots.erase(i); + m_free_slots.push_back(m_current_slot); } - // case 2 - else if (!this_should_move && other_should_move) + + const int slot1_size = static_cast(m_info.piece_size(piece_index)); + const int slot2_size = other_piece >= 0 ? static_cast(m_info.piece_size(other_piece)) : 0; + std::vector buf1(slot1_size); + m_storage.read(&buf1[0], m_current_slot, 0, slot1_size); + if (slot2_size > 0) { - assert(piece_index != current_slot); - - const int other_piece = current_slot; - const int other_slot = m_piece_to_slot[other_piece]; - assert(other_slot >= 0); - - m_slot_to_piece[current_slot] = other_piece; - m_slot_to_piece[other_slot] = piece_index; - m_piece_to_slot[other_piece] = current_slot; - if (piece_index >= 0) m_piece_to_slot[piece_index] = other_slot; - - if (piece_index == unassigned) - { - m_free_slots.push_back(other_slot); - } - - const int slot1_size = static_cast(m_info.piece_size(other_piece)); - const int slot2_size = piece_index >= 0 ? static_cast(m_info.piece_size(piece_index)) : 0; - std::vector buf1(slot1_size); - m_storage.read(&buf1[0], other_slot, 0, slot1_size); - if (slot2_size > 0) - { - std::vector buf2(slot2_size); - m_storage.read(&buf2[0], current_slot, 0, slot2_size); - m_storage.write(&buf2[0], other_slot, 0, slot2_size); - } - m_storage.write(&buf1[0], current_slot, 0, slot1_size); - assert(m_slot_to_piece[current_slot] == unassigned - || m_piece_to_slot[m_slot_to_piece[current_slot]] == current_slot); + std::vector buf2(slot2_size); + m_storage.read(&buf2[0], piece_index, 0, slot2_size); + m_storage.write(&buf2[0], m_current_slot, 0, slot2_size); } - else if (this_should_move && other_should_move) + m_storage.write(&buf1[0], piece_index, 0, slot1_size); + assert(m_slot_to_piece[m_current_slot] == unassigned + || m_piece_to_slot[m_slot_to_piece[m_current_slot]] == m_current_slot); + } + // case 2 + else if (!this_should_move && other_should_move) + { + assert(piece_index != m_current_slot); + + const int other_piece = m_current_slot; + const int other_slot = m_piece_to_slot[other_piece]; + assert(other_slot >= 0); + + m_slot_to_piece[m_current_slot] = other_piece; + m_slot_to_piece[other_slot] = piece_index; + m_piece_to_slot[other_piece] = m_current_slot; + if (piece_index >= 0) m_piece_to_slot[piece_index] = other_slot; + + if (piece_index == unassigned) { - assert(piece_index != current_slot); - assert(piece_index >= 0); + m_free_slots.push_back(other_slot); + } - const int piece1 = m_slot_to_piece[piece_index]; - const int piece2 = current_slot; - const int slot1 = piece_index; - const int slot2 = m_piece_to_slot[piece2]; + const int slot1_size = static_cast(m_info.piece_size(other_piece)); + const int slot2_size = piece_index >= 0 ? static_cast(m_info.piece_size(piece_index)) : 0; + std::vector buf1(slot1_size); + m_storage.read(&buf1[0], other_slot, 0, slot1_size); + if (slot2_size > 0) + { + std::vector buf2(slot2_size); + m_storage.read(&buf2[0], m_current_slot, 0, slot2_size); + m_storage.write(&buf2[0], other_slot, 0, slot2_size); + } + m_storage.write(&buf1[0], m_current_slot, 0, slot1_size); + assert(m_slot_to_piece[m_current_slot] == unassigned + || m_piece_to_slot[m_slot_to_piece[m_current_slot]] == m_current_slot); + } + else if (this_should_move && other_should_move) + { + assert(piece_index != m_current_slot); + assert(piece_index >= 0); - assert(slot1 >= 0); - assert(slot2 >= 0); - assert(piece2 >= 0); - - if (slot1 == slot2) - { - // this means there are only two pieces involved in the swap - assert(piece1 >= 0); + const int piece1 = m_slot_to_piece[piece_index]; + const int piece2 = m_current_slot; + const int slot1 = piece_index; + const int slot2 = m_piece_to_slot[piece2]; - // movement diagram: - // +-----------------------------+ - // | | - // +--> slot1 --> current_slot --+ + assert(slot1 >= 0); + assert(slot2 >= 0); + assert(piece2 >= 0); - m_slot_to_piece[slot1] = piece_index; - m_slot_to_piece[current_slot] = piece1; + if (slot1 == slot2) + { + // this means there are only two pieces involved in the swap + assert(piece1 >= 0); - m_piece_to_slot[piece_index] = slot1; - m_piece_to_slot[piece1] = current_slot; - - assert(piece1 == current_slot); - assert(piece_index == slot1); + // movement diagram: + // +-------------------------------+ + // | | + // +--> slot1 --> m_current_slot --+ - const int slot1_size = static_cast(m_info.piece_size(piece1)); - const int slot3_size = static_cast(m_info.piece_size(piece_index)); - std::vector buf1(static_cast(slot1_size)); - std::vector buf2(static_cast(slot3_size)); + m_slot_to_piece[slot1] = piece_index; + m_slot_to_piece[m_current_slot] = piece1; - m_storage.read(&buf2[0], current_slot, 0, slot3_size); - m_storage.read(&buf1[0], slot1, 0, slot1_size); - m_storage.write(&buf1[0], current_slot, 0, slot1_size); - m_storage.write(&buf2[0], slot1, 0, slot3_size); + m_piece_to_slot[piece_index] = slot1; + m_piece_to_slot[piece1] = m_current_slot; - assert(m_slot_to_piece[current_slot] == unassigned - || m_piece_to_slot[m_slot_to_piece[current_slot]] == current_slot); + assert(piece1 == m_current_slot); + assert(piece_index == slot1); - continue; - } - + const int slot1_size = static_cast(m_info.piece_size(piece1)); + const int slot3_size = static_cast(m_info.piece_size(piece_index)); + std::vector buf1(static_cast(slot1_size)); + std::vector buf2(static_cast(slot3_size)); + + m_storage.read(&buf2[0], m_current_slot, 0, slot3_size); + m_storage.read(&buf1[0], slot1, 0, slot1_size); + m_storage.write(&buf1[0], m_current_slot, 0, slot1_size); + m_storage.write(&buf2[0], slot1, 0, slot3_size); + + assert(m_slot_to_piece[m_current_slot] == unassigned + || m_piece_to_slot[m_slot_to_piece[m_current_slot]] == m_current_slot); + } + else + { assert(slot1 != slot2); assert(piece1 != piece2); // movement diagram: - // +---------------------------------------+ - // | | - // +--> slot1 --> slot2 --> current_slot --+ + // +-----------------------------------------+ + // | | + // +--> slot1 --> slot2 --> m_current_slot --+ m_slot_to_piece[slot1] = piece_index; m_slot_to_piece[slot2] = piece1; - m_slot_to_piece[current_slot] = piece2; + m_slot_to_piece[m_current_slot] = piece2; m_piece_to_slot[piece_index] = slot1; - m_piece_to_slot[current_slot] = piece2; + m_piece_to_slot[m_current_slot] = piece2; if (piece1 >= 0) m_piece_to_slot[piece1] = slot2; if (piece1 == unassigned) @@ -1572,96 +1617,102 @@ namespace libtorrent std::vector buf1(static_cast(m_info.piece_length())); std::vector buf2(static_cast(m_info.piece_length())); - m_storage.read(&buf2[0], current_slot, 0, slot3_size); + m_storage.read(&buf2[0], m_current_slot, 0, slot3_size); m_storage.read(&buf1[0], slot2, 0, slot2_size); - m_storage.write(&buf1[0], current_slot, 0, slot2_size); + m_storage.write(&buf1[0], m_current_slot, 0, slot2_size); if (slot1_size > 0) { m_storage.read(&buf1[0], slot1, 0, slot1_size); m_storage.write(&buf1[0], slot2, 0, slot1_size); } m_storage.write(&buf2[0], slot1, 0, slot3_size); - assert(m_slot_to_piece[current_slot] == unassigned - || m_piece_to_slot[m_slot_to_piece[current_slot]] == current_slot); - } - else - { - assert(m_piece_to_slot[current_slot] == has_no_slot || piece_index != current_slot); - assert(m_slot_to_piece[current_slot] == unallocated); - assert(piece_index == unassigned || m_piece_to_slot[piece_index] == has_no_slot); - - // the slot was identified as piece 'piece_index' - if (piece_index != unassigned) - m_piece_to_slot[piece_index] = current_slot; - else - m_free_slots.push_back(current_slot); - - m_slot_to_piece[current_slot] = piece_index; - - assert(m_slot_to_piece[current_slot] == unassigned - || m_piece_to_slot[m_slot_to_piece[current_slot]] == current_slot); + assert(m_slot_to_piece[m_current_slot] == unassigned + || m_piece_to_slot[m_slot_to_piece[m_current_slot]] == m_current_slot); } } - catch (file_error&) + else { - // find the file that failed, and skip all the blocks in that file - size_type file_offset = 0; - size_type current_offset = current_slot * m_info.piece_length(); - for (torrent_info::file_iterator i = m_info.begin_files(); - i != m_info.end_files(); ++i) - { - file_offset += i->size; - if (file_offset > current_offset) break; - } + assert(m_piece_to_slot[m_current_slot] == has_no_slot || piece_index != m_current_slot); + assert(m_slot_to_piece[m_current_slot] == unallocated); + assert(piece_index == unassigned || m_piece_to_slot[piece_index] == has_no_slot); - assert(file_offset > current_offset); - int skip_blocks = static_cast( + // the slot was identified as piece 'piece_index' + if (piece_index != unassigned) + m_piece_to_slot[piece_index] = m_current_slot; + else + m_free_slots.push_back(m_current_slot); + + m_slot_to_piece[m_current_slot] = piece_index; + + assert(m_slot_to_piece[m_current_slot] == unassigned + || m_piece_to_slot[m_slot_to_piece[m_current_slot]] == m_current_slot); + } + } + catch (file_error&) + { + // find the file that failed, and skip all the blocks in that file + size_type file_offset = 0; + size_type current_offset = m_current_slot * m_info.piece_length(); + for (torrent_info::file_iterator i = m_info.begin_files(); + i != m_info.end_files(); ++i) + { + file_offset += i->size; + if (file_offset > current_offset) break; + } + + assert(file_offset > current_offset); + int skip_blocks = static_cast( (file_offset - current_offset + m_info.piece_length() - 1) / m_info.piece_length()); - for (int i = current_slot; i < current_slot + skip_blocks; ++i) - { - assert(m_slot_to_piece[i] == unallocated); - m_unallocated_slots.push_back(i); - } - - // current slot will increase by one at the end of the for-loop too - current_slot += skip_blocks - 1; - } - - // Update progress meter and check if we've been requested to abort + for (int i = m_current_slot; i < m_current_slot + skip_blocks; ++i) { - boost::mutex::scoped_lock lock(mutex); - data.progress = (float)current_slot / m_info.num_pieces(); - if (data.abort || (data.torrent_ptr && data.torrent_ptr->is_aborted())) - return; + assert(m_slot_to_piece[i] == unallocated); + m_unallocated_slots.push_back(i); } + + // current slot will increase by one at the end of the for-loop too + m_current_slot += skip_blocks - 1; + } + ++m_current_slot; + + if (m_current_slot >= m_info.num_pieces()) + { + assert(m_current_slot == m_info.num_pieces()); + + // clear the memory we've been using + std::vector().swap(m_piece_data); + std::multimap().swap(m_hash_to_piece); + m_state = state_allocating; + return std::make_pair(false, 1.f); } - post_check(mutex, data); - - // TODO: sort m_free_slots and m_unallocated_slots? + return std::make_pair(false, (float)m_current_slot / m_info.num_pieces()); } - void piece_manager::check_pieces( - boost::mutex& mutex - , detail::piece_checker_data& data - , std::vector& pieces + bool piece_manager::check_fastresume( + detail::piece_checker_data& d, std::vector& pieces , bool compact_mode) { - m_pimpl->check_pieces(mutex, data, pieces, compact_mode); + return m_pimpl->check_fastresume(d, pieces, compact_mode); } - + + std::pair piece_manager::check_files( + std::vector& pieces) + { + return m_pimpl->check_files(pieces); + } + int piece_manager::impl::allocate_slot_for_piece(int piece_index) { // synchronization ------------------------------------------------------ boost::recursive_mutex::scoped_lock lock(m_mutex); // ---------------------------------------------------------------------- -// INVARIANT_CHECK; + // INVARIANT_CHECK; assert(piece_index >= 0); - assert(piece_index < (int)m_piece_to_slot.size()); + assert(piece_index < (int)m_piece_to_slot.size()); assert(m_piece_to_slot.size() == m_slot_to_piece.size()); int slot_index = m_piece_to_slot[piece_index]; @@ -1711,7 +1762,7 @@ namespace libtorrent m_slot_to_piece[slot_index] = piece_index; m_piece_to_slot[piece_index] = slot_index; - + // there is another piece already assigned to // the slot we are interested in, swap positions if (slot_index != piece_index @@ -1762,7 +1813,7 @@ namespace libtorrent } assert(slot_index >= 0); - assert(slot_index < (int)m_slot_to_piece.size()); + assert(slot_index < (int)m_slot_to_piece.size()); return slot_index; } @@ -1799,7 +1850,7 @@ namespace libtorrent boost::condition& m_cond; boost::mutex& m_monitor; }; - + } void piece_manager::impl::allocate_slots(int num_slots) @@ -1809,18 +1860,18 @@ namespace libtorrent // this object will syncronize the allocation with // potential other threads allocation_syncronization sync_obj( - m_allocating - , m_allocating_condition - , m_allocating_monitor); + m_allocating + , m_allocating_condition + , m_allocating_monitor); // synchronization ------------------------------------------------------ boost::recursive_mutex::scoped_lock lock(m_mutex); // ---------------------------------------------------------------------- -// INVARIANT_CHECK; + // INVARIANT_CHECK; assert(!m_unallocated_slots.empty()); - + const int piece_size = static_cast(m_info.piece_length()); std::vector buffer(piece_size, 0); @@ -1828,7 +1879,7 @@ namespace libtorrent for (int i = 0; i < num_slots && !m_unallocated_slots.empty(); ++i) { int pos = m_unallocated_slots.front(); -// int piece_pos = pos; + // int piece_pos = pos; bool write_back = false; int new_free_slot = pos; diff --git a/src/torrent.cpp b/src/torrent.cpp index 28f4bfe4e..6ebbd47ed 100755 --- a/src/torrent.cpp +++ b/src/torrent.cpp @@ -141,6 +141,7 @@ namespace libtorrent { torrent::torrent( detail::session_impl& ses + , detail::checker_impl& checker , entry const& metadata , boost::filesystem::path const& save_path , address const& net_interface @@ -159,6 +160,7 @@ namespace libtorrent , m_incomplete(-1) , m_policy() , m_ses(ses) + , m_checker(checker) , m_picker(0) , m_trackers(m_torrent_file.trackers()) , m_last_working_tracker(-1) @@ -220,6 +222,7 @@ namespace libtorrent torrent::torrent( detail::session_impl& ses + , detail::checker_impl& checker , char const* tracker_url , sha1_hash const& info_hash , boost::filesystem::path const& save_path @@ -239,6 +242,7 @@ namespace libtorrent , m_incomplete(-1) , m_policy() , m_ses(ses) + , m_checker(checker) , m_picker(0) , m_last_working_tracker(-1) , m_currently_trying_tracker(0) @@ -1018,30 +1022,27 @@ namespace libtorrent } - void torrent::check_files(detail::piece_checker_data& data, - boost::mutex& mutex, bool lock_session) + bool torrent::check_fastresume(detail::piece_checker_data& data) + { + assert(m_storage.get()); + return m_storage->check_fastresume(data, m_have_pieces, m_compact_mode); + } + + std::pair torrent::check_files() { assert(m_storage.get()); - m_storage->check_pieces(mutex, data, m_have_pieces, m_compact_mode); + return m_storage->check_files(m_have_pieces); + } - // TODO: temporary solution. This function should only - // be called from the checker thread, and then this - // hack can be removed (because the session should always - // be locked then) - boost::mutex temp; - boost::mutex* m = &temp; - if (lock_session) m = &m_ses.m_mutex; - - boost::mutex::scoped_lock l(mutex); - if (data.abort) return; - - boost::mutex::scoped_lock l2(*m); + void torrent::files_checked(std::vector const& + unfinished_pieces) + { m_num_pieces = std::count( m_have_pieces.begin() , m_have_pieces.end() , true); - m_picker->files_checked(m_have_pieces, data.unfinished_pieces); + m_picker->files_checked(m_have_pieces, unfinished_pieces); } alert_manager& torrent::alerts() const @@ -1469,18 +1470,29 @@ namespace libtorrent return false; } - m_torrent_file.parse_info_section(bdecode(m_metadata.begin(), m_metadata.end())); + entry metadata = bdecode(m_metadata.begin(), m_metadata.end()); + m_torrent_file.parse_info_section(metadata); - init(); + { + boost::mutex::scoped_lock(m_checker.m_mutex); - boost::mutex m; - detail::piece_checker_data d; - d.abort = false; - // TODO: this check should be moved to the checker thread - // not really a high priority, since no files would usually - // be available if the metadata wasn't available. - check_files(d, m, false); + boost::shared_ptr d( + new detail::piece_checker_data); + d->torrent_ptr = shared_from_this(); + d->save_path = m_save_path; + d->info_hash = m_torrent_file.info_hash(); + // add the torrent to the queue to be checked + m_checker.m_torrents.push_back(d); + typedef detail::session_impl::torrent_map torrent_map; + torrent_map::iterator i = m_ses.m_torrents.find( + m_torrent_file.info_hash()); + assert(i != m_ses.m_torrents.end()); + m_ses.m_torrents.erase(i); + // and notify the thread that it got another + // job in its queue + m_checker.m_cond.notify_one(); + } if (m_ses.m_alerts.should_post(alert::info)) { m_ses.m_alerts.post_alert(metadata_received_alert( @@ -1489,6 +1501,7 @@ namespace libtorrent // all peer connections have to initialize themselves now that the metadata // is available + // TODO: is it ok to initialize the connections before the file check? typedef std::map conn_map; for (conn_map::iterator i = m_connections.begin() , end(m_connections.end()); i != end; ++i) diff --git a/src/torrent_handle.cpp b/src/torrent_handle.cpp index de1119313..44b6160eb 100755 --- a/src/torrent_handle.cpp +++ b/src/torrent_handle.cpp @@ -236,8 +236,11 @@ namespace libtorrent { torrent_status st; - if (d == &m_chk->m_torrents.front()) + if (d->processing) + { + // TODO: this could be both checking or allocating st.state = torrent_status::checking_files; + } else st.state = torrent_status::queued_for_checking; st.progress = d->progress; diff --git a/test/test_storage.cpp b/test/test_storage.cpp index 858dde3f9..89cf28ead 100644 --- a/test/test_storage.cpp +++ b/test/test_storage.cpp @@ -79,7 +79,11 @@ int test_main() libtorrent::detail::piece_checker_data d; std::vector pieces; - pm.check_pieces(lock, d, pieces, true); + TEST_CHECK(pm.check_fastresume(d, pieces, true) == false); + bool finished = false; + float progress; + while (!finished) + boost::tie(finished, progress) = pm.check_files(pieces); pm.read(piece, 0, 0, piece_size); TEST_CHECK(std::equal(piece, piece + piece_size, piece0));