Libtorrent (Rasterbar)
Encyclopedia
libtorrent is an open source
implementation of the BitTorrent protocol. It is written in and has its main library interface in C++
. Its most notable features are support for Mainline DHT
, IPv6
(but not BEP 32), HTTP seeds and µTorrent's peer exchange.
libtorrent uses Boost, specifically Boost.Asio
to gain its platform independence. It is known to build on at least Windows
, Mac OS X
, Linux
, and FreeBSD
. In many package managers this library is known as libtorrent-rasterbar or the older rb-libtorrent.
Its original author is Arvid Norberg.
libtorrent was the first client to support the extension protocol together with µTorrent, which is now a foundation that many other extensions build upon.
Similarly, for write requests, blocks are cached and flushed to disk once one full piece is complete or the piece is the least recently updated one when more cache space is needed. The cache dynamically allocates space between the write and read cache. The write cache is strictly prioritized over the read cache.
The cache blocks that are in used, are locked into physical memory to avoid it being paged out to disk. Allowing the disk cache to be paged out to disk means that it would become extremely inefficient to flush it, since it would have to be read back into physical memory only to be flushed back out to disk again.
In order to conserve memory, and system calls, iovec file operations
are used to flush multiple cache blocks in a single call.
On low-memory systems, the disk cache can be disabled altogether or set to smaller limit, to save memory.
In order to minimize the number of times received data is copied, the receive buffer for payload data is received directly into a page aligned disk buffer. If the connection is encrypted, the buffer is decrypted in-place. The buffer is then moved into the disk cache without being copied. Once all the blocks for a piece have been received, or the cache needs to be flushed, all the blocks are passed directly to writev
to flush them in a single syscall. This means a single copy into user space memory, and a single copy back into kernel memory.
When seeding and uploading in general, unnecessary copying is avoided by caching blocks in aligned buffers, that are copied once into the peer's send buffer. The peer's send buffer is not guaranteed to be aligned, even though it is most of the time. The send buffer is then encrypted with the peer specific key and chained onto the iovec
for sending.
This means there is one user space copy in order to allow unaligned peer requests and peer-specific encryption.
The piece picker allows to combine the availability of a piece with a priority. Together they determine the sort order of the piece list. Pieces with priority 0 will never be picked, which is used for the selective download feature.
In order to have as few partially finished pieces as possible, peers have an affinity towards picking blocks from the same pieces as other peers in the same speed category.
The speed category is a coarse categorization of peers based on their download rate. This makes slow peers pick blocks from the same piece, and fast peers pick from the same piece, and hence decreasing the likelihood of slow peers blocking the completion of pieces.
The piece picker can also be set to download pieces in sequential order.
With regular torrents, clients have to request multiple blocks for pieces, typically from different peers, before the data can be verified against the piece hash. The larger the pieces are, the longer it will take to download a complete piece and verify it. Before the piece is verified, it cannot be shared with the swarm, which means the larger piece sizes, the slower turnaround data has when it is downloaded by peers.
Since on average the data has to sit around, waiting, in client buffers before it has been verified and can be uploaded again.
Another problem with large piece sizes is that it is harder for a client to pinpoint the malicious or buggy peer when a piece fails, and it will take longer to re-download it and take more tries before the piece succeeds the larger the pieces are.
The piece size in regular torrents is a tradeoff between the size of the .torrent file itself and the piece size. Often, for files that are 4 GB, the piece size is 2 or 4 MB, just to avoid making the .torrent file too big.
Merkle torrents solves these problems by removing the tradeoff between .torrent size and piece size. With merkle torrents, the piece size can be the minimum block size (16 kB), which lets peers verify every block of data received from peers, immediately. This gives a minimum turnaround time and completely removes the problem of identifying malicious peers.
Open source
The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
implementation of the BitTorrent protocol. It is written in and has its main library interface in C++
C++
C++ is a statically typed, free-form, multi-paradigm, compiled, general-purpose programming language. It is regarded as an intermediate-level language, as it comprises a combination of both high-level and low-level language features. It was developed by Bjarne Stroustrup starting in 1979 at Bell...
. Its most notable features are support for Mainline DHT
Distributed hash table
A distributed hash table is a class of a decentralized distributed system that provides a lookup service similar to a hash table; pairs are stored in a DHT, and any participating node can efficiently retrieve the value associated with a given key...
, IPv6
IPv6
Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...
(but not BEP 32), HTTP seeds and µTorrent's peer exchange.
libtorrent uses Boost, specifically Boost.Asio
Asio C++ library
Asio is a freely available, open source, cross-platform C++ library for network programming. It provides developers with a consistent asynchronous I/O model using a modern C++ approach....
to gain its platform independence. It is known to build on at least Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
, Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
, Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
, and FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
. In many package managers this library is known as libtorrent-rasterbar or the older rb-libtorrent.
Its original author is Arvid Norberg.
Overview
libtorrent is kept up to date with the most useful bittorrent extensions and is constantly being optimized to work in a broader set of environments. Many of its features can be disabled at compile time to not include code that would not be used in a particular use case. It aims to be the most suitable libtorrent implementation for embedded devices as well as desktops and seed-servers. Some of its implementation details are described in the features section.libtorrent was the first client to support the extension protocol together with µTorrent, which is now a foundation that many other extensions build upon.
Features
- plugin interface for implementing custom bittorrent extensions without having to modify libtorrent
- supports trackerless torrents (using the Mainline kademlia DHT protocol). BEP 5
- supports the bittorrent extension protocol, BEP 10.
- supports the µTorrent metadata transfer protocol (i.e. magnet links) BEP 9
- supports the µTorrent peer exchange protocol (PEX).
- supports local peer discovery (multicasts for peers on the same local network)
- multitracker extension support (supports both strict BEP 12 and the µTorrent interpretation).
- tracker scrapes
- supports lt_trackers extension, to exchange trackers between peers
- HTTP seeding, as specified in BEP 17 and BEP 19.
- supports the udp-tracker protocol. (BEP 15).
- supports the no_peer_id=1 extension that will ease the load off trackers.
- supports the compact=1 tracker parameter.
- super seeding/initial seeding (BEP16).
- private torrents (BEP 27).
- support for IPv6, including BEP 7 and BEP 24.
- support for merkle hash tree torrents. This makes the size of torrent files scale well with the size of the content.
- uses a separate disk I/O thread to not have the disk ever block on network or client interaction.
- supports files larger than 2 gigabytes on systems that support it.
- fast resume support, a way to get rid of the costly piece check at the start of a resumed torrent. Saves the storage state, piece_picker state as well as all local peers in a separate fast-resume file.
- has an adjustable read and write disk cache for improved disk throughput.
- queues torrents for file check, instead of checking all of them in parallel.
- does not have any requirements on the piece order in a torrent that it resumes. This means it can resume a torrent downloaded by any client.
- supports both sparse files and compact file allocation (where pieces are kept consolidated on disk)
- seed mode, where the files on disk are assumed to be complete, and each piece's hash is verified the first time it is requested.
- adjusts the length of the request queue depending on download rate.
- serves multiple torrents on a single port and in a single thread
- supports http proxies and basic proxy authentication
- supports gzipped tracker-responses
- can limit the upload and download bandwidth usage and the maximum number of unchoked peers
- possibility to limit the number of connections.
- delays have messages if there's no other outgoing traffic to the peer, and doesn't send have messages to peers that already has the piece. This saves bandwidth.
- selective downloading. The ability to select which parts of a torrent you want to download.
- ip filter to disallow ip addresses and ip ranges from connecting and being connected
- NAT-PMP and UPnP support (automatic port mapping on routers that support it)
Disk caching
All disk I/O in libtorrent is done asynchronously to the network thread, by the disk io thread. When a block is read, the disk io thread reads all subsequent blocks from that piece into the read cache, assuming that the peer requesting the block will also request more blocks from the same piece. This decreases the number of syscalls for reading data. It also decreases delay from seeking.Similarly, for write requests, blocks are cached and flushed to disk once one full piece is complete or the piece is the least recently updated one when more cache space is needed. The cache dynamically allocates space between the write and read cache. The write cache is strictly prioritized over the read cache.
The cache blocks that are in used, are locked into physical memory to avoid it being paged out to disk. Allowing the disk cache to be paged out to disk means that it would become extremely inefficient to flush it, since it would have to be read back into physical memory only to be flushed back out to disk again.
In order to conserve memory, and system calls, iovec file operations
Vectored I/O
In computing, vectored I/O, also known as scatter/gather I/O, is a method of input and output by which a single procedure-call sequentially writes data from multiple buffers to a single data stream or reads data from a data stream to multiple buffers. The buffers are given in a vector of buffers...
are used to flush multiple cache blocks in a single call.
On low-memory systems, the disk cache can be disabled altogether or set to smaller limit, to save memory.
Network buffers
On CPUs with small L2 caches, copying memory can be expensive operations. It is important to keep copying to a minimum on such machines. This mostly applies to embedded systems.In order to minimize the number of times received data is copied, the receive buffer for payload data is received directly into a page aligned disk buffer. If the connection is encrypted, the buffer is decrypted in-place. The buffer is then moved into the disk cache without being copied. Once all the blocks for a piece have been received, or the cache needs to be flushed, all the blocks are passed directly to writev
Vectored I/O
In computing, vectored I/O, also known as scatter/gather I/O, is a method of input and output by which a single procedure-call sequentially writes data from multiple buffers to a single data stream or reads data from a data stream to multiple buffers. The buffers are given in a vector of buffers...
to flush them in a single syscall. This means a single copy into user space memory, and a single copy back into kernel memory.
When seeding and uploading in general, unnecessary copying is avoided by caching blocks in aligned buffers, that are copied once into the peer's send buffer. The peer's send buffer is not guaranteed to be aligned, even though it is most of the time. The send buffer is then encrypted with the peer specific key and chained onto the iovec
Vectored I/O
In computing, vectored I/O, also known as scatter/gather I/O, is a method of input and output by which a single procedure-call sequentially writes data from multiple buffers to a single data stream or reads data from a data stream to multiple buffers. The buffers are given in a vector of buffers...
for sending.
This means there is one user space copy in order to allow unaligned peer requests and peer-specific encryption.
Piece picker
The piece picker is a central component in a bittorrent implementation. The piece picker in libtorrent is optimized for quickly finding the rarest pieces. It keeps a list of all available pieces sorted by rarity, and pieces with the same rarity, shuffled. The rarest first mode is the dominant piece picker mode. Other modes are supported as well, and used by peers in specific situations.The piece picker allows to combine the availability of a piece with a priority. Together they determine the sort order of the piece list. Pieces with priority 0 will never be picked, which is used for the selective download feature.
In order to have as few partially finished pieces as possible, peers have an affinity towards picking blocks from the same pieces as other peers in the same speed category.
The speed category is a coarse categorization of peers based on their download rate. This makes slow peers pick blocks from the same piece, and fast peers pick from the same piece, and hence decreasing the likelihood of slow peers blocking the completion of pieces.
The piece picker can also be set to download pieces in sequential order.
Merkle hash tree torrents
Merkle hash tree torrents is an extension that lets a torrent file only contain the root hash of the hash tree forming the piece hashes. The main benefit of this feature is that regardless of how many pieces there are in a torrent, the .torrent file will always be the same size. It will only grow with the number of files (since it still has to contain the file names).With regular torrents, clients have to request multiple blocks for pieces, typically from different peers, before the data can be verified against the piece hash. The larger the pieces are, the longer it will take to download a complete piece and verify it. Before the piece is verified, it cannot be shared with the swarm, which means the larger piece sizes, the slower turnaround data has when it is downloaded by peers.
Since on average the data has to sit around, waiting, in client buffers before it has been verified and can be uploaded again.
Another problem with large piece sizes is that it is harder for a client to pinpoint the malicious or buggy peer when a piece fails, and it will take longer to re-download it and take more tries before the piece succeeds the larger the pieces are.
The piece size in regular torrents is a tradeoff between the size of the .torrent file itself and the piece size. Often, for files that are 4 GB, the piece size is 2 or 4 MB, just to avoid making the .torrent file too big.
Merkle torrents solves these problems by removing the tradeoff between .torrent size and piece size. With merkle torrents, the piece size can be the minimum block size (16 kB), which lets peers verify every block of data received from peers, immediately. This gives a minimum turnaround time and completely removes the problem of identifying malicious peers.
Applications
Some applications that use libtorrent:- Arctic Torrent, Windows BitTorrent client
- BitRocket, Mac OS X BitTorrent client
- BitSlug, Mac OS X BitTorrent client
- BTG, Linux BitTorrent client
- RutrackerDownloader, Android BitTorrent client
- Deluge, cross-platform BitTorrent client
- Electric SheepElectric SheepElectric Sheep is a distributed computing project for animating and evolving fractal flames, which are in turn distributed to the networked computers, which display them as a screensaver.-Process:...
screen saver, BitTorrent client for screensaver - Free Download ManagerFree Download ManagerFree Download Manager is a free open source graphical download manager for the Microsoft Windows operating system.FDM was previously proprietary software, but with the release of version 2.5 it has been open source. Starting with version 3.0.850 , source code is no longer provided as a packaged...
, Windows open source download manager - FatRat, Linux Qt 4-based download/upload manager
- Halite, Windows BitTorrent client
- hrktorrent, Linux BitTorrent client
- LeechCraft, C++ / Qt4 cross-platform multiprotocol performance-concerned client
- LimeWireLimeWireLimeWire is a free peer-to-peer file sharing client program that runs on Windows, Mac OS X, Linux, and other operating systems supported by the Java software platform. LimeWire uses the gnutella network as well as the BitTorrent protocol. A free software version and a purchasable "enhanced"...
, multi-platform file sharing client - Linkage, Linux BitTorrent client
- Miro, a cross-platform Internet television application
- MooPolice, Windows BitTorrent client
- qBittorrentQbittorrentqBittorrent is a free software cross-platform BitTorrent client GUI written with Qt4. The program uses libtorrent-rasterbar C++ library for the torrent back-end functionality. It is developed by Christophe Dumez, from the University of Technology of Belfort-Montbeliard in France...
, C++ / Qt4 BitTorrent client - Lince, C++ / GTK BitTorrent client
- p2ptube, Program to stream movies on the internet
- SharkTorrent, QtQt (toolkit)Qt is a cross-platform application framework that is widely used for developing application software with a graphical user interface , and also used for developing non-GUI programs such as command-line tools and consoles for servers...
4 cross-platform BitTorrent client - tvitty, BitTorrent client plugin for media center
- ZipTorrentZipTorrentThis article refers to the original Ziptorrent C++ Bittorrent client, and the 2007 version of Azureus modified specifically by MediaDefender Inc. to disrupt Bittorrent downloads....
, Windows BitTorrent client - Other applications that use libtorrent
- Runes of Magic, an MMORPG whose FOG downloader uses libtorrent for updating the game client.
See also
- Comparison of BitTorrent libraries