Segmented downloading - AbsoluteAstronomy.com

Segmented downloading can be a more efficient way of downloading file

Computer file

A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...

s from many peers at once. The one single file is downloaded, in parallel, from several distinct sources or uploaders of the file. This can help a group of users with asymmetric

Asymmetric

Something which is asymmetric displays asymmetry. Specific uses of the term may include:*Asymmetric relation for information on such relations in mathematics and set theory*Asymmetric warfare for information and theories of modern war...

connections, such as ADSL to provide a high total bandwidth

Bandwidth (computing)

In computer networking and computer science, bandwidth, network bandwidth, data bandwidth, or digital bandwidth is a measure of available or consumed data communication resources expressed in bits/second or multiples of it .Note that in textbooks on wireless communications, modem data transmission,...

to one downloader, and to handle peaks in download demand.

History

Segmented downloads probably have an origin with NASA

NASA

The National Aeronautics and Space Administration is the agency of the United States government that is responsible for the nation's civilian space program and for aeronautics and aerospace research...

and the magnetic tape

Magnetic tape

Magnetic tape is a medium for magnetic recording, made of a thin magnetizable coating on a long, narrow strip of plastic. It was developed in Germany, based on magnetic wire recording. Devices that record and play back audio and video using magnetic tape are tape recorders and video tape recorders...

based file systems used on Deep Space Network

Deep Space Network

The Deep Space Network, or DSN, is a world-wide network of large antennas and communication facilities that supports interplanetary spacecraft missions. It also performs radio and radar astronomy observations for the exploration of the solar system and the universe, and supports selected...

craft such as those in the Voyager Program

Voyager program

The Voyager program is a U.S program that launched two unmanned space missions, scientific probes Voyager 1 and Voyager 2. They were launched in 1977 to take advantage of a favorable planetary alignment of the late 1970s...

.

NASA missions using some kind of segmented downloading

Mars Rovers (for ICER
ICER
ICER is a wavelet-based image compression file format used by the NASA Mars Rovers. ICER has both lossy and lossless compression modes.The Mars Exploration Rovers “Spirit” and “Opportunity” both use ICER...

image files)
New Horizons
New Horizons
New Horizons is a NASA robotic spacecraft mission currently en route to the dwarf planet Pluto. It is expected to be the first spacecraft to fly by and study Pluto and its moons, Charon, Nix, Hydra and S/2011 P 1. Its estimated arrival date at the Pluto-Charon system is July 14th, 2015...

(for Jupiter
Jupiter
Jupiter is the fifth planet from the Sun and the largest planet within the Solar System. It is a gas giant with mass one-thousandth that of the Sun but is two and a half times the mass of all the other planets in our Solar System combined. Jupiter is classified as a gas giant along with Saturn,...

flyby data)
Voyager Program
Voyager program
The Voyager program is a U.S program that launched two unmanned space missions, scientific probes Voyager 1 and Voyager 2. They were launched in 1977 to take advantage of a favorable planetary alignment of the late 1970s...

(historical)

Swarmcast

Swarmcast is a provider of Internet television service. It is headquartered in Minneapolis and Tokyo with branch offices in Zurich, San Francisco and New York City. It employs approximately 50 people.- History :...

was the first significant peer-to-peer

Peer-to-peer

Peer-to-peer computing or networking is a distributed application architecture that partitions tasks or workloads among peers. Peers are equally privileged, equipotent participants in the application...

(P2P) content delivery system that implemented a kind of segmented downloading technology. The program and protocol was invented and developed in 1999 by Justin Chapweske and sold to Opencola

OpenCola

OpenCola or Opencola may refer to:* OpenCola , a brand of cola with an openly available recipe* Opencola , Canadian software company...

, which released the software under a GPL license.

A lot of the terms used in segmented downloading technology have their origin with Swarmcast

Swarmcast

, with BitTorrent being the only other significant contributor to the terms in use.

Network implications

Most IP networks are designed for users to download more than they upload, usually with an expected (Download:Upload) ratio of 3:1 or more.

Segmented downloading, when used by only 20% of an ISP's user base, can upset the ISP's network to a point of requiring substantial reprogramming of routers and a rethink of network design.

Traditional web object caching technology (like the Squid proxy) is of no use here.
Universal adoption of IPv6
IPv6
Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...

cannot help either, as it only allows all users to have fixed IP addresses. Fixed IP address don't fully address the routing table problems associated with segmented downloading.
Typical downloading configurations can have a single user in touch with up to 10 to 30 ephemeral users per file scattered across the global internet.
IP router tables can become bloated with routes to these ephemeral users slowing down table lookups.

Network advantages

Large files can be made available efficiently to many other users by someone who does not have large upload bandwidth.
routes to the more obscure parts of the Internet can assert themselves across most of the Internet—this is especially true for dial-up users
segmented downloading does save some transmission capacity, as the number of lost or redundant megabytes is minimal compared to losing a prolonged http or ftp download

Most ISPs have learned to cope with segmented downloading technology, but coping has meant the mandatory deployment of TCP/IP traffic shaping technology.

Limitations

Segmented downloading technology cannot magically solve all downloading problems. There are mathematical constraints on the effectiveness of the technology.

In a group of users that has insufficient upload-bandwidth, with demand

Demand (economics)

In economics, demand is the desire to own anything, the ability to pay for it, and the willingness to pay . The term demand signifies the ability or the willingness to buy a particular commodity at a given point of time....

higher than supply

Supply (economics)

In economics, supply is the amount of some product producers are willing and able to sell at a given price all other factors being held constant. Usually, supply is plotted as a supply curve showing the relationship of price to the amount of product businesses are willing to sell.In economics the...

. Segmented downloading can however very nicely handle traffic peaks, and it can also, to some degree, let uploaders upload "more often" to better utilize their connection.

Data integrity issues

Very simple implementations of segmented downloading technology can often result in varying levels of file corruption, as there often is no way of knowing if all sources are actually uploading segments of the same file.
Data corruption problems have led to most programs using segmented downloading using some sort of checksum
Checksum
A checksum or hash sum is a fixed-size datum computed from an arbitrary block of digital data for the purpose of detecting accidental errors that may have been introduced during its transmission or storage. The integrity of the data can be checked at any later time by recomputing the checksum and...

or hash algorithm to ensure file integrity (to receive file intact) and uniqueness (to not receive bits of other similar files).
Usually MD5
MD5
The MD5 Message-Digest Algorithm is a widely used cryptographic hash function that produces a 128-bit hash value. Specified in RFC 1321, MD5 has been employed in a wide variety of security applications, and is also commonly used to check data integrity...

and SHA-1 hashes are preferred in most segmented download protocols, but CRC-64-ECMA would suffice in most cases. In cases where only MPEG files are being sent CRC-32-MPEG would also be acceptable.
In the future most segmented downloading technologies will probably use layered hashes and checksums like WHIRLPOOL, SHA-256, SHA-512 and CRC-64-ECMA (for individual segments) to unquestionably guarantee data integrity. MD5
MD5
The MD5 Message-Digest Algorithm is a widely used cryptographic hash function that produces a 128-bit hash value. Specified in RFC 1321, MD5 has been employed in a wide variety of security applications, and is also commonly used to check data integrity...

and SHA-1 have been determined to be cryptographically weak with respect to protecting data integrity.

Segmented uploading

Although with respect to BitTorrent and other distributed file transfer protocols there is no difference between uploading and downloading (as clients can do both) or any meaningful distinction between client and server (as both are the same) there are some segmented uploading technologies that do exist.

Space segment based telecom systems are the only widely known cases where segmented uploading technologies have emerged. This is mainly due to the limited bandwidth and other space segment constraints.

CCSDS software uploading protocols have the capability of segmented uploading, but current deployed systems have not been in need of the protocol being used in its most BitTorrent like capability.
Satellite direct to home subscription systems deployed in Europe and North America have employed an approach of upgrading software on customer devices by only sending a few bytes at a time (~2k or less) over a long period of time. Generally these segmented upload approaches are proprietary and related to the SIM
Subscriber Identity Module
A subscriber identity module or subscriber identification module is an integrated circuit that securely stores the International Mobile Subscriber Identity and the related key used to identify and authenticate subscriber on mobile telephony devices .A SIM is held on a removable SIM card, which...

card security and subscription mechanism.

With respect to Direct To Home TV systems using segmented uploading to outwit "hackers" — only SkyTV (UK) and DirecTV (USA) have been possibly linked to having the capability to do so or have done so in the past. However, one can assume that any modern MPEG2 DVB DTH mass subscriber system has the ability to accept software upgrades trickled to it at the rate of 8kb/day or less.

Examples