User agent
Encyclopedia
In computing
, a user agent is a client
application implementing a network protocol used in communications within a client–server distributed computing system. The term most notably refers to applications that access the World Wide Web
, but other systems, such as the Session Initiation Protocol
(SIP), use the term user agent to refer to both end points of a communications session.
Web user agents range from Web browser
s to search engine
crawler
s (spiders), as well as mobile phone
s, screen reader
s and braille
browsers used by people with disabilities.
When a user agent operates, it typically identifies itself, its application type, operating system
, software vendor, or software revision, by submitting a characteristic identification string
to its operating peer. In the HTTP
, SIP, and SMTP/NNTP protocols, this is transmitted in a header field User-Agent. Bot
s, such as Web crawlers, often also include a URL
and/or e-mail address
so that the Webmaster
can contact the operator of the bot.
s and for smartphone
s.
The user agent string is one of the criteria by which Web crawlers may be excluded from accessing certain parts of a Web site using the Robots Exclusion Standard
(robots.txt file).
The parts of this string are as follows:
An unofficial format, based on the above, used by Web browsers is as follows:
Mozilla/[version] ([system and browser information]) [platform] ([platform details]) [extensions]. For example, Safari on the iPad has used the following:
Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405
The components of this string are as follows:
(W3C) or the Internet Engineering Task Force
(IETF). Web sites often include code to detect browser version to adjust the page design sent according to the user agent string received. This may mean that less-popular browsers are not sent complex content, even though they might be able to deal with it correctly or, in extreme cases, refused all content. Thus, various browsers have a feature to cloak or spoof their identification to force certain server-side content. For example, the Android browser identifies itself as Safari
in order to aid compatibility.
Other HTTP client programs, like download manager
s and offline browsers, also have the ability to change the user agent string.
Spam bots and Web scrapers often use fake user agents.
At times it has been popular among Web developers to initiate Viewable With Any Browser campaigns, encouraging developers to design Web pages that work equally well with any browser.
A result of user agent spoofing may be that collected statistics of Web browser usage
are inaccurate.
2003's Outlook Web Access feature. When viewed with Internet Explorer 6 (or newer), more functionality is displayed compared to the same page in any older browsers, because older browsers could not render the same content. User agent sniffing is mostly considered poor practice, since it encourages browser-specific design and penalizes new browsers with unrecognized user agent identifications. Instead, the W3C recommends creating HTML markup that is standard, allowing correct rendering in as many browsers as possible, and to test for specific browser features rather than particular browser versions or brands.
Web sites specifically targeted towards mobile phones, like NTT DoCoMo
's I-Mode
or Vodafone
's Vodafone Live!
portals, often rely heavily on user agent sniffing, since mobile browsers often differ greatly from each other. Many developments in mobile browsing have been made in the last few years, while many older phones that do not possess these new technologies are still heavily used. Therefore, mobile Web portals will often generate completely different markup code depending on the mobile phone used to browse them. These differences can be small, e.g., resizing of certain images to fit smaller screens, or quite extensive, e.g., rendering of the page in WML
instead of XHTML
.
and Internet Explorer
, use the letters U, I, and N to specify the encryption
strength in the user agent string. Until the United States government allowed encryption with keys longer than 40 bits to be exported, in 1996, vendors shipped various browser versions with different encryption strengths. "U" stands for "USA" (for the version with 128-bit encryption), "I" stands for "International" — the browser has 40-bit encryption and can be used anywhere in the world — and "N" stands (de facto) for "None" (no encryption). Following the lifting of export restrictions, most vendors supported 256-bit encryption.
Computing
Computing is usually defined as the activity of using and improving computer hardware and software. It is the computer-specific part of information technology...
, a user agent is a client
Client (computing)
A client is an application or system that accesses a service made available by a server. The server is often on another computer system, in which case the client accesses the service by way of a network....
application implementing a network protocol used in communications within a client–server distributed computing system. The term most notably refers to applications that access the World Wide Web
World Wide Web
The World Wide Web is a system of interlinked hypertext documents accessed via the Internet...
, but other systems, such as the Session Initiation Protocol
Session Initiation Protocol
The Session Initiation Protocol is an IETF-defined signaling protocol widely used for controlling communication sessions such as voice and video calls over Internet Protocol . The protocol can be used for creating, modifying and terminating two-party or multiparty sessions...
(SIP), use the term user agent to refer to both end points of a communications session.
Web user agents range from Web browser
Web browser
A web browser is a software application for retrieving, presenting, and traversing information resources on the World Wide Web. An information resource is identified by a Uniform Resource Identifier and may be a web page, image, video, or other piece of content...
s to search engine
Web search engine
A web search engine is designed to search for information on the World Wide Web and FTP servers. The search results are generally presented in a list of results often referred to as SERPS, or "search engine results pages". The information may consist of web pages, images, information and other...
crawler
Web crawler
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.This process is called Web...
s (spiders), as well as mobile phone
Mobile phone
A mobile phone is a device which can make and receive telephone calls over a radio link whilst moving around a wide geographic area. It does so by connecting to a cellular network provided by a mobile network operator...
s, screen reader
Screen reader
A screen reader is a software application that attempts to identify and interpret what is being displayed on the screen . This interpretation is then re-presented to the user with text-to-speech, sound icons, or a Braille output device...
s and braille
Braille
The Braille system is a method that is widely used by blind people to read and write, and was the first digital form of writing.Braille was devised in 1825 by Louis Braille, a blind Frenchman. Each Braille character, or cell, is made up of six dot positions, arranged in a rectangle containing two...
browsers used by people with disabilities.
When a user agent operates, it typically identifies itself, its application type, operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
, software vendor, or software revision, by submitting a characteristic identification string
String (computer science)
In formal languages, which are used in mathematical logic and theoretical computer science, a string is a finite sequence of symbols that are chosen from a set or alphabet....
to its operating peer. In the HTTP
Hypertext Transfer Protocol
The Hypertext Transfer Protocol is a networking protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web....
, SIP, and SMTP/NNTP protocols, this is transmitted in a header field User-Agent. Bot
Internet bot
Internet bots, also known as web robots, WWW robots or simply bots, are software applications that run automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone...
s, such as Web crawlers, often also include a URL
Uniform Resource Locator
In computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
and/or e-mail address
E-mail address
An email address identifies an email box to which email messages are delivered. An example format of an email address is lewis@example.net which is read as lewis at example dot net...
so that the Webmaster
Webmaster
A webmaster , also called a web architect, web developer, site author, or website administrator is a person responsible for maintaining one or many websites...
can contact the operator of the bot.
User agent identification
Some user agents identify their software as part of the client–server conversation. In HTTP and SIP, the identity is transmitted via theUser-Agent field in the request header, as described by RFC 1945. This string is then used by the communications partner to characterize the client and optionally select suitable content or operating parameters for the session. For example, this may be used to provide properly formatted content for desktop computerPersonal computer
A personal computer is any general-purpose computer whose size, capabilities, and original sales price make it useful for individuals, and which is intended to be operated directly by an end-user with no intervening computer operator...
s and for smartphone
Smartphone
A smartphone is a high-end mobile phone built on a mobile computing platform, with more advanced computing ability and connectivity than a contemporary feature phone. The first smartphones were devices that mainly combined the functions of a personal digital assistant and a mobile phone or camera...
s.
The user agent string is one of the criteria by which Web crawlers may be excluded from accessing certain parts of a Web site using the Robots Exclusion Standard
Robots Exclusion Standard
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to...
(robots.txt file).
Possible privacy issue
The information could help distinguish Internet users from one another because it differs, often considerably, from user to user and on average browser versions carry 10.5 bits of identifying information.Format
RFC 1945 merely requires the format of a user agent to be a string made up of the product and optional comments. For example if your product were called WikiBrowser, your user agent string may be similar to WikiBrowser/1.0 Gecko/1.0.The parts of this string are as follows:
- Product name and version (WikiBrowser/1.0)
- Layout engine and version(Gecko/1.0). In this case, this indicates the Layout engine and version.
An unofficial format, based on the above, used by Web browsers is as follows:
Mozilla/[version] ([system and browser information]) [platform] ([platform details]) [extensions]. For example, Safari on the iPad has used the following:
Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405
The components of this string are as follows:
- Mozilla/5.0: Previously used to indicate compatibility with the Mozilla rendering engine
- (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us): Details of the system in which the browser is running
- AppleWebKit/531.21.10: The platform the browser uses
- (KHTML, like Gecko): Browser platform details
- Mobile/7B405: This is used by the browser to indicate specific enhancements that are available directly in the browser or through third parties. An example of this is Microsoft Live Meeting which registers an extension so that the Live Meeting service knows if the software is already installed, which means it can provide a streamlined experience to joining meetings.
User agent spoofing
The popularity of various Web browser products has varied throughout the Web's history, and this has influenced the design of Web sites in such a way that Web sites are sometimes designed to work well only with particular browsers, rather than according to uniform standards by the World Wide Web ConsortiumWorld Wide Web Consortium
The World Wide Web Consortium is the main international standards organization for the World Wide Web .Founded and headed by Tim Berners-Lee, the consortium is made up of member organizations which maintain full-time staff for the purpose of working together in the development of standards for the...
(W3C) or the Internet Engineering Task Force
Internet Engineering Task Force
The Internet Engineering Task Force develops and promotes Internet standards, cooperating closely with the W3C and ISO/IEC standards bodies and dealing in particular with standards of the TCP/IP and Internet protocol suite...
(IETF). Web sites often include code to detect browser version to adjust the page design sent according to the user agent string received. This may mean that less-popular browsers are not sent complex content, even though they might be able to deal with it correctly or, in extreme cases, refused all content. Thus, various browsers have a feature to cloak or spoof their identification to force certain server-side content. For example, the Android browser identifies itself as Safari
Safari (web browser)
Safari is a web browser developed by Apple Inc. and included with the Mac OS X and iOS operating systems. First released as a public beta on January 7, 2003 on the company's Mac OS X operating system, it became Apple's default browser beginning with Mac OS X v10.3 "Panther". Safari is also the...
in order to aid compatibility.
Other HTTP client programs, like download manager
Download manager
A download manager is a computer program dedicated to the task of downloading possibly unrelated stand-alone files from the Internet for storage...
s and offline browsers, also have the ability to change the user agent string.
Spam bots and Web scrapers often use fake user agents.
At times it has been popular among Web developers to initiate Viewable With Any Browser campaigns, encouraging developers to design Web pages that work equally well with any browser.
A result of user agent spoofing may be that collected statistics of Web browser usage
Usage share of web browsers
The usage share of a web browser is the proportion, often expressed as a percentage, of users of all web browsers who use that particular browser. This figure can only be estimated, typically by determining the proportion of visitors to a group of websites that use a particular web browser...
are inaccurate.
User agent sniffing
The term user agent sniffing refers to the practice of Web sites showing different content when viewed with a certain user agent. On the Internet, this will result in a different site being shown when browsing the page with a specific browser. A useful example of this is Microsoft Exchange ServerMicrosoft Exchange Server
Microsoft Exchange Server is the server side of a client–server, collaborative application product developed by Microsoft. It is part of the Microsoft Servers line of server products and is used by enterprises using Microsoft infrastructure products...
2003's Outlook Web Access feature. When viewed with Internet Explorer 6 (or newer), more functionality is displayed compared to the same page in any older browsers, because older browsers could not render the same content. User agent sniffing is mostly considered poor practice, since it encourages browser-specific design and penalizes new browsers with unrecognized user agent identifications. Instead, the W3C recommends creating HTML markup that is standard, allowing correct rendering in as many browsers as possible, and to test for specific browser features rather than particular browser versions or brands.
Web sites specifically targeted towards mobile phones, like NTT DoCoMo
NTT DoCoMo
is the predominant mobile phone operator in Japan. The name is officially an abbreviation of the phrase, "do communications over the mobile network", and is also from a compound word dokomo, meaning "everywhere" in Japanese. Docomo provides phone, video phone , i-mode , and mail services...
's I-Mode
I-mode
NTT DoCoMo's i-mode is a mobile internet service popular in Japan. Unlike Wireless Application Protocol, i-mode encompasses a wider variety of internet standards, including web access, e-mail and the packet-switched network that delivers the data...
or Vodafone
Vodafone
Vodafone Group Plc is a global telecommunications company headquartered in London, United Kingdom. It is the world's largest mobile telecommunications company measured by revenues and the world's second-largest measured by subscribers , with around 341 million proportionate subscribers as of...
's Vodafone Live!
Vodafone live!
Vodafone live! was the brand name for the multimedia portal service of mobile phone operator Vodafone, which was initially developed by Japan's J-Phone under the J-Sky brand...
portals, often rely heavily on user agent sniffing, since mobile browsers often differ greatly from each other. Many developments in mobile browsing have been made in the last few years, while many older phones that do not possess these new technologies are still heavily used. Therefore, mobile Web portals will often generate completely different markup code depending on the mobile phone used to browse them. These differences can be small, e.g., resizing of certain images to fit smaller screens, or quite extensive, e.g., rendering of the page in WML
Wireless Markup Language
Wireless Markup Language , based on XML, is a markup language intended for devices that implement the Wireless Application Protocol specification, such as mobile phones. It provides navigational support, data input, hyperlinks, text and image presentation, and forms, much like HTML...
instead of XHTML
XHTML
XHTML is a family of XML markup languages that mirror or extend versions of the widely-used Hypertext Markup Language , the language in which web pages are written....
.
Encryption strength notations
Web browsers created in the United States, such as Netscape NavigatorNetscape Navigator
Netscape Navigator was a proprietary web browser that was popular in the 1990s. It was the flagship product of the Netscape Communications Corporation and the dominant web browser in terms of usage share, although by 2002 its usage had almost disappeared...
and Internet Explorer
Internet Explorer
Windows Internet Explorer is a series of graphical web browsers developed by Microsoft and included as part of the Microsoft Windows line of operating systems, starting in 1995. It was first released as part of the add-on package Plus! for Windows 95 that year...
, use the letters U, I, and N to specify the encryption
Encryption
In cryptography, encryption is the process of transforming information using an algorithm to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key. The result of the process is encrypted information...
strength in the user agent string. Until the United States government allowed encryption with keys longer than 40 bits to be exported, in 1996, vendors shipped various browser versions with different encryption strengths. "U" stands for "USA" (for the version with 128-bit encryption), "I" stands for "International" — the browser has 40-bit encryption and can be used anywhere in the world — and "N" stands (de facto) for "None" (no encryption). Following the lifting of export restrictions, most vendors supported 256-bit encryption.
See also
- Robots Exclusion StandardRobots Exclusion StandardThe Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to...
- Web crawlerWeb crawlerA Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.This process is called Web...
- Wireless Universal Resource File (WURFL)WurflWURFL stands for Wireless Universal Resource FiLe. It is a community effort focused on mobile device detection: the problem of presenting content on the wide variety of wireless devices. WURFL is a set of proprietary API's and an XML configuration file which contains information about device...
- User Agent Profile (UAProf)UAProfThe UAProf specification is concerned with capturing capability and preference information for wireless devices...
- Apache Mobile FilterApache Mobile FilterApache Mobile Filter or AMF, is an open source project created by Idel Fuschini and distributed under the GNU General Public License...