E-mail authentication
Encyclopedia
Email authentication is the effort to equip messages of the email
transport system with enough verifiable information, so that recipients can recognize the nature of each incoming message automatically. It differs from content filtering
.
(as email can be filtered based on such an identity), forgery, fraud, and even more serious crimes. The Simple Mail Transfer Protocol
(SMTP) is continuously evolving, but when it was designed, in the early 1980s, it was the purview of academia and government agencies, and as such, there was no cause to consider security. It provided for no formal verification of sender.
Signing emails is a good first step towards identifying the origin of the message, but it does not establish whether that identity has a good reputation or whether it should be trusted.
This article explains how email identities are forged and the steps that are being taken now to prevent it.
Thanks to the Transmission Control Protocol
and to IP address
registries
the senders IP address is automatically verified by the receiver. However, there is no provision for the verification of the author and sender information that is eventually saved in the relevant headers. Thus, it is quite easy for a spammer to make an exact copy of an email from example.com, including a long complicated sequence of headers and a genuine logo in the body of an email, then change the content to send readers to a website that appears to be genuine, but is actually a phishing
scam designed to capture names, passwords, and credit card numbers.
IP addresses within the headers thus cannot be used to identify the source of a forged email. In addition to forged headers being used to throw off identification attempts, it is not uncommon for the legitimate headers within the email to reference a system insulated from the forger; the sender may, for example, be a zombie
that has been compromised by a Trojan horse
. Service providers can block this type of email using IP address reputation technology. IP address reputation is a technology that monitors emails based on their source IP address. The technology tracks the email for spam activity and blocks messages from that sender once spam activity begins. Service providers use this technology to prevent zombie
s from sending spam. Amir Lev, from email security company Commtouch
, described the technology as, "IP[address] reputation pulls the mask off of anonymous spam and malware senders, exposing their identity as spammers at the outer edge of the enterprise, long before any of their viruses, worms and trojans can penetrate into the organization."
. Within that block, they assign individual addresses to customers as needed. A dial-up customer may get a new IP address each time they connect. By the time that address appears on blacklists all over the world, the spammer will have new addresses for the next run. There are 4 billion possible IPv4
addresses on the Internet. The game of keeping up with these rapidly changing IP addresses has been facetiously called "whack-a-mole".
So called policy lists are blacklists that contain IP addresses on a preventive basis. An IP address can be listed therein even if no spam has ever been sent from it, because it has been variously classified as a dial-up address, end-user address, or residential address, with no formal definition of such classification schemes. Not requiring evidence of spam for each listed address, these lists can collect a greater number of addresses and thus block more spam. However, the policies devised are not authoritative, since they have not been issued by the legitimate user of an IP address, and the resulting lists are therefore not universally accepted.
Some ISPs have been quite successful, but others don't care to make the effort. With spam now over 80% of all email traffic, we can expect that there will always be ISPs who are not willing to take the necessary steps. The measures mentioned above don't directly help the entity who operates them to reduce incoming spam. By reducing outgoing spam, they help generic Internet users.
(MTA), it is then possible to treat suspected forgeries with suspicion, reject known forgeries, and block email from known spamming domains. It is also possible to "whitelist" email from known reputable domains, and bypass content-based filtering, which always loses some valid emails in the flood of spam. The fourth category, email from unknown domains, can be treated like we now treat all email – increasingly rigorous filtering, return challenges to the sender, etc.
Success of a domain-rating system should encourage reputable ISPs to stop their outgoing spam and get a good rating. It has been in the ISP's interest to shift to blocking by the domain, rather than by the IP address that they directly control.
A number of methods claim to authenticate a sender's domain name ( SPF
, SenderID, DKIM ). However SPF and SenderID only authorize the IP address of the outbound MTA. It is uncertain whether the PRA (Purported Responsible Address) associated with the SPF record was intended, and thereby protected when the outbound MTA is shared by other domains which is why sensitive domains must take additional precautions. While the outbound MTA, not being authorized when there is a strict SPF record, can block much of the forgery now prevalent, it is never safe to expect that the MTA, being authorized, represents a form of authentication. None exclude the use of other methods, although SPF / DKIM, SPF / SenderID, and SenderID / DKIM pairwise appear to be competing for the same niches. The most widely used will likely be the ones that require the least effort on the part of ISPs and others currently operating public mail servers.
SPF and SenderID use the domain to authorize the outbound MTA. DKIM uses a digital signature
to authenticate a domain name and the entire content of a message. SPF can reject a forgery before any data transfer. SenderID and DKIM must see at least the headers, so the entire message must be transmitted. SPF has a problem with forwarders (that Sender Rewriting Scheme
defines a fix for), SenderID also with mailing lists (see below).
SPF and SenderID work by having the domain publish what might be a sizeable list of IP addresses that identify MTAs they authorize to carry their email. This works as a type of path registration, but does not scale well since it also combines both IPv4 and IPv6 addresses. Every incoming email has an IP address that is difficult to forge, a bunch of domain names in the email headers, and a few more in the commands from the sender's SMTP server. The methods differ in which of these names to use as the sender's domain name. All of them can be faked, but what cannot be faked is a domain name held by a DNS server for that section of the Internet.
The simplest and by far most widely deployed authentication scheme begins with a reverse DNS lookup of the connecting IP address. If there is no answer, it's a safe bet that the address is not a legitimate sender. If there is an answer, a forward DNS lookup of that answer authenticates the sender if it returns the connecting IP address. In other words, we look up the name of the connecting IP address, and look up the IP address of that name, and they must match.
The procedure to authenticate is basically simple. When a request to deliver an email arrives, the claimed sender's domain name is sent in a query to a high-level DNS server. That DNS server in turn, refers to lower level servers until an answer is found that is authoritative for the domain in question. The answer returned to the receiver includes the information to authenticate the email. For SPF and SenderID, the query returns the IP addresses which are authorized to send mail on behalf of that domain. Typically there will be very few authorized SMTP sending addresses, even from a domain with millions of dynamically assignable IP addresses. For DKIM, the query returns the public key for the domain, which then validates the signature in an email header. A successful validation proves the email originated from the same people responsible for the DNS servers for that domain (as only the domain owners would have proper access to the private key that matches the public key published in DNS), and neither the headers nor the body of the email were altered on its way from the sender.
A spammer has no access to any of the connections between these DNS servers. Even if he were to falsify records in the DNS server for his own domain, he would not be able to forge records for someone else's domain name. When a spammer tries to send an email claiming to be from amazon.com, for example, the receiver queries the .com DNS server, then a server in a secure building at Amazon. The IP address on the message from the spammer won't match any of Amazon's authorized IP addresses, and the email can be rejected. Alternatively, DKIM will show the signature in the email is invalid because it is highly unlikely the spammer has access to the amazon.com private key.
Use of the DNS database to register authentication information for a domain is relatively new. The new information is added to existing DNS records, and queries for this information are handled the same way as any other DNS query. Publishing authentication records in DNS is voluntary, and many domains probably won't bother. However, any legitimate domain, even those that don't intend to operate public mail servers, will most likely want to block others from using their name to forge emails. A simple code in their DNS record will tell the world, "Block all mail claiming to be from our domain. We have no public mail servers."
SPF and SenderID have in essence the same limitation, they don't work directly behind the "border" ( MX
) of the receiver. For SPF forwarders to third parties could rewrite the Return-Path (MAIL FROM) in a similar way like mailing lists. This approach emulates the SMTP behaviour before RFC 1123 deprecated source routes; for a technical explanation see SRS
.
For SenderID, forwarders to third parties and mailing lists are asked to add a Sender: or Resent-Sender: header. For many mailing lists, the former is already the case, but other forwarders avoid any modifications of the mail in addition to the mandatory Received-timestamp line.
Use of a forwarder prevents the receiver from directly seeing the sender's IP address. The incoming IP packets have only the forwarder's IP address. Two solutions are possible if one can trust all forwarders. Either one trusts the forwarder to authenticate the sender, or one trusts the forwarder to at least accurately record the incoming IP address and pass it on, so one can do their own authentication.
The situation gets complicated when there is more than one forwarder. A sender can explicitly authorize a forwarder to send on its behalf, in effect extending its boundary to the public Internet. A receiver can trust a forwarder that it pays to handle email, in effect designating a new receiver. There may be additional "MTA relays" in the middle, however. These are sometimes used for administrative control, traffic aggregation, and routing control. All it takes is one broken link in the chain-of-trust from sender to receiver, and it is no longer possible to authenticate the sender.
Forwarders have one other responsibility, and that is to route Bounce message
s (a.k.a. DSNs) in case the forwarding fails (or if it is requested anyway). Email forwarding
is different from remailing when it comes to which address should receive DSNs. Spam bounces should not be sent to any address that may be forged. These bounces may go back by the same path they came, if that path has been authenticated.
Email
Electronic mail, commonly known as email or e-mail, is a method of exchanging digital messages from an author to one or more recipients. Modern email operates across the Internet or other computer networks. Some early email systems required that the author and the recipient both be online at the...
transport system with enough verifiable information, so that recipients can recognize the nature of each incoming message automatically. It differs from content filtering
Content filtering
Content filtering is the technique whereby content is blocked or allowed based on analysis of its content, rather than its source or other criteria. It is most widely used on the internet to filter email and web access.- Content filtering of email :...
.
Rationale
Ensuring a valid identity on an email has become a vital step in stopping spamSpam (electronic)
Spam is the use of electronic messaging systems to send unsolicited bulk messages indiscriminately...
(as email can be filtered based on such an identity), forgery, fraud, and even more serious crimes. The Simple Mail Transfer Protocol
Simple Mail Transfer Protocol
Simple Mail Transfer Protocol is an Internet standard for electronic mail transmission across Internet Protocol networks. SMTP was first defined by RFC 821 , and last updated by RFC 5321 which includes the extended SMTP additions, and is the protocol in widespread use today...
(SMTP) is continuously evolving, but when it was designed, in the early 1980s, it was the purview of academia and government agencies, and as such, there was no cause to consider security. It provided for no formal verification of sender.
Signing emails is a good first step towards identifying the origin of the message, but it does not establish whether that identity has a good reputation or whether it should be trusted.
This article explains how email identities are forged and the steps that are being taken now to prevent it.
Sender's IP verification
An email features four key players: the authors or originators of the email, the sender or agent who first puts the email on the public Internet, the receiver or agent who receives the email from the Internet, and the recipients who are the persons intended to read the email. For the sake of this discussion, the flow may be simplified as
Thanks to the Transmission Control Protocol
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
and to IP address
IP address
An Internet Protocol address is a numerical label assigned to each device participating in a computer network that uses the Internet Protocol for communication. An IP address serves two principal functions: host or network interface identification and location addressing...
registries
Regional Internet registry
A regional Internet registry is an organization that manages the allocation and registration of Internet number resources within a particular region of the world...
the senders IP address is automatically verified by the receiver. However, there is no provision for the verification of the author and sender information that is eventually saved in the relevant headers. Thus, it is quite easy for a spammer to make an exact copy of an email from example.com, including a long complicated sequence of headers and a genuine logo in the body of an email, then change the content to send readers to a website that appears to be genuine, but is actually a phishing
Phishing
Phishing is a way of attempting to acquire information such as usernames, passwords, and credit card details by masquerading as a trustworthy entity in an electronic communication. Communications purporting to be from popular social web sites, auction sites, online payment processors or IT...
scam designed to capture names, passwords, and credit card numbers.
IP addresses within the headers thus cannot be used to identify the source of a forged email. In addition to forged headers being used to throw off identification attempts, it is not uncommon for the legitimate headers within the email to reference a system insulated from the forger; the sender may, for example, be a zombie
Zombie computer
In computer science, a zombie is a computer connected to the Internet that has been compromised by a cracker, computer virus or trojan horse and can be used to perform malicious tasks of one sort or another under remote direction. Botnets of zombie computers are often used to spread e-mail spam...
that has been compromised by a Trojan horse
Trojan horse (computing)
A Trojan horse, or Trojan, is software that appears to perform a desirable function for the user prior to run or install, but steals information or harms the system. The term is derived from the Trojan Horse story in Greek mythology.-Malware:A destructive program that masquerades as a benign...
. Service providers can block this type of email using IP address reputation technology. IP address reputation is a technology that monitors emails based on their source IP address. The technology tracks the email for spam activity and blocks messages from that sender once spam activity begins. Service providers use this technology to prevent zombie
Zombie computer
In computer science, a zombie is a computer connected to the Internet that has been compromised by a cracker, computer virus or trojan horse and can be used to perform malicious tasks of one sort or another under remote direction. Botnets of zombie computers are often used to spread e-mail spam...
s from sending spam. Amir Lev, from email security company Commtouch
Commtouch
Commtouch is an Internet security technology company founded in 1991. The company is headquartered in Netanya, Israel, with a subsidiary in Sunnyvale, California....
, described the technology as, "IP
Blacklisting
Attempts to stop spam by blacklisting sender's IP addresses still allows a small percentage of those emails through. Most IP addresses are dynamic, i.e. they are frequently changing. An ISP, or any organization directly connected to the Internet, gets a block of real Internet addresses when they register in the DNSDomain name system
The Domain Name System is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities...
. Within that block, they assign individual addresses to customers as needed. A dial-up customer may get a new IP address each time they connect. By the time that address appears on blacklists all over the world, the spammer will have new addresses for the next run. There are 4 billion possible IPv4
IPv4
Internet Protocol version 4 is the fourth revision in the development of the Internet Protocol and the first version of the protocol to be widely deployed. Together with IPv6, it is at the core of standards-based internetworking methods of the Internet...
addresses on the Internet. The game of keeping up with these rapidly changing IP addresses has been facetiously called "whack-a-mole".
So called policy lists are blacklists that contain IP addresses on a preventive basis. An IP address can be listed therein even if no spam has ever been sent from it, because it has been variously classified as a dial-up address, end-user address, or residential address, with no formal definition of such classification schemes. Not requiring evidence of spam for each listed address, these lists can collect a greater number of addresses and thus block more spam. However, the policies devised are not authoritative, since they have not been issued by the legitimate user of an IP address, and the resulting lists are therefore not universally accepted.
Controlling users
There are a number of things that ISPs have done to stop zombies and deliberate spamming by their customers:- portTCP and UDP portIn computer networking, a port is an application-specific or process-specific software construct serving as a communications endpoint in a computer's host operating system. A port is associated with an IP address of the host, as well as the type of protocol used for communication...
25 can be blocked by access providers in favor of the Mail submission agentMail submission agentA mail submission agent is a computer program or software agent that receives electronic mail messages from a mail user agent and cooperates with a mail transfer agent for delivery of the mail...
's port 587 that should always require authentication, - the number of existing Received headers in relayed mail can be limited. A small sample of reports from SpamCopSpamCopSpamCop is a free spam reporting service, allowing recipients of unsolicited bulk email and unsolicited commercial email to report offenders to the senders' Internet Service Providers , and sometimes their web hosts...
seems to validate this. - infected computers can be cleared of viruses and patched to resist further infection,
- outgoing email can be monitored for any sudden increase in flow or in content that is typical of spam.
Some ISPs have been quite successful, but others don't care to make the effort. With spam now over 80% of all email traffic, we can expect that there will always be ISPs who are not willing to take the necessary steps. The measures mentioned above don't directly help the entity who operates them to reduce incoming spam. By reducing outgoing spam, they help generic Internet users.
Authenticating senders
Email "authentication" greatly simplifies and automates the process of identifying senders. After identifying and verifying that a claimed domain name has been authenticated or has authorized the sending Mail transfer agentMail transfer agent
Within Internet message handling services , a message transfer agent or mail transfer agent or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture...
(MTA), it is then possible to treat suspected forgeries with suspicion, reject known forgeries, and block email from known spamming domains. It is also possible to "whitelist" email from known reputable domains, and bypass content-based filtering, which always loses some valid emails in the flood of spam. The fourth category, email from unknown domains, can be treated like we now treat all email – increasingly rigorous filtering, return challenges to the sender, etc.
Success of a domain-rating system should encourage reputable ISPs to stop their outgoing spam and get a good rating. It has been in the ISP's interest to shift to blocking by the domain, rather than by the IP address that they directly control.
A number of methods claim to authenticate a sender's domain name ( SPF
Sender Policy Framework
Sender Policy Framework is an email validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP addresses. SPF allows administrators to specify which hosts are allowed to send mail from a given domain by creating a specific SPF...
, SenderID, DKIM ). However SPF and SenderID only authorize the IP address of the outbound MTA. It is uncertain whether the PRA (Purported Responsible Address) associated with the SPF record was intended, and thereby protected when the outbound MTA is shared by other domains which is why sensitive domains must take additional precautions. While the outbound MTA, not being authorized when there is a strict SPF record, can block much of the forgery now prevalent, it is never safe to expect that the MTA, being authorized, represents a form of authentication. None exclude the use of other methods, although SPF / DKIM, SPF / SenderID, and SenderID / DKIM pairwise appear to be competing for the same niches. The most widely used will likely be the ones that require the least effort on the part of ISPs and others currently operating public mail servers.
SPF and SenderID use the domain to authorize the outbound MTA. DKIM uses a digital signature
Digital signature
A digital signature or digital signature scheme is a mathematical scheme for demonstrating the authenticity of a digital message or document. A valid digital signature gives a recipient reason to believe that the message was created by a known sender, and that it was not altered in transit...
to authenticate a domain name and the entire content of a message. SPF can reject a forgery before any data transfer. SenderID and DKIM must see at least the headers, so the entire message must be transmitted. SPF has a problem with forwarders (that Sender Rewriting Scheme
Sender Rewriting Scheme
Sender Rewriting Scheme is a technique to re-mail an email message so that eventual Delivery Status Notifications can reach the original message sender...
defines a fix for), SenderID also with mailing lists (see below).
SPF and SenderID work by having the domain publish what might be a sizeable list of IP addresses that identify MTAs they authorize to carry their email. This works as a type of path registration, but does not scale well since it also combines both IPv4 and IPv6 addresses. Every incoming email has an IP address that is difficult to forge, a bunch of domain names in the email headers, and a few more in the commands from the sender's SMTP server. The methods differ in which of these names to use as the sender's domain name. All of them can be faked, but what cannot be faked is a domain name held by a DNS server for that section of the Internet.
The simplest and by far most widely deployed authentication scheme begins with a reverse DNS lookup of the connecting IP address. If there is no answer, it's a safe bet that the address is not a legitimate sender. If there is an answer, a forward DNS lookup of that answer authenticates the sender if it returns the connecting IP address. In other words, we look up the name of the connecting IP address, and look up the IP address of that name, and they must match.
The procedure to authenticate is basically simple. When a request to deliver an email arrives, the claimed sender's domain name is sent in a query to a high-level DNS server. That DNS server in turn, refers to lower level servers until an answer is found that is authoritative for the domain in question. The answer returned to the receiver includes the information to authenticate the email. For SPF and SenderID, the query returns the IP addresses which are authorized to send mail on behalf of that domain. Typically there will be very few authorized SMTP sending addresses, even from a domain with millions of dynamically assignable IP addresses. For DKIM, the query returns the public key for the domain, which then validates the signature in an email header. A successful validation proves the email originated from the same people responsible for the DNS servers for that domain (as only the domain owners would have proper access to the private key that matches the public key published in DNS), and neither the headers nor the body of the email were altered on its way from the sender.
A spammer has no access to any of the connections between these DNS servers. Even if he were to falsify records in the DNS server for his own domain, he would not be able to forge records for someone else's domain name. When a spammer tries to send an email claiming to be from amazon.com, for example, the receiver queries the .com DNS server, then a server in a secure building at Amazon. The IP address on the message from the spammer won't match any of Amazon's authorized IP addresses, and the email can be rejected. Alternatively, DKIM will show the signature in the email is invalid because it is highly unlikely the spammer has access to the amazon.com private key.
Use of the DNS database to register authentication information for a domain is relatively new. The new information is added to existing DNS records, and queries for this information are handled the same way as any other DNS query. Publishing authentication records in DNS is voluntary, and many domains probably won't bother. However, any legitimate domain, even those that don't intend to operate public mail servers, will most likely want to block others from using their name to forge emails. A simple code in their DNS record will tell the world, "Block all mail claiming to be from our domain. We have no public mail servers."
Difficulties with email forwarding
There are some additional details when an email forwarder is involved. Forwarders perform a useful service in allowing you to have one simple permanent address, even if you change jobs or ISPs. List servers perform a similar function, forwarding email to many receivers on behalf of one sender. Forwarders pose no problem for an end-to-end authentication method like DKIM, as long as the signed message is not modified (some lists do this).SPF and SenderID have in essence the same limitation, they don't work directly behind the "border" ( MX
MX record
A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available...
) of the receiver. For SPF forwarders to third parties could rewrite the Return-Path (MAIL FROM) in a similar way like mailing lists. This approach emulates the SMTP behaviour before RFC 1123 deprecated source routes; for a technical explanation see SRS
Sender Rewriting Scheme
Sender Rewriting Scheme is a technique to re-mail an email message so that eventual Delivery Status Notifications can reach the original message sender...
.
For SenderID, forwarders to third parties and mailing lists are asked to add a Sender: or Resent-Sender: header. For many mailing lists, the former is already the case, but other forwarders avoid any modifications of the mail in addition to the mandatory Received-timestamp line.
Use of a forwarder prevents the receiver from directly seeing the sender's IP address. The incoming IP packets have only the forwarder's IP address. Two solutions are possible if one can trust all forwarders. Either one trusts the forwarder to authenticate the sender, or one trusts the forwarder to at least accurately record the incoming IP address and pass it on, so one can do their own authentication.
The situation gets complicated when there is more than one forwarder. A sender can explicitly authorize a forwarder to send on its behalf, in effect extending its boundary to the public Internet. A receiver can trust a forwarder that it pays to handle email, in effect designating a new receiver. There may be additional "MTA relays" in the middle, however. These are sometimes used for administrative control, traffic aggregation, and routing control. All it takes is one broken link in the chain-of-trust from sender to receiver, and it is no longer possible to authenticate the sender.
Forwarders have one other responsibility, and that is to route Bounce message
Bounce message
In the Internet's standard e-mail protocol SMTP, a bounce message, also called a Non-Delivery Report/Receipt , a Delivery Status Notification message, a Non-Delivery Notification or simply a bounce, is an automated electronic mail message from a mail system informing the sender of another...
s (a.k.a. DSNs) in case the forwarding fails (or if it is requested anyway). Email forwarding
Email forwarding
Email forwarding generically refers to the operation of re-sending an email message delivered to one email address on to a possibly different email address...
is different from remailing when it comes to which address should receive DSNs. Spam bounces should not be sent to any address that may be forged. These bounces may go back by the same path they came, if that path has been authenticated.
Criticism
Some experts take the position that the ISP is the major drawback to proper elimination of spam. The premise is that the regulating agencies have no authority, and the ISP's have no incentive.- “Authentication cannot stop spam, unless the cop/Reputation Service/Certificate Authority in charge revokes certificates for spamming. If that could happen, then ISPs would also be willing and even enthusiastic about terminating accounts or otherwise controlling (e.g. port block) their spammers. If ISPs would do that, then there would be no spam to need authentication to stop spam and so need for a CA playing cop. As long as ISPs remain unwilling to police their own spamming customers, they would never deal with a CA willing to play cop.
- Authentication involving TLS, SMTP-AUTH, or S/MIME cannot stop backscatter for the same reasons SPF, DKIM, and the rest were, are, and always will be powerless against it. Some of those reasons are why Yahoo still does not sign DKIM on all outgoing mail, Hotmail still publishes whishywashy SPF RRs and neither requires their snakeoil forgery solution on incoming mail.”
-
- --Vernon Schryver (Distributed Checksum ClearinghouseDistributed Checksum ClearinghouseDistributed Checksum Clearinghouse is a hash sharing method of spam email detection.The basic logic in DCC is that most spam mails are sent to many recipients. The same message body appearing many times is therefore bulk email. DCC identifies bulk email by taking a checksum and sending that...
operator)
- --Vernon Schryver (Distributed Checksum Clearinghouse
-