Sender Policy Framework
Encyclopedia
Sender Policy Framework is an email
validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP address
es. SPF allows administrators to specify which hosts
are allowed to send mail from a given domain by creating a specific SPF record (or TXT record) in the Domain Name System
(DNS). Mail exchangers use the DNS to check that mail from a given domain is being sent by a host sanctioned by that domain's administrators.
Sender Policy Framework is defined in IETF publication RFC 4408.
permits any computer to send email claiming to be from any source address. This is exploited by spammers
who often use forged email addresses, making it more difficult to trace a message back to its sender, and easy for spammers to hide their identity in order to avoid responsibility. Many believe that the ability for anyone to forge sender addresses is a security flaw in modern SMTP.
SPF allows the owner of an Internet domain to specify which computers are authorized to send mail with sender addresses in that domain, using special Domain Name System
(DNS) records (SPF, type 99). Receivers verifying the SPF records may reject messages from unauthorized sources before receiving the body of the message. Thus, the principles of operation are similar to those of DNS-based blackhole lists (DNSBL
), except that SPF uses the authority delegation scheme of the Domain Name System. Early implementations used TXT records for implementation before the new record type was commonly available in DNS software. Use of TXT records for SPF is intended as a transitional mechanism. However, according to the current RFC, RFC 4408, section 3.1.1, "An SPF-compliant domain name SHOULD have SPF records of both RR types. A compliant domain name MUST have a record of at least one type," and as such, TXT record use is not deprecated.
The sender address is transmitted at the beginning of the SMTP dialog. If the server
rejects the sender, the unauthorized client
should receive a rejection message, and if that client was a relaying message transfer agent (MTA), a bounce message
to the original sending address may be generated. If the server accepts the sender, and subsequently also accepts the recipients and the body of the message, it should insert a Return-Path field in the message header in order to save the sender address. While the address in the Return-Path often matches other originator addresses in the mail header such as From or Sender, this is not necessarily the case, and SPF does not prevent forgery of these other addresses.
Spammers can send email with an SPF PASS result if they have an account in a domain with a sender policy, or abuse a compromised system in this domain. However, doing so makes the spammer easier to trace.
The main benefit of SPF is to the owners of e-mail addresses that are forged in the Return-Path. They receive large amounts of unsolicited error messages and other auto-replies. If such receivers use SPF to specify their legitimate source IP addresses and indicate FAIL result for all other addresses, receivers checking SPF can reject forgeries, thus reducing or eliminating the amount of backscatter
.
SPF has potential advantages beyond helping identify unwanted mail. In particular, if a sender provides SPF information, then receivers can use SPF PASS results in combination with a white list to identify known reliable senders. Scenarios like compromised systems and shared sending mailers limit this use.
This is a necessary and obvious feature of SPF – checks behind the "border" MTA
(MX
) of the receiver cannot work directly.
Publishers of SPF FAIL policies must accept this potential problem. They should test (e.g., with a SOFTFAIL policy) until they are satisfied with the results. See below for a list of alternatives to plain message forwarding.
and other auto-replies, an SPF check of the HELO-identity is mandatory.
With a bogus HELO identity the result NONE would not help, but for valid host names SPF also protects the HELO identity. This SPF feature was always supported as an option for receivers, and later SPF drafts including the final specification recommend to check the HELO always.
This allows to white list sending mailers based on a HELO PASS, or to reject all mails after a HELO FAIL. It can also be used in reputation system
s (any white or black list is a simple case of a reputation system).
Publish a policy : Domains and hosts identify the machines authorized to send e-mail on their behalf. They do this by adding additional records to their existing DNS information: every domain name
or host that has an A record or MX record
deserves an SPF record specifying the policy if it is used either in an email address or as HELO/EHLO argument. Hosts which do not send mail should have an SPF record published which indicate such ("v=spf1 -all"). It is highly recommended to validate the SPF record using record testing tools such as those provided on the SPF Project webpage.
Check and use SPF information : Receivers use ordinary DNS queries, which are typically cached to enhance performance. Receivers then interpret the SPF information as specified and act upon the result.
Revise mail forwarding
Thus, the key issue in SPF is the specification for the new DNS information that domains set and receivers use. The records laid out below are in typical DNS syntax. Note that RFC 4408 recommends that both an SPF and TXT record be used, although just one is allowed:
example.com. IN TXT "v=spf1 a mx -all"
example.com. IN SPF "v=spf1 a mx -all"
"v=" defines the version of SPF used. The following words provide mechanisms to use to determine if a domain is eligible to send mail. The "a" and "mx" specify the systems permitted to send messages for the given domain. The "-all" at the end specifies that, if the previous mechanisms did not match, the message should be rejected.
Another safety guard is the maximum of ten mechanisms querying DNS, i.e. any mechanism except from IP4, IP6, and ALL. Implementations can abort the evaluation with result SOFTERROR when it takes too long or a DNS query times out, but they must return PERMERROR if the policy directly or indirectly needs more than ten queries for mechanisms. Any redirect= also counts towards this processing limit.
A typical SPF HELO policy v=spf1 a -all may execute up to three DNS queries: (1) SPF, (2) TXT (deprecated, but for backwards compatibility during the transition), and (3) A or AAAA. This last query counts as the first mechanism towards the limit (10). In this example it is also the last, because ALL needs no DNS lookup.
a user that wishes to send an email from a private PC or a mobile phone: the
user uses his corporate email address but may use a different outgoing SMTP
server not passed by the SPF record. The corporate domain may have won by
blocking all email that does not originate from themselves, but have thereby
limited their own users. Many organizations consider this compromise
acceptable and even desirable.
SPF PASS is useful for authenticating the domain for use as a parameter
to a spam classification engine. That is, the domain in the sender address
can be considered to be authentic if the originating IP yields an SPF PASS.
The domain can then be referenced against a reputation database.
SPF results other than PASS (used in combination with a reputation system)
and FAIL cannot be meaningfully mapped to PASS and FAIL. However, a reputation
system can easily track independent reputations for each SPF result, i.e. example.com:PASS and example.com:NEUTRAL would have different reputations, and ditto for the other results. This approach is useful even without whitelisting plain forwarders, since
the FAIL results from the plain forwarders simply accrue an independent reputation.
The meaning of PASS, SOFTFAIL, FAIL is sometimes incorrectly interpreted
to mean "not-spam", "maybe-spam", "spam" respectively. However SPF does
nothing of the sort. SPF merely offers an organization firstly the
means to classify emails based on their domain name instead of their
IP address (SPF PASS); and secondly, the means to block unauthorized use
of their domain (SPF FAIL).
behalf of another user since only the domain part of the address is
used to locate the SPF policy record. In more sophisticated implementations, the domain owner can specify separate policies for each user by mean of SPF "macros" that reference the "localpart" (user) as defined in RFC 4408, or simply require all mail submissions for the domain to use SMTP AUTH (RFC 4954). The latter is highly recommended anyway for many reasons.
Other downstream hosts, for instance in a forwarding scenario, can only perform SPF checks based on "Received" headers. This is cumbersome and error-prone. A better approach is for the MX host to check SPF without blocking any email, and then add a "Received-SPF" header field as specified in RFC 4408. Downstream hosts can then look at the Received-SPF header and set their own policy of whether to reject, accept, or quarantine based on the SPF result and other factors.
), not the message contents (header and body) – this is the distinction between SMTP (as specified in STD
10 or RFC 5321) and Internet Message Format (as specified in STD 11 or RFC 5322). It is orthogonal and complementary to DomainKeys Identified Mail
(DKIM), which signs the contents (including headers).
In brief, SPF validates MAIL FROM vs. its source server; DKIM validates "From:" by cryptographic means.
RFC 4406, is a parallel solution to the problem of message validation, and defines a pair of closely related tests. One validates a message's Purported Responsible Address (PRA) as defined in RFC 4407. The other validates a message's Reverse-Path (also known as MAIL-FROM address) as defined in RFC 4408.
Quoting from RFC4407:
"Note that the Sender ID experiment may use DNS records that may have been created for the current SPF experiment or earlier versions in this set of experiments. Depending on the content of the record, this may mean that sender-ID heuristics would be applied incorrectly to a message. Depending on the actions associated by the recipient with those heuristics, the message may not be delivered or may be discarded on receipt."
Those publishing SPF DNS records should consider the advice given in section 3.4 of RFC 4406 and may wish to publish both v=spf1 and spf2.0 records to avoid the conflict.
Caveats:
seminar-for-you.ru. 14400 IN TXT "v=spf1 a mx ip4:55.11.65.20/2 ip4:90.2.123.112/2 ip4:176.33.87.19/2 ip4:212.63.89.33/2 -all"
worldwidemail.ru. 13733 IN TXT "v=spf1 a mx ip4:55.11.65.20/2 ip4:90.2.123.112/2 ip4:176.33.87.19/2 ip4:212.63.89.33/2 -all"
example.com. 21600 IN SPF "v=spf1 +all"
This last record says that any host on the Internet may send mail on behalf of the domain/hostname example.com. Although syntactically valid, "+all" is indicative of an administrator who does not care about SPF or the mail forgeries it detects.
For stable domains, this simply means that any reputation attached to the domain is the same with or without SPF and such spam domains are easily learned and rejected. The real value of wide-mask SPF policies to spammers is with "throw-away" domains that are registered, used to send spam from botnets for a few days, and then abandoned.
posted his own SPF-like specification on the same list. These posts ignited a lot of interest, and eventually led to the forming of the IETF Anti-Spam Research Group (ASRG) and their mailing list, where the SPF idea was debated among a subscriber base that seemed to grow exponentially day by day. Among the proposals submitted to the ASRG were "Reverse MX
" by Hadmut Danisch, and "Designated Mailer Protocol" by Gordon Fecyk.
In June 2003, Meng Weng Wong
merged the RMX and DMP specifications and solicited suggestions from other programmers. Over the next six months, a large number of changes were made and a large community had started working on SPF.
Originally SPF stood for Sender Permitted From and was sometimes also called SMTP+SPF, but it was changed to Sender Policy Framework in February 2004.
In early 2004, the IETF created the MARID
working group and tried to use SPF and Microsoft's CallerID proposal as the basis for what is now known as Sender ID
.
After the collapse of MARID the SPF community returned to the original "classic" version of SPF. In July 2005 this version of the specification was approved by the IESG
as an IETF experiment, inviting the community to observe SPF during the two years following publication. On April 28, 2006, the SPF RFC was published as experimental RFC 4408.
wrote an e-mail that discusses his concerns with SPF. Some of these include:
There are other concerns about the impact of widespread use of SPF, notably the impact on various legitimate forms of email spoofing, such as forwarding services, SMTP use by people with multiple identities, etc. (For example, a person who uses their home ISP's SMTP servers to send mail with their work email as the address.) On the other hand, many of these uses may be "expected" yet not "legitimate". To a certain extent this is more a question of ownership and expectations than a technical question.
version 3.0.0 and ASSP
implement SPF. Many mail transfer agent
s (MTAs) support SPF directly such as Courier
, CommuniGate Pro, Wildcat
, MDaemon, and Microsoft Exchange
, or have patches/plug-ins available that support SPF, including Postfix
, Sendmail
, Exim
, and qmail
. More than one million domains publish SPF FAIL -all policies.
In a survey published in 2007, 5% of the .com and .net domains had some kind of SPF policy. In 2009, a continuous survey run at Nokia Research reports that 51% of the tested domains specify an SPF policy. These results can include trivial policies like v=spf1 ?all. In April 2007, BITS, a division of the Financial Services Roundtable, published e-mail security recommendations for its members including SPF deployment.
In 2008, the Messaging Anti-Abuse Working Group (MAAWG
) published a paper about email-authentication covering SPF, Sender ID
, and DKIM
. In their "Sender Best Communication Practices" the MAAWG stated: "At the very least, senders should incorporate SPF records for their mailing domains".
In August 2005 it was learned that EarthLink
would refuse to allow hosted domains the ability to enter SPF records.
Email
Electronic mail, commonly known as email or e-mail, is a method of exchanging digital messages from an author to one or more recipients. Modern email operates across the Internet or other computer networks. Some early email systems required that the author and the recipient both be online at the...
validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP address
IP address
An Internet Protocol address is a numerical label assigned to each device participating in a computer network that uses the Internet Protocol for communication. An IP address serves two principal functions: host or network interface identification and location addressing...
es. SPF allows administrators to specify which hosts
Host (network)
A network host is a computer connected to a computer network. A network host may offer information resources, services, and applications to users or other nodes on the network. A network host is a network node that is assigned a network layer host address....
are allowed to send mail from a given domain by creating a specific SPF record (or TXT record) in the Domain Name System
Domain name system
The Domain Name System is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities...
(DNS). Mail exchangers use the DNS to check that mail from a given domain is being sent by a host sanctioned by that domain's administrators.
Sender Policy Framework is defined in IETF publication RFC 4408.
Principles of operation
The Simple Mail Transfer ProtocolSimple Mail Transfer Protocol
Simple Mail Transfer Protocol is an Internet standard for electronic mail transmission across Internet Protocol networks. SMTP was first defined by RFC 821 , and last updated by RFC 5321 which includes the extended SMTP additions, and is the protocol in widespread use today...
permits any computer to send email claiming to be from any source address. This is exploited by spammers
E-mail spam
Email spam, also known as junk email or unsolicited bulk email , is a subset of spam that involves nearly identical messages sent to numerous recipients by email. Definitions of spam usually include the aspects that email is unsolicited and sent in bulk. One subset of UBE is UCE...
who often use forged email addresses, making it more difficult to trace a message back to its sender, and easy for spammers to hide their identity in order to avoid responsibility. Many believe that the ability for anyone to forge sender addresses is a security flaw in modern SMTP.
SPF allows the owner of an Internet domain to specify which computers are authorized to send mail with sender addresses in that domain, using special Domain Name System
Domain name system
The Domain Name System is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities...
(DNS) records (SPF, type 99). Receivers verifying the SPF records may reject messages from unauthorized sources before receiving the body of the message. Thus, the principles of operation are similar to those of DNS-based blackhole lists (DNSBL
DNSBL
A DNSBL is a list of IP addresses published through the Internet Domain Name Service either as a zone file that can be used by DNS server software, or as a live DNS zone that can be queried in real-time...
), except that SPF uses the authority delegation scheme of the Domain Name System. Early implementations used TXT records for implementation before the new record type was commonly available in DNS software. Use of TXT records for SPF is intended as a transitional mechanism. However, according to the current RFC, RFC 4408, section 3.1.1, "An SPF-compliant domain name SHOULD have SPF records of both RR types. A compliant domain name MUST have a record of at least one type," and as such, TXT record use is not deprecated.
The sender address is transmitted at the beginning of the SMTP dialog. If the server
Server (computing)
In the context of client-server architecture, a server is a computer program running to serve the requests of other programs, the "clients". Thus, the "server" performs some computational task on behalf of "clients"...
rejects the sender, the unauthorized client
Client (computing)
A client is an application or system that accesses a service made available by a server. The server is often on another computer system, in which case the client accesses the service by way of a network....
should receive a rejection message, and if that client was a relaying message transfer agent (MTA), a bounce message
Bounce message
In the Internet's standard e-mail protocol SMTP, a bounce message, also called a Non-Delivery Report/Receipt , a Delivery Status Notification message, a Non-Delivery Notification or simply a bounce, is an automated electronic mail message from a mail system informing the sender of another...
to the original sending address may be generated. If the server accepts the sender, and subsequently also accepts the recipients and the body of the message, it should insert a Return-Path field in the message header in order to save the sender address. While the address in the Return-Path often matches other originator addresses in the mail header such as From or Sender, this is not necessarily the case, and SPF does not prevent forgery of these other addresses.
Spammers can send email with an SPF PASS result if they have an account in a domain with a sender policy, or abuse a compromised system in this domain. However, doing so makes the spammer easier to trace.
The main benefit of SPF is to the owners of e-mail addresses that are forged in the Return-Path. They receive large amounts of unsolicited error messages and other auto-replies. If such receivers use SPF to specify their legitimate source IP addresses and indicate FAIL result for all other addresses, receivers checking SPF can reject forgeries, thus reducing or eliminating the amount of backscatter
Backscatter (e-mail)
Backscatter is incorrect automated bounce messages sent by mail servers, typically as a side effect of incoming spam....
.
SPF has potential advantages beyond helping identify unwanted mail. In particular, if a sender provides SPF information, then receivers can use SPF PASS results in combination with a white list to identify known reliable senders. Scenarios like compromised systems and shared sending mailers limit this use.
Reasons to implement
If a domain publishes an SPF record, spammers and phishers are less likely to forge e-mails pretending to be from that domain, since the forged e-mails are more likely to be caught in spam filters which check the SPF record. Therefore, an SPF-protected domain is less attractive to spammers and phishers. Since an SPF-protected domain is less attractive as a spoofed address, it is less likely to be blacklisted by spam filters and so ultimately the legitimate e-mail from the domain is more likely to get through.FAIL and forwarding
SPF does not allow plain message forwarding. When a domain publishes an SPF FAIL policy, then legitimate messages sent to receivers forwarding their mail to third parties can be rejected and bounced if all of the following occur:- The forwarder does not rewrite the Return-Path, unlike mailing lists.
- The next hop does not white list the forwarder.
- This hop checks SPF.
This is a necessary and obvious feature of SPF – checks behind the "border" MTA
Mail transfer agent
Within Internet message handling services , a message transfer agent or mail transfer agent or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture...
(MX
MX record
A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available...
) of the receiver cannot work directly.
Publishers of SPF FAIL policies must accept this potential problem. They should test (e.g., with a SOFTFAIL policy) until they are satisfied with the results. See below for a list of alternatives to plain message forwarding.
HELO tests
For an empty Return-Path as used in error messagesBounce message
In the Internet's standard e-mail protocol SMTP, a bounce message, also called a Non-Delivery Report/Receipt , a Delivery Status Notification message, a Non-Delivery Notification or simply a bounce, is an automated electronic mail message from a mail system informing the sender of another...
and other auto-replies, an SPF check of the HELO-identity is mandatory.
With a bogus HELO identity the result NONE would not help, but for valid host names SPF also protects the HELO identity. This SPF feature was always supported as an option for receivers, and later SPF drafts including the final specification recommend to check the HELO always.
This allows to white list sending mailers based on a HELO PASS, or to reject all mails after a HELO FAIL. It can also be used in reputation system
Reputation system
A reputation system computes and publishes reputation scores for a set of objects within a community or domain, based on a collection of opinions that other entities hold about the objects...
s (any white or black list is a simple case of a reputation system).
Implementation
Compliance with SPF consists of three loosely related tasks:Publish a policy : Domains and hosts identify the machines authorized to send e-mail on their behalf. They do this by adding additional records to their existing DNS information: every domain name
Domain name
A domain name is an identification string that defines a realm of administrative autonomy, authority, or control in the Internet. Domain names are formed by the rules and procedures of the Domain Name System ....
or host that has an A record or MX record
MX record
A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available...
deserves an SPF record specifying the policy if it is used either in an email address or as HELO/EHLO argument. Hosts which do not send mail should have an SPF record published which indicate such ("v=spf1 -all"). It is highly recommended to validate the SPF record using record testing tools such as those provided on the SPF Project webpage.
Check and use SPF information : Receivers use ordinary DNS queries, which are typically cached to enhance performance. Receivers then interpret the SPF information as specified and act upon the result.
Revise mail forwarding
- Plain mail forwarding is not allowed by SPF. The alternatives are
- remailing, i.e. replacing the original sender with one belonging to the local domain,
- refusing, i.e. answering
551 User not local; please try, - whitelisting on the target server, so that it will not refuse a forwarded message, and
- Sender Rewriting SchemeSender Rewriting SchemeSender Rewriting Scheme is a technique to re-mail an email message so that eventual Delivery Status Notifications can reach the original message sender...
, a more complicated mechanism that handles routing non-delivery notifications to the original sender.
Thus, the key issue in SPF is the specification for the new DNS information that domains set and receivers use. The records laid out below are in typical DNS syntax. Note that RFC 4408 recommends that both an SPF and TXT record be used, although just one is allowed:
example.com. IN TXT "v=spf1 a mx -all"
example.com. IN SPF "v=spf1 a mx -all"
"v=" defines the version of SPF used. The following words provide mechanisms to use to determine if a domain is eligible to send mail. The "a" and "mx" specify the systems permitted to send messages for the given domain. The "-all" at the end specifies that, if the previous mechanisms did not match, the message should be rejected.
Mechanisms
Eight mechanisms are defined:| ALL | Matches always; used for a default result like -all for all IPs not matched by prior mechanisms. |
| A | If the domain name has an address record (A or AAAA) that can be resolved to the sender's address, it will match. |
| IP4 | If the sender is in a given IPv4 IPv4 Internet Protocol version 4 is the fourth revision in the development of the Internet Protocol and the first version of the protocol to be widely deployed. Together with IPv6, it is at the core of standards-based internetworking methods of the Internet... address range, match. |
| IP6 | If the sender is in a given IPv6 IPv6 Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4... address range, match. |
| MX | If the domain name has an MX record MX record A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available... resolving to the sender's address, it will match (i.e. the mail comes from one of the domain's mail servers). |
| PTR | If the domain name (PTR record) for the client's address is in the given domain and that domain name resolves to the client's address (forward-confirmed reverse DNS), match. |
| EXISTS | If the given domain name resolves to any address, match (no matter the address it resolves to). This is rarely used. Along with the SPF macro language it offers more complex matches like DNSBL DNSBL A DNSBL is a list of IP addresses published through the Internet Domain Name Service either as a zone file that can be used by DNS server software, or as a live DNS zone that can be queried in real-time... -queries. |
| INCLUDE | If the included (a misnomer) policy passes the test this mechanism matches. This is typically used to include policies of more than one ISP Internet service provider An Internet service provider is a company that provides access to the Internet. Access ISPs directly connect customers to the Internet using copper wires, wireless or fiber-optic connections. Hosting ISPs lease server space for smaller businesses and host other people servers... . |
Qualifiers
Each mechanism can be combined with one of four qualifiers:- + for a PASS result. This can be omitted; e.g., +mx is the same as mx.
- ? for a NEUTRAL result interpreted like NONE (no policy).
- ~ for SOFTFAIL, a debugging aid between NEUTRAL and FAIL. Typically, messages that return a SOFTFAIL are accepted but tagged.
- - for FAIL, the mail should be rejected (see below).
Modifiers
The modifiers allow for future extensions to the framework. To date only the two modifiers defined in the RFC 4408 have been widely deployed:- exp=some.example.com gives the name of a domain with a DNSDomain name systemThe Domain Name System is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities...
TXT record (interpreted using SPF's macro language) to get an explanation for FAIL results—typically a URLUniform Resource LocatorIn computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
which is added to the SMTP error code. This feature is rarely used. - redirect=some.example.com can be used instead of the ALL-mechanism to link to the policy record of another domain. This modifier is easier to understand than the somewhat similar INCLUDE-mechanism.
Error handling
As soon as SPF implementations detect syntax errors in a sender policy they must abort the evaluation with result PERMERROR. Skipping erroneous mechanisms cannot work as expected, therefore include:bad.example and redirect=bad.example also cause a PERMERROR.Another safety guard is the maximum of ten mechanisms querying DNS, i.e. any mechanism except from IP4, IP6, and ALL. Implementations can abort the evaluation with result SOFTERROR when it takes too long or a DNS query times out, but they must return PERMERROR if the policy directly or indirectly needs more than ten queries for mechanisms. Any redirect= also counts towards this processing limit.
A typical SPF HELO policy v=spf1 a -all may execute up to three DNS queries: (1) SPF, (2) TXT (deprecated, but for backwards compatibility during the transition), and (3) A or AAAA. This last query counts as the first mechanism towards the limit (10). In this example it is also the last, because ALL needs no DNS lookup.
Interpretation
SPF FAIL policies can be an effective but problematic tool. A typical example isa user that wishes to send an email from a private PC or a mobile phone: the
user uses his corporate email address but may use a different outgoing SMTP
server not passed by the SPF record. The corporate domain may have won by
blocking all email that does not originate from themselves, but have thereby
limited their own users. Many organizations consider this compromise
acceptable and even desirable.
SPF PASS is useful for authenticating the domain for use as a parameter
to a spam classification engine. That is, the domain in the sender address
can be considered to be authentic if the originating IP yields an SPF PASS.
The domain can then be referenced against a reputation database.
SPF results other than PASS (used in combination with a reputation system)
and FAIL cannot be meaningfully mapped to PASS and FAIL. However, a reputation
system can easily track independent reputations for each SPF result, i.e. example.com:PASS and example.com:NEUTRAL would have different reputations, and ditto for the other results. This approach is useful even without whitelisting plain forwarders, since
the FAIL results from the plain forwarders simply accrue an independent reputation.
The meaning of PASS, SOFTFAIL, FAIL is sometimes incorrectly interpreted
to mean "not-spam", "maybe-spam", "spam" respectively. However SPF does
nothing of the sort. SPF merely offers an organization firstly the
means to classify emails based on their domain name instead of their
IP address (SPF PASS); and secondly, the means to block unauthorized use
of their domain (SPF FAIL).
Intra-domain forgery
In a naive implementation, SPF does not prevent a user with the same domain sending an email onbehalf of another user since only the domain part of the address is
used to locate the SPF policy record. In more sophisticated implementations, the domain owner can specify separate policies for each user by mean of SPF "macros" that reference the "localpart" (user) as defined in RFC 4408, or simply require all mail submissions for the domain to use SMTP AUTH (RFC 4954). The latter is highly recommended anyway for many reasons.
Checkpoints
SPF needs to operate on the host indicated by the receiving domain's MX record. This means the host(s) that are the direct recipient of remote TCP connections; since such a host can easily deduce the originating IP address from the TCP session. These hosts are able to block the email directly in the envelope, avoiding bounce messages and their inherent problems.Other downstream hosts, for instance in a forwarding scenario, can only perform SPF checks based on "Received" headers. This is cumbersome and error-prone. A better approach is for the MX host to check SPF without blocking any email, and then add a "Received-SPF" header field as specified in RFC 4408. Downstream hosts can then look at the Received-SPF header and set their own policy of whether to reject, accept, or quarantine based on the SPF result and other factors.
DoS attack
An Internet draft discussed concerns related to the scale of an SPF answer leading to network exploits as a means to corrupt the DNS. This issue is also covered in the security considerations of the SPF RFC. The SPF project did a detailed analysis of this draft and claimed that SPF does not pose any unique threat of DNS DoS.Relationship with DKIM
SPF validates the message envelope (the SMTP bounce addressBounce address
A bounce address is an e-mail address to which bounce messages are delivered. There are many variants of the name, none of them used universally, including return path, reverse path, envelope from, envelope sender, MAIL FROM, 5321-FROM, return address, From_, Errors-to, etc...
), not the message contents (header and body) – this is the distinction between SMTP (as specified in STD
Internet standard
In computer network engineering, an Internet Standard is a normative specification of a technology or methodology applicable to the Internet. Internet Standards are created and published by the Internet Engineering Task Force .-Overview:...
10 or RFC 5321) and Internet Message Format (as specified in STD 11 or RFC 5322). It is orthogonal and complementary to DomainKeys Identified Mail
DomainKeys Identified Mail
DomainKeys Identified Mail is a method for associating a domain name to an email message, thereby allowing a person, role, or organization to claim some responsibility for the message. The association is set up by means of a digital signature which can be validated by recipients...
(DKIM), which signs the contents (including headers).
In brief, SPF validates MAIL FROM vs. its source server; DKIM validates "From:" by cryptographic means.
Sender ID
Sender IDSender ID
Sender ID is an anti-spoofing proposal from the former MARID IETF working group that tried to join Sender Policy Framework and Caller ID. Sender ID is defined primarily in Experimental RFC 4406, but there are additional parts in RFC 4405, RFC 4407 and RFC 4408.- Principles of operation :Sender ID...
RFC 4406, is a parallel solution to the problem of message validation, and defines a pair of closely related tests. One validates a message's Purported Responsible Address (PRA) as defined in RFC 4407. The other validates a message's Reverse-Path (also known as MAIL-FROM address) as defined in RFC 4408.
Quoting from RFC4407:
"Note that the Sender ID experiment may use DNS records that may have been created for the current SPF experiment or earlier versions in this set of experiments. Depending on the content of the record, this may mean that sender-ID heuristics would be applied incorrectly to a message. Depending on the actions associated by the recipient with those heuristics, the message may not be delivered or may be discarded on receipt."
Those publishing SPF DNS records should consider the advice given in section 3.4 of RFC 4406 and may wish to publish both v=spf1 and spf2.0 records to avoid the conflict.
Caveats:
- Because PRA is defined as a fall-back hierarchy, the only header that is actually protected is "Resent-Sender". Any other header can be forged by simply adding a higher-priority header with a different domain (the highest priority being Resent-Sender). DKIM cryptographically protects an arbitrary number of header fields.
- Mail with a Return-Path failing in an SPF check can be rejected during the SMTP envelope phase, before bandwidth has been wasted on the actual email. Because SMTP does not provide a way to reject an email after the headers, but before the body, the entire email must be received before PRA can be checked.
Wide-mask vulnerability
Some spammers use SPF to decrease spam-rating by specifying wide mask in valid server address, so any spam from botnets becomes spf-valid and probability to pass spam-filters increasesseminar-for-you.ru. 14400 IN TXT "v=spf1 a mx ip4:55.11.65.20/2 ip4:90.2.123.112/2 ip4:176.33.87.19/2 ip4:212.63.89.33/2 -all"
worldwidemail.ru. 13733 IN TXT "v=spf1 a mx ip4:55.11.65.20/2 ip4:90.2.123.112/2 ip4:176.33.87.19/2 ip4:212.63.89.33/2 -all"
example.com. 21600 IN SPF "v=spf1 +all"
This last record says that any host on the Internet may send mail on behalf of the domain/hostname example.com. Although syntactically valid, "+all" is indicative of an administrator who does not care about SPF or the mail forgeries it detects.
For stable domains, this simply means that any reputation attached to the domain is the same with or without SPF and such spam domains are easily learned and rejected. The real value of wide-mask SPF policies to spammers is with "throw-away" domains that are registered, used to send spam from botnets for a few days, and then abandoned.
History
The idea to limit by IP address who could send mail using a given sender domain may date back as far as 1997. The first public mention of the concept was in 2000 but went mostly unnoticed. No mention was made of the concept again until a first attempt at an SPF-like specification was published in 2002 on the IETF "namedroppers" mailing list by David Green-Lank (formerly David Green), who was unaware of the 2000 mention of the idea. The very next day, Paul VixiePaul Vixie
Paul Vixie is an American Internet pioneer, the author of several RFCs and well-known Unix software.Vixie attended George Washington High School in San Francisco, California. He received a Ph.D in computer science from Keio University in 2011....
posted his own SPF-like specification on the same list. These posts ignited a lot of interest, and eventually led to the forming of the IETF Anti-Spam Research Group (ASRG) and their mailing list, where the SPF idea was debated among a subscriber base that seemed to grow exponentially day by day. Among the proposals submitted to the ASRG were "Reverse MX
MX record
A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available...
" by Hadmut Danisch, and "Designated Mailer Protocol" by Gordon Fecyk.
In June 2003, Meng Weng Wong
Meng Weng Wong
Meng Weng Wong is a serial entrepreneur. In 1994 he founded pobox.com, an email services company. In 2003 he led the group that designed the Sender Policy Framework standard which was later embraced and extended by Microsoft...
merged the RMX and DMP specifications and solicited suggestions from other programmers. Over the next six months, a large number of changes were made and a large community had started working on SPF.
Originally SPF stood for Sender Permitted From and was sometimes also called SMTP+SPF, but it was changed to Sender Policy Framework in February 2004.
In early 2004, the IETF created the MARID
MARID
MARID was an IETF working group in the applications area tasked to propose standards for E-mail authentication in 2004.The name is an acronym of MTA Authorization Records In DNS.- Background :Lightweight MTA Authentication Protocol...
working group and tried to use SPF and Microsoft's CallerID proposal as the basis for what is now known as Sender ID
Sender ID
Sender ID is an anti-spoofing proposal from the former MARID IETF working group that tried to join Sender Policy Framework and Caller ID. Sender ID is defined primarily in Experimental RFC 4406, but there are additional parts in RFC 4405, RFC 4407 and RFC 4408.- Principles of operation :Sender ID...
.
After the collapse of MARID the SPF community returned to the original "classic" version of SPF. In July 2005 this version of the specification was approved by the IESG
Internet Engineering Steering Group
The Internet Engineering Steering Group is a body composed of the Internet Engineering Task Force chair and area directors.It provides the final technical review of Internet standards and is responsible for day-to-day management of the IETF...
as an IETF experiment, inviting the community to observe SPF during the two years following publication. On April 28, 2006, the SPF RFC was published as experimental RFC 4408.
Controversy
In 2004, Steven M. BellovinSteven M. Bellovin
Steven M. Bellovin is a researcher on computer networking and security. He is currently a Professor in the Computer Science department at Columbia University, having previously been a Fellow at AT&T Labs Research in Florham Park, New Jersey.- Career :...
wrote an e-mail that discusses his concerns with SPF. Some of these include:
- SPF originally used TXT records in DNS, which are supposed to be free-form text with no semantics attached. SPF proponents readily acknowledge that it would be better to have records specifically designated for SPF, but this choice was made to enable rapid implementation of SPF. In July 2005, IANAInternet Assigned Numbers AuthorityThe Internet Assigned Numbers Authority is the entity that oversees global IP address allocation, autonomous system number allocation, root zone management in the Domain Name System , media types, and other Internet Protocol-related symbols and numbers...
assigned the Resource Record type 99 to SPF. During the transition, SPF publishers may publish both record types and SPF checkers may check for either type. It may likely take many years before all DNS software fully supports this new record. - As of the time he wrote his message, there was no consensus that this is the right way to go. Some major e-mail service providers have not bought into this scheme. Unless and until they do, it does not help much, either for their customers (who make up a substantial proportion of the user population) or for everyone else (since their addresses could be forged). It is worth noting that since this concern was raised, Google Mail and AOL, among others, have embraced SPF.
- Bellovin's strongest concerns involve the underlying assumptions of SPF (SPF's "semantic model"). When using SPF, the SPF DNS records determine how a sender is allowed to send. That means that the owner of the domain will control how senders are allowed to send. People who use "portable" e-mail addresses (such as e-mail addresses created by professional organizations) will be required to use the domain owner's SMTP sender, which may not currently even exist. Organizations providing these "portable" addresses could, however, create their own mail submission agentMail submission agentA mail submission agent is a computer program or software agent that receives electronic mail messages from a mail user agent and cooperates with a mail transfer agent for delivery of the mail...
s (MSAs) (RFC 6409) or offer VPNsVirtual private networkA virtual private network is a network that uses primarily public telecommunication infrastructure, such as the Internet, to provide remote offices or traveling users access to a central organizational network....
or simply not publish an SPF record. Besides, SPF only ties the SMTPSimple Mail Transfer ProtocolSimple Mail Transfer Protocol is an Internet standard for electronic mail transmission across Internet Protocol networks. SMTP was first defined by RFC 821 , and last updated by RFC 5321 which includes the extended SMTP additions, and is the protocol in widespread use today...
Return-Path to permitted MSAs; users are still free to use their RFC 5322 addresses elsewhere.
There are other concerns about the impact of widespread use of SPF, notably the impact on various legitimate forms of email spoofing, such as forwarding services, SMTP use by people with multiple identities, etc. (For example, a person who uses their home ISP's SMTP servers to send mail with their work email as the address.) On the other hand, many of these uses may be "expected" yet not "legitimate". To a certain extent this is more a question of ownership and expectations than a technical question.
Deployment
Anti-spam software such as SpamAssassinSpamAssassin
SpamAssassin is a computer program released under the Apache License 2.0 used for e-mail spam filtering based on content-matching rules. It is now part of the Apache Foundation....
version 3.0.0 and ASSP
Anti-Spam SMTP Proxy
The Anti-Spam SMTP Proxy server project is an Open Source, Perl based, platform-independent transparent SMTP proxy server available at SourceForge.net that leverages numerous methodologies and technologies to both rigidly and adaptively identify e-mail spam...
implement SPF. Many mail transfer agent
Mail transfer agent
Within Internet message handling services , a message transfer agent or mail transfer agent or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture...
s (MTAs) support SPF directly such as Courier
Courier Mail Server
The Courier mail server is a mail transfer agent server that provides ESMTP, IMAP, POP3, SMAP, webmail, and mailing list services with individual components. It is best known for its IMAP server component....
, CommuniGate Pro, Wildcat
Wildcat! BBS
Wildcat! BBS was a bulletin board system server application that Mustang Software developed in 1986 for DOS, and later ported to Microsoft Windows. By the release of Version 4 it was the basis for more than 50,000 bulletin board systems worldwide....
, MDaemon, and Microsoft Exchange
Microsoft Exchange Server
Microsoft Exchange Server is the server side of a client–server, collaborative application product developed by Microsoft. It is part of the Microsoft Servers line of server products and is used by enterprises using Microsoft infrastructure products...
, or have patches/plug-ins available that support SPF, including Postfix
Postfix (software)
In computing, Postfix is a free and open-source mail transfer agent that routes and delivers electronic mail. It is intended as a fast, easier-to-administer, and secure alternative to the widely-used Sendmail MTA....
, Sendmail
Sendmail
Sendmail is a general purpose internetwork email routing facility that supports many kinds of mail-transfer and -delivery methods, including the Simple Mail Transfer Protocol used for email transport over the Internet....
, Exim
Exim
Exim is a mail transfer agent used on Unix-like operating systems. Exim is free software distributed under the terms of the GNU General Public License, and it aims to be a general and flexible mailer with extensive facilities for checking incoming e-mail....
, and qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
. More than one million domains publish SPF FAIL -all policies.
In a survey published in 2007, 5% of the .com and .net domains had some kind of SPF policy. In 2009, a continuous survey run at Nokia Research reports that 51% of the tested domains specify an SPF policy. These results can include trivial policies like v=spf1 ?all. In April 2007, BITS, a division of the Financial Services Roundtable, published e-mail security recommendations for its members including SPF deployment.
In 2008, the Messaging Anti-Abuse Working Group (MAAWG
MAAWG
The Messaging Anti-Abuse Working Group started as a group of internet service providers, mobile network operators, telecommunications companies and infrastructure vendors and anti-spam technology vendors in early 2004. It has since expanded to include e-mail service providers and other forms of...
) published a paper about email-authentication covering SPF, Sender ID
Sender ID
Sender ID is an anti-spoofing proposal from the former MARID IETF working group that tried to join Sender Policy Framework and Caller ID. Sender ID is defined primarily in Experimental RFC 4406, but there are additional parts in RFC 4405, RFC 4407 and RFC 4408.- Principles of operation :Sender ID...
, and DKIM
DomainKeys Identified Mail
DomainKeys Identified Mail is a method for associating a domain name to an email message, thereby allowing a person, role, or organization to claim some responsibility for the message. The association is set up by means of a digital signature which can be validated by recipients...
. In their "Sender Best Communication Practices" the MAAWG stated: "At the very least, senders should incorporate SPF records for their mailing domains".
In August 2005 it was learned that EarthLink
EarthLink
EarthLink , is an Internet service provider headquartered in Atlanta, Georgia, USA. It claims 1.94 million subscribers.- Business :EarthLink provides a variety of Internet connection types, including dial-up, DSL, satellite, and cable. Both dial-up and high speed Internet access are available...
would refuse to allow hosted domains the ability to enter SPF records.
External links
- SPF homepage and SPF Mailing List Archives
- Diagram of e-mail flow and SPF's impact (PDF, PNG )
- MIT Spam Conference Handout on SPF (Jan 2004)
- SpamCop FAQ entry about bogus bounces discusses also SPF (2005)
- M. Wong's Deployment White Paper (PDF, Nov 2004)
- libspf2 - an open source implementation of the SPF protocol (2010)
