Callback verification
Encyclopedia
Callback verification, also known as callout verification or Sender Address Verification, is a technique used by SMTP software in order to validate e-mail address
es. The most common target of verification is the sender address from the message envelope (the address specified during the SMTP dialogue as "MAIL FROM"). It is mostly used as an anti-spam measure.
has forged sender ("mfrom") addresses, some spam can be detected by checking whether forging resulted in an invalid address, using this method.
A related technique is "call forwards", in which a secondary or firewall mail exchanger can verify recipients at the primary mail exchanger for the domain in order to decide whether the address is deliverable.
s), pretending to be creating a bounce, but stopping just before any e-mail is sent. The commands sent out are:
HELO
MAIL FROM:<>
RCPT TO:
QUIT
Equivalently, the MAIL FROM and RCPT TO commands can be replaced by the VRFY command, however the VRFY command is not required to be supported and is usually disabled in modern MTAs.
Both of these techniques are technically compliant with the relevant SMTP RFCs (RFC 5321), however RFC 2505 (a Best Current Practice
) recommends, by default, disabling the VRFY command to prevent directory harvest attack
s. (One widespread interpretation implies that the MAIL FROM/RCPT TO pair of commands should also respond the same way, but this is not stated by the RFCs.)
To work around a limitation (see below), the MAIL FROM can be non-blank.
and exim
caution against the use of this technique and mention many limitations to SMTP callbacks. In particular, there are many situations where it is either ineffective or causes problems to the systems that receive the callbacks.
Some of these problems are caused by originating systems violating or stretching the limits of RFCs; verification problems are only reflecting these problems back to the senders, like unintentionally used invalid addresses, rejection of the null sender, or greylisting (where, for example, the delay caused by the verifying recipient is closely related to the delay caused by the originator). In many cases this in turn helps originator system to detect the problems, and fix them (like unintentionally not being able to receive valid bounces).
Several of the above problems are reduced by caching
of verification results. In particular, systems that give no useful information (not rejecting at the RCPT TO time, have catch-all e-email, etc.) can be remembered and no future call backs to those systems need to be made. Also, results (positive or negative) for specific e-mail addressas can be remembered. MTAs like Exim
have caching built in.
E-mail address
An email address identifies an email box to which email messages are delivered. An example format of an email address is lewis@example.net which is read as lewis at example dot net...
es. The most common target of verification is the sender address from the message envelope (the address specified during the SMTP dialogue as "MAIL FROM"). It is mostly used as an anti-spam measure.
Purpose
Since a large percentage of e-mail spamE-mail spam
Email spam, also known as junk email or unsolicited bulk email , is a subset of spam that involves nearly identical messages sent to numerous recipients by email. Definitions of spam usually include the aspects that email is unsolicited and sent in bulk. One subset of UBE is UCE...
has forged sender ("mfrom") addresses, some spam can be detected by checking whether forging resulted in an invalid address, using this method.
A related technique is "call forwards", in which a secondary or firewall mail exchanger can verify recipients at the primary mail exchanger for the domain in order to decide whether the address is deliverable.
Process
The receiving mail server verifies the sender address by making an SMTP connection to the mail exchanger for it (found via the usual MX recordMX record
A mail exchanger record is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient's domain, and a preference value used to prioritize mail delivery if multiple mail servers are available...
s), pretending to be creating a bounce, but stopping just before any e-mail is sent. The commands sent out are:
HELO
MAIL FROM:<>
RCPT TO:
QUIT
Equivalently, the MAIL FROM and RCPT TO commands can be replaced by the VRFY command, however the VRFY command is not required to be supported and is usually disabled in modern MTAs.
Both of these techniques are technically compliant with the relevant SMTP RFCs (RFC 5321), however RFC 2505 (a Best Current Practice
Best Current Practice
A best current practice is a de facto, dynamic level of performance in engineering and information technology. It is more flexible than a standard, since techniques and tools are continually evolving....
) recommends, by default, disabling the VRFY command to prevent directory harvest attack
Directory Harvest Attack
A Directory Harvest Attack or DHA is a technique used by spammers in an attempt to find valid/existent e-mail addresses at a domain by using brute force. The attack is usually carried out by way of a standard dictionary attack, where valid e-mail addresses are found by brute force guessing valid...
s. (One widespread interpretation implies that the MAIL FROM/RCPT TO pair of commands should also respond the same way, but this is not stated by the RFCs.)
To work around a limitation (see below), the MAIL FROM can be non-blank.
Limitations
The documentation for both postfixPostfix (software)
In computing, Postfix is a free and open-source mail transfer agent that routes and delivers electronic mail. It is intended as a fast, easier-to-administer, and secure alternative to the widely-used Sendmail MTA....
and exim
Exim
Exim is a mail transfer agent used on Unix-like operating systems. Exim is free software distributed under the terms of the GNU General Public License, and it aims to be a general and flexible mailer with extensive facilities for checking incoming e-mail....
caution against the use of this technique and mention many limitations to SMTP callbacks. In particular, there are many situations where it is either ineffective or causes problems to the systems that receive the callbacks.
- Some regular mail exchangers do not give useful results to callbacks:
- Servers that reject all bounce mails (contrary to the RFCs). To work around this problem, postfix, for example, uses either the local postmasterPostmaster (computing)In computers and technology, postmaster is a term used to identify the administrator of a mail server. Nearly every domain will have the e-mail address postmaster@example.com where errors in e-mail processing are directed...
address or an address of "double-bounce" in the MAIL FROM part of the callout. This work-around, however, will fail if Bounce Address Tag ValidationBounce Address Tag ValidationIn computing, Bounce Address Tag Validation is a method, defined in an Internet Draft, for determining whether the bounce address specified in an E-mail message is valid...
is used to reduce backscatterBackscatter (e-mail)Backscatter is incorrect automated bounce messages sent by mail servers, typically as a side effect of incoming spam....
. Callback verification can still work if rejecting all bounces happens at the DATA stage instead of the earlier MAIL FROM stage, while rejecting invalid e-mail addresses remains at the RCPT TO stage instead of also being moved to the DATA stage. - Servers that accept all e-mail address at RCPT TO stage but reject invalid ones at DATA stage. This is commonly done in order to prevent directory harvest attackDirectory Harvest AttackA Directory Harvest Attack or DHA is a technique used by spammers in an attempt to find valid/existent e-mail addresses at a domain by using brute force. The attack is usually carried out by way of a standard dictionary attack, where valid e-mail addresses are found by brute force guessing valid...
s and will, by design, give no information about whether an e-mail address is valid and thus prevent callback verification from working. - Servers that accept all mails during the SMTP dialogue (and generate their own bounces later). This problem can be alleviated by testing a random non-existent address as well as the desired address (if the test succeeds, further verification is useless).
- Servers that implement catch-all e-mail will, by definition, consider all e-mail addresses to be valid and accept them. Like systems that accept-then-bounce, a random non-existent address can detect this.
- Servers that reject all bounce mails (contrary to the RFCs). To work around this problem, postfix, for example, uses either the local postmaster
- The callback process can cause delays in delivery because the mail server where an address is verified may use slow anti-spam techniques, including "greet delays" (causing a connection delay) and greylisting (causing a verification deferral).
- If the system being called back to uses greylistingGreylistingGreylisting is a method of defending e-mail users against spam. A mail transfer agent using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate the originating server will, after a delay, try again and, if sufficient time has elapsed, the...
the callback may return no useful information until the greylisting time has expired. Greylisting works by returning a "temporary failure" (a 4xx response code) when it sees an unfamiliar MAIL FROM/RCPT TO pair of email addresses. A greylisting system may not give a "permanent failure" (a 5xx response code) when given an invalid e-mail address for the RCPT TO, and may instead continue to return a 4xx response code. - Some e-mail may be legitimate but not have a valid "envelope from" address due to user error or just misconfiguration. The positive aspect is that the verification process will usually cause an outright rejection, so if the sender was not a spammer but a real user, they will be notified of the problem.
- If a server receives a lot of spam may do a lot of callbacks. If those addresses are invalid or spamtrapSpamtrapA spamtrap is a honeypot used to collect spam.Spamtraps are usually e-mail addresses that are created not for communication, but rather to lure spam...
, the server will look very similar to a spammer who is doing a dictionary attack to harvest addresses. This in turn might get the server blacklisted elsewhere. - Every callback places an unasked for burden on the system being called back to, with very few effective ways for that system to avoid the burden. In extreme cases, if a spammer abuses the same sender address and uses it at a sufficiently diverse set of receiving MXs, all of which use this method, they might all try the callback, overloading the MX for the forged address with requests (effectively a Distributed Denial of Service attack).
- Callback verification has no effect if spammers spoof real email addresses or use the null bounce address.
Some of these problems are caused by originating systems violating or stretching the limits of RFCs; verification problems are only reflecting these problems back to the senders, like unintentionally used invalid addresses, rejection of the null sender, or greylisting (where, for example, the delay caused by the verifying recipient is closely related to the delay caused by the originator). In many cases this in turn helps originator system to detect the problems, and fix them (like unintentionally not being able to receive valid bounces).
Several of the above problems are reduced by caching
Cache
In computer engineering, a cache is a component that transparently stores data so that future requests for that data can be served faster. The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere...
of verification results. In particular, systems that give no useful information (not rejecting at the RCPT TO time, have catch-all e-email, etc.) can be remembered and no future call backs to those systems need to be made. Also, results (positive or negative) for specific e-mail addressas can be remembered. MTAs like Exim
Exim
Exim is a mail transfer agent used on Unix-like operating systems. Exim is free software distributed under the terms of the GNU General Public License, and it aims to be a general and flexible mailer with extensive facilities for checking incoming e-mail....
have caching built in.