Expect
Encyclopedia
Expect is a Unix
automation and testing tool, written by Don Libes
as an extension to the Tcl
scripting language, for interactive applications
such as telnet
, ftp, passwd
, fsck
, rlogin
, tip
, ssh
, and others. It uses Unix pseudo terminal
s to wrap up subprocesses transparently, allowing the automation of arbitrary applications that are accessed over a terminal. With Tk
, interactive applications can be wrapped in X11 GUI
s.
pattern matching and general program capabilities, allowing simple scripts to intelligently control programs such as telnet, ftp, and ssh, all of which lack a programming language, macros, or any other program mechanism. The result is that Expect scripts provide old tools with significant new power and flexibility.
spawn telnet $remote_server
expect "username:"
send "$my_user_id\r"
expect "password:"
send "$my_password\r"
expect "%"
send "$my_command\r"
expect "%"
set results $expect_out(buffer)
send "exit\r"
expect eof
Another example is a script that automates ftp:
set timeout -1
spawn ftp $remote_server
expect "username:"
send "$my_user_id\r"
expect "password:"
send "$my_password\r"
expect "ftp>"
send "bin\r"
expect "ftp>"
send "prompt\r"
expect "ftp>"
send "mget *\r"
expect "ftp>"
send "bye\r"
expect eof
Below is an example that automates sftp, with password:
proc connect {passw} {
expect {
"Password:" {
send "$passw\r"
expect {
"sftp*" {
return 0
}
}
}
}
# timed out
return 1
}
set user [lindex $argv 0]
set passw [lindex $argv 1]
set host [lindex $argv 2]
set location [lindex $argv 3]
set file1 [lindex $argv 4]
set file2 [lindex $argv 5]
if { $user
puts "Usage: \n"
exit 1
}
spawn sftp $user@$host
set rez [connect $passw]
if { $rez 0 } {
send "cd $location\r"
set timeout -1
send "put $file2\r"
send "put $file1\r"
send "ls -l\r"
send "quit\r"
expect eof
exit 0
}
puts "\nError connecting to server: $host, user: $user and password: $passw!\n"
exit 1
another example of automated ssh login in user machine
set time_out 60
spawn ssh $user@machine
while {1} {
expect {
eof {break}
"The authenticity of host" {send "yes\r"}
"password:" {send "$password\r"}
"*\]" { send "exit\r"}
}
}
wait
close $spawn_id
Usage
Expect serves as a "glue" to link existing utilities together. The general idea is to try to figure out how to make Expect utilize the system's existing tools rather than figure out how to solve a problem inside of Expect.
A key usage of Expect involves commercial software products. Many of these products provide some type of command-line interface, but these usually lack the power needed to write scripts. They were built to service the users administering the product, but the company often doesn't spend the resources to fully implement a robust scripting language. An Expect script can spawn a shell, look up environmental variables, perform some Unix commands to retrieve more information, and then enter into the product's command-line interface armed with the necessary information to achieve the user's goal. After looking up information inside the product's command-line interface, the script can make an intelligent decision about what action to take, if any.
Every time an Expect operation is completed, the results are stored in a local variable called $expect_out. This allows the script to harvest information to feedback to the user, and it also allows conditional behavior of what to send next based on the circumstances.
A common use of Expect is to set up a testing suite, whether it be for programs, utilities or embedded systems. DejaGnu
is a testing suite written using Expect for use in testing. It has been used extensively for testing gcc
and is very well suited to testing remote targets such as embedded development.
You can automate the generation of an expect script using a tool called 'autoexpect'. This tool observes your actions and generates an expect script using heuristics. Though generated code may be large and somewhat cryptic, you can always tweak the generated script to get the exact code.
to encapsulate system administration tasks. This works because Expect merely uses system administration tools already located on the host computer. No extra tools need to be learned. If the programmer has already learned Tcl
, then migrating to Expect is a relatively easy transition. The same programming structures and syntax exist, but with additional features built in.
There is large support in the industry for using Expect for many in-house administration tasks. It is widely used by companies such as Silicon Graphics, IBM, HP, Sun, Xerox, Amdahl, Tektronix, AT&T, ComputerVision and the World Bank to run in-house automated testing for development projects, file transfers, account administration, and network testing.
Expect has been ported to Python, Perl and Java languages in various add-on module projects. Subroutines generally are an interpretation of the original version - with equivalent functionality. Once one understands the concept, one can trivially move to other languages as needed.
, and Perl
, Expect has a different twist. It is sometimes challenging to remember when a variable must be prefixed with a "$", and when it must not. There are versions of Expect available for Perl
and Python
for those familiar with their syntax.
Another limitation is the difficulty in porting Expect scripts between platforms. For example, an Expect script that was written to use several Unix-based tools, might not be suitable if migrated to a Windows platform. If possible, the programmer must find counterpart command-line applications that provide the same information, and this will probably require changing the send/expects, which can be a major part of the script. This is not an issue if you load tcl, perl or python on the machines in question, and use those languages' native POSIX
interfaces for accessing files, and standard POSIX utilities (telnet, ftp etc.) for remote interaction.
A less obvious argument against Expect is that it can enable sub-optimal solutions. For example, a systems administrator needing to log into multiple servers for automated changes might use Expect with stored passwords, rather than the better solution of ssh
agent keys. The ability to automate interactive tools is attractive, but there are frequently other options that can accomplish the same tasks in a more robust manner.
Expect cannot automate GUI
based tools. This is generally only a problem on Windows where for many tasks a GUI
based interface is the only option. In these situations tools like Autohotkey
, AutoIt
or Winbatch
can be used instead.
External links
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
automation and testing tool, written by Don Libes
Don Libes
Don Libes is a computer scientist at NIST performing computer science research on interoperability. He works in the Manufacturing Systems Integration Division, which performs research on software integration methods, creating custom software that implements draft standards and serves as an...
as an extension to the Tcl
Tcl
Tcl is a scripting language created by John Ousterhout. Originally "born out of frustration", according to the author, with programmers devising their own languages intended to be embedded into applications, Tcl gained acceptance on its own...
scripting language, for interactive applications
Application software
Application software, also known as an application or an "app", is computer software designed to help the user to perform specific tasks. Examples include enterprise software, accounting software, office suites, graphics software and media players. Many application programs deal principally with...
such as telnet
TELNET
Telnet is a network protocol used on the Internet or local area networks to provide a bidirectional interactive text-oriented communications facility using a virtual terminal connection...
, ftp, passwd
Passwd (command)
passwd is a tool on most Unix and Unix-like operating systems used to change a user's password. The password entered by the user is run through a key derivation function to create a hashed version of the new password, which is saved...
, fsck
Fsck
The system utility fsck is a tool for checking the consistency of a file system in Unix and Unix-like operating systems such as Linux.-Use:...
, rlogin
Rlogin
rlogin is a software utility for Unix-like computer operating systems that allows users to log in on another host via a network, communicating via TCP port 513.It was first distributed as part of the 4.2BSD release....
, tip
Tip (unix utility)
tip is a unix utility for establishing a terminal connection to a remote system via a modem. It is commonly associated with Sun's Solaris as it comes with that operating system.-Basics:...
, ssh
Secure Shell
Secure Shell is a network protocol for secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel over an insecure network: a server and a client...
, and others. It uses Unix pseudo terminal
Pseudo terminal
In some operating systems, including Unix, a pseudo terminal is a pseudo-device pair that provides a text terminal interface without an associated device, such as a virtual console, computer terminal or serial port...
s to wrap up subprocesses transparently, allowing the automation of arbitrary applications that are accessed over a terminal. With Tk
Tk (framework)
Tk is an open source, cross-platform widget toolkit that provides a library of basic elements for building a graphical user interface in many different programming languages....
, interactive applications can be wrapped in X11 GUI
Gui
Gui or guee is a generic term to refer to grilled dishes in Korean cuisine. These most commonly have meat or fish as their primary ingredient, but may in some cases also comprise grilled vegetables or other vegetarian ingredients. The term derives from the verb, "gupda" in Korean, which literally...
s.
Basics
Expect has regular expressionRegular expression
In computing, a regular expression provides a concise and flexible means for "matching" strings of text, such as particular characters, words, or patterns of characters. Abbreviations for "regular expression" include "regex" and "regexp"...
pattern matching and general program capabilities, allowing simple scripts to intelligently control programs such as telnet, ftp, and ssh, all of which lack a programming language, macros, or any other program mechanism. The result is that Expect scripts provide old tools with significant new power and flexibility.
Examples
A simple example is a script that automates a telnet session:- Assume $remote_server, $my_user_id, $my_password, and $my_command were read in earlier
- in the script.
- Open a telnet session to a remote server, and wait for a username prompt.
spawn telnet $remote_server
expect "username:"
- Send the username, and then wait for a password prompt.
send "$my_user_id\r"
expect "password:"
- Send the password, and then wait for a shell prompt.
send "$my_password\r"
expect "%"
- Send the prebuilt command, and then wait for another shell prompt.
send "$my_command\r"
expect "%"
- Capture the results of the command into a variable. This can be displayed, or written to disk.
set results $expect_out(buffer)
- Exit the telnet session, and wait for a special end-of-file character.
send "exit\r"
expect eof
Another example is a script that automates ftp:
- Set timeout parameter to a proper value.
- For example, the file size is indeed big and the network speed is really one problem, you'd better set this parameter a value.
set timeout -1
- Open an ftp session to a remote server, and wait for a username prompt.
spawn ftp $remote_server
expect "username:"
- Send the username, and then wait for a password prompt.
send "$my_user_id\r"
expect "password:"
- Send the password, and then wait for an ftp prompt.
send "$my_password\r"
expect "ftp>"
- Switch to binary mode, and then wait for an ftp prompt.
send "bin\r"
expect "ftp>"
- Turn off prompting.
send "prompt\r"
expect "ftp>"
- Get all the files
send "mget *\r"
expect "ftp>"
- Exit the ftp session, and wait for a special end-of-file character.
send "bye\r"
expect eof
Below is an example that automates sftp, with password:
- !/usr/local/bin/expect -f #<---insert here your expect program location
- procedure to attempt connecting; result 0 if OK, 1 otherwise
proc connect {passw} {
expect {
"Password:" {
send "$passw\r"
expect {
"sftp*" {
return 0
}
}
}
}
# timed out
return 1
}
- read the input parameters
set user [lindex $argv 0]
set passw [lindex $argv 1]
set host [lindex $argv 2]
set location [lindex $argv 3]
set file1 [lindex $argv 4]
set file2 [lindex $argv 5]
- puts "Argument data:\n";
- puts "user: $user";
- puts "passw: $passw";
- puts "host: $host";
- puts "location: $location";
- puts "file1: $file1";
- puts "file2: $file2";
- check if all were provided
if { $user
"" || $passw
"" || $host"" || $location
"" || $file1"" || $file2
"" } {puts "Usage:
exit 1
}
- sftp to specified host and send the files
spawn sftp $user@$host
set rez [connect $passw]
if { $rez 0 } {
send "cd $location\r"
set timeout -1
send "put $file2\r"
send "put $file1\r"
send "ls -l\r"
send "quit\r"
expect eof
exit 0
}
puts "\nError connecting to server: $host, user: $user and password: $passw!\n"
exit 1
another example of automated ssh login in user machine
- time_out is predefine variable in expect which is default set to 10 sec
- spawn_id is another default variable in expect. It is good practice to close spawn_id handle created by spawn command
set time_out 60
spawn ssh $user@machine
while {1} {
expect {
eof {break}
"The authenticity of host" {send "yes\r"}
"password:" {send "$password\r"}
"*\]" { send "exit\r"}
}
}
wait
close $spawn_id
Usage
Expect serves as a "glue" to link existing utilities together. The general idea is to try to figure out how to make Expect utilize the system's existing tools rather than figure out how to solve a problem inside of Expect.
A key usage of Expect involves commercial software products. Many of these products provide some type of command-line interface, but these usually lack the power needed to write scripts. They were built to service the users administering the product, but the company often doesn't spend the resources to fully implement a robust scripting language. An Expect script can spawn a shell, look up environmental variables, perform some Unix commands to retrieve more information, and then enter into the product's command-line interface armed with the necessary information to achieve the user's goal. After looking up information inside the product's command-line interface, the script can make an intelligent decision about what action to take, if any.
Every time an Expect operation is completed, the results are stored in a local variable called $expect_out. This allows the script to harvest information to feedback to the user, and it also allows conditional behavior of what to send next based on the circumstances.
A common use of Expect is to set up a testing suite, whether it be for programs, utilities or embedded systems. DejaGnu
DejaGnu
DejaGnu is a framework for testing other programs. It has a main script called runtest that goes through a directory looking at configuration files and then runs some tests with given criteria. The purpose of the DejaGnu package is to provide a single front end for all tests. It is a part of the...
is a testing suite written using Expect for use in testing. It has been used extensively for testing gcc
GNU Compiler Collection
The GNU Compiler Collection is a compiler system produced by the GNU Project supporting various programming languages. GCC is a key component of the GNU toolchain...
and is very well suited to testing remote targets such as embedded development.
You can automate the generation of an expect script using a tool called 'autoexpect'. This tool observes your actions and generates an expect script using heuristics. Though generated code may be large and somewhat cryptic, you can always tweak the generated script to get the exact code.
Pros
Expect can be run at regular intervals through the use of cronCron
Cron is a time-based job scheduler in Unix-like computer operating systems. Cron enables users to schedule jobs to run periodically at certain times or dates...
to encapsulate system administration tasks. This works because Expect merely uses system administration tools already located on the host computer. No extra tools need to be learned. If the programmer has already learned Tcl
Tcl
Tcl is a scripting language created by John Ousterhout. Originally "born out of frustration", according to the author, with programmers devising their own languages intended to be embedded into applications, Tcl gained acceptance on its own...
, then migrating to Expect is a relatively easy transition. The same programming structures and syntax exist, but with additional features built in.
There is large support in the industry for using Expect for many in-house administration tasks. It is widely used by companies such as Silicon Graphics, IBM, HP, Sun, Xerox, Amdahl, Tektronix, AT&T, ComputerVision and the World Bank to run in-house automated testing for development projects, file transfers, account administration, and network testing.
Expect has been ported to Python, Perl and Java languages in various add-on module projects. Subroutines generally are an interpretation of the original version - with equivalent functionality. Once one understands the concept, one can trivially move to other languages as needed.
Cons
Expect inherits the same syntax convention as Tcl, which may seem unfamiliar if accustomed to other script languages. Compared to languages such as bash, cshC shell
The C shell is a Unix shell that was created by Bill Joy while a graduate student at University of California, Berkeley in the late 1970s. It has been distributed widely, beginning with the 2BSD release of the BSD Unix system that Joy began distributing in 1978...
, and Perl
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
, Expect has a different twist. It is sometimes challenging to remember when a variable must be prefixed with a "$", and when it must not. There are versions of Expect available for Perl
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
and Python
Python (programming language)
Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Python claims to "[combine] remarkable power with very clear syntax", and its standard library is large and comprehensive...
for those familiar with their syntax.
Another limitation is the difficulty in porting Expect scripts between platforms. For example, an Expect script that was written to use several Unix-based tools, might not be suitable if migrated to a Windows platform. If possible, the programmer must find counterpart command-line applications that provide the same information, and this will probably require changing the send/expects, which can be a major part of the script. This is not an issue if you load tcl, perl or python on the machines in question, and use those languages' native POSIX
POSIX
POSIX , an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems...
interfaces for accessing files, and standard POSIX utilities (telnet, ftp etc.) for remote interaction.
A less obvious argument against Expect is that it can enable sub-optimal solutions. For example, a systems administrator needing to log into multiple servers for automated changes might use Expect with stored passwords, rather than the better solution of ssh
Secure Shell
Secure Shell is a network protocol for secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel over an insecure network: a server and a client...
agent keys. The ability to automate interactive tools is attractive, but there are frequently other options that can accomplish the same tasks in a more robust manner.
Expect cannot automate GUI
Gui
Gui or guee is a generic term to refer to grilled dishes in Korean cuisine. These most commonly have meat or fish as their primary ingredient, but may in some cases also comprise grilled vegetables or other vegetarian ingredients. The term derives from the verb, "gupda" in Korean, which literally...
based tools. This is generally only a problem on Windows where for many tasks a GUI
Gui
Gui or guee is a generic term to refer to grilled dishes in Korean cuisine. These most commonly have meat or fish as their primary ingredient, but may in some cases also comprise grilled vegetables or other vegetarian ingredients. The term derives from the verb, "gupda" in Korean, which literally...
based interface is the only option. In these situations tools like Autohotkey
AutoHotkey
AutoHotkey is a free, open source macro-creation and automation software utility which allows users to automate repetitive tasks. Any application user interface can be modified by AutoHotkey...
, AutoIt
AutoIt
AutoIt is a freeware automation language for Microsoft Windows. In its earliest release, the software was primarily intended to create automation scripts for Microsoft Windows programs but has since grown to include enhancements in both programming language design and overall functionality.With...
or Winbatch
Winbatch
Winbatch is a Microsoft Windows scripting language developed by Wilson WindowWare. Its environment includes an interpreter and a code editor along with a dialog designer and optional compiler to create self-contained executables....
can be used instead.
External links
- Expect on SourceForge (current)
- Official homepage
- The Tcler's Wiki -- Expect page
- Perl Expect.pm module
- Pexpect a Pure Python Expect-like module
- Empty - expectlike tool to run command-line interactive programs in UNIX shell-scripts
- Expect-lite -- a wrapper for expect, making automation easy, quick, and fun
- Article "Advanced Programming in Expect: A Bulletproof Interface"
- ExpectJ - a Java implementation of the Unix expect utility