The case of color ls(1)

The command ls -G displays files with colors on OSX and FreeBSD only when standard input is attached to a terminal. When using the subprocess module, we will not see any of these qualities:

import subprocess
print(subprocess.check_output(['ls', '-G', '/dev']))

With an explicit -G parameter, the output of this program is still colorless. This quick example shows that some programs behave differently when attached to a terminal.

Interactive

Furthermore, some programs are only interactive when attached to a terminal. The python executable is an example of this. When we run python directly from a terminal, we receive an interactive REPL:

$ python
Python 3.5.0 (default, Oct 28 2015, 21:00:27)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print(4+4)
8
>>> exit()

If we run these commands by piping them to standard input, it will not display such decorators, demonstrated here using the standard shell:

$ printf 'print(2+2)\nexit()' | python
4

And strangely enough, executing Python from Python, using the subprocess module demonstrates the same output:

import subprocess, sys
python = subprocess.Popen(
    sys.executable, stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(python.communicate(input=b"print(2+2)\nexit()"))

(b'4\n', b'')

With a keyboard attached, a terminal may be expected to provide input at any non-determinate future time. Programs such as python test whether any of the standard file descriptors (stdin, stdout, stderr) are attached to a terminal to conditionally offer this behaviour.

We can reproduce this conditional check of isatty(3) easily from shell:

$ python -c 'import sys,os;print(os.isatty(sys.stdin.fileno()))'
True

$ echo | python -c 'import sys,os;print(os.isatty(sys.stdin.fileno()))'
False

As stdin is piped, this fails the test for isatty(3) test.

Cheating isatty(3)

The remainder of this article will focus on tricking isatty(3) into returning True even when the standard descriptors are not actually terminal. This peculiar behavior begins by a call to the standard python pty.fork function. This behaves exactly as os.fork, except that a pseudo terminal (pty(4)) is wedged between the child and parent process.

Why is this useful? Let's examine some programs that make use of pty(4) and fork(2) to explain for themselves:

• tmux(1) and screen(1) make use of pty(4) to perform their magic: the real terminal may leave (detach), while the child continues to believe it is connected with a terminal.
• script(1) records interactive sessions, ensuring all terminal sequences are written to file typescript for analysis.
• ttyrec(1) records sessions like script(1), but with timing information. This is the driving technology behind https://asciinema.org/ for example.
• IPython notebook executes programs through a pty(4) for color output.
• Travis CI uses a pty(4) so test runners produce colorized output.

Finally, the traditional Unix expect(1) by Don Libes uses a pty(4) to allow "programmed dialogue with interactive programs". The remainder of this article will use pexpect: a variant of expect(1) authored by Noah Spurrier