Reading from processes safely in Haskell

2015-03-04T00:00:00Z

Quite a bit of my work lately has been implementing HTTP interfaces to existing systems. In a few cases this required invoking existing command-line tools and parsing their output. The naive approach to invoking a process in Haskell and reading its output goes something like this:

import System.Exit
import System.Process

main :: IO ()
main = do
    let p = (shell "cat /usr/share/dict/words")
            { std_in  = Inherit
            , std_out = CreatePipe
            , std_err = Inherit
            }
    (Nothing, Just out, Nothing, ph) <- createProcess p
    ec <- waitForProcess ph
    case ph of
        ExitSuccess   -> hGetContents out >>= print
        ExitFailure _ -> error "Bad things happened. :-("

There is a potential problem in this code: we wait until the process has terminated before reading the Handle allowing its output to accumulate in the pipe buffer managed by the operating system in the mean time. This buffer has a fixed size on most systems (this is a good thing!); when it fills up, the writing process will go to sleep until the reader has consumed some data and freed some buffer space to hold the next write. Alas, the reader (the Haskell code above) is sleeping, waiting for the writer to terminate. The reader is sleeping, waiting for the writer to terminate; and the writer is sleeping, waiting for the reader to read. This is a deadlock!

The solution is to do the Right Thing (tm) and take care of any buffering behaviour we want ourselves. Thankfully this is pretty straightforward and it’s the sort of code you generally only need to write once. The very simplest case – reading from a process with a single output Handle – looks like this:

gatherOutput :: ProcessHandle -> Handle -> IO (ExitCode, ByteString)
gatherOutput ph h = work mempty
  where
    work acc = do
        -- Read any outstanding input.
        bs <- BS.hGetNonBlocking h (64 * 1024)
        let acc' = acc <> bs
        -- Check on the process.
        s <- getProcessExitCode ph
        -- Exit or loop.
        case s of
            Nothing -> work acc'
            Just ec -> do
                -- Get any last bit written between the read and the status
                -- check.
                last <- BS.hGetContents h
                return (ec, acc' <> last)

This is essentially a loop which reads some input from the Handle (possibly an empty string), checks to see if the process has terminated, and either returns the accumulated input or loops again. Extending this to gather the output of two handles (like stderr and stdout) is relatively straightforward.

Passing Curiosity: Posts tagged deadlock

Reading from processes safely in Haskell