Unix: How can I write a code for PIPE (|) in C/ shell script/python?

This is much, much easier to do in Python than in C, primarily because Python comes with a few libraries that do all of this for you; you have to do all the gory details yourself if you use C. Python There are a few modules you can use to start a subprocess and communicate with it, but one of the most convenient is, unsurprisingly, the http://docs.python.org/2/library/subprocess.html. Use subprocess.Popen() to start the subprocess and return a handle to it. Use the handle to send it input, wait for it to finish, and get its output. Here's an example REPL session that uses grep to filter some input: >>> import subprocess >>> grep = subprocess.Popen(["grep", "-n", "^ba", "-"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) >>> grep.stdin.write("foo\nbar\nbaz\nqux\n") >>> grep.stdin.close() >>> grep.wait() 0 >>> grep.stderr.read() '' >>> grep.stdout.readlines() ['2:bar\n', '3:baz\n'] C This task requires intimate knowledge of UNIX http://en.wikipedia.org/wiki/File_descriptor and processes. First, here are the system calls used to accomplish what you're asking: read: reads from a file descriptor to an in-memory buffer write: writes from an in-memory buffer to a file descriptor close: closes a file descriptor; processes reading from it will be notified that there is no more data to be read ("end of file") fork: spawns an identical process as the calling program, which runs independently of the calling process, continuing execution from the fork call; the calling process is the "parent" and the new copy is the "child"; one important detail of fork is that each process' file descriptors refer to the same file descriptors exec: starts the given program and then makes the calling process become it, using the calling process' standard input, standard output, and standard error exactly as the calling process left them; after calling exec, the calling process is no more -- it finishes execution as the program given as the argument waitpid: waits for a child process to finish pipe: creates two file descriptors: one for the read end of the pipe and one for the write end; data written to the write file descriptor can be read from the read file descriptor (duh) dup2: clobbers a file descriptor with another; useful for redirecting a process' standard input, standard output, and standard error to/from pipes instead of the terminal The method goes something like this: Create pipes to interface with the process you want to call, for whichever of standard input, standard output, and standard error you need to use. Fork. The child process: Close the write end of the standard input pipe (if such a pipe was made) and the read ends of the standard output and standard error pipes (if they've been made); these are for the parent's use. Use dup2 to force the child to use the pipes as standard input, output, and error. For example, after doing this, anything the child tries to write to standard output (e.g. printf statements) will go to the write end of the standard output pipe instead. Finally, have the child use exec to become the program you want to run. All the changes you've made to the child's standard input, output, and error will carry over, forcing the program to use those pipes. The parent process: Close the read end of the standard input pipe and the write ends of the standard output and error pipes; these are for the child's use. Pass whatever data needed by the child process, potentially via the write end of the standard input pipe. If the parent uses the write end of the standard input pipe, it should close it when it finishes to signify to the child process that there is no more data. Wait for the child to terminate. Read whatever output data you need from the pipes the child wrote to. The following is a very simple, error-prone program which calls another program and obtains its output. It takes the program to be called and any arguments it should be given as its own arguments. For the sake of clarity, I've skipped over a lot of error checking and handling, which you should not do if you want your program to be stable. Every system call, for example, returns a status code that should be checked; I don't do this to avoid clutter in this example. #include <libgen.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/wait.h> #include <unistd.h> void read_all(int src, int dst) { char buf[BUFSIZ]; ssize_t bytes_read, bytes_written; while ((bytes_read = read(src, buf, BUFSIZ)) > 0) { bytes_written = 0; while (bytes_written < bytes_read) bytes_written += write(dst, buf + bytes_written, bytes_read - bytes_written); } } int main(int argc, char** argv) { if (argc < 2) { printf("usage: %s <program> [<arg> ...]\n", basename(argv[0])); return EXIT_FAILURE; } // create pipes for standard input, output, and error int stdin_pipe[2]; int stdout_pipe[2]; int stderr_pipe[2]; pipe(stdin_pipe); pipe(stdout_pipe); pipe(stderr_pipe); if (fork() == 0) { // child process // close write end of stdin and read ends of stdout and stderr close(stdin_pipe[1]); close(stdout_pipe[0]); close(stderr_pipe[0]); // change child's stdin, stdout, and stderr to use pipes dup2(stdin_pipe[0], STDIN_FILENO); dup2(stdout_pipe[1], STDOUT_FILENO); dup2(stderr_pipe[1], STDERR_FILENO); // exec the given program if (execvp(argv[1], argv+1) == -1) { perror("failed to start subprocess"); return EXIT_FAILURE; } } // parent process // close read end of stdin and write ends of stdout and stderr close(stdin_pipe[0]); close(stdout_pipe[1]); close(stderr_pipe[1]); // pass input to the child process read_all(STDIN_FILENO, stdin_pipe[1]); close(stdin_pipe[1]); // wait for child to finish wait(NULL); // read child's stdout and stderr puts("\nchild's stdout:"); fflush(stdout); read_all(stdout_pipe[0], STDOUT_FILENO); close(stdout_pipe[0]); puts("\nchild's stderr:"); fflush(stdout); read_all(stderr_pipe[0], STDOUT_FILENO); close(stderr_pipe[0]); return EXIT_SUCCESS; } Here are some example runs (assuming that this program is called pipe_test): Calling grep -n '^ba' - (search for lines beginning with "ba"), and passing in input via the terminal. $ ./pipe_test grep -n '^ba' - foo bar baz qux ^D child's stdout: 2: bar 3: baz child's stderr: Example that writes to standard error: $ ./pipe_test cat -x < /dev/null child's stdout: child's stderr: cat: invalid option -- 'x' Try 'cat --help' for more information.

Typically, after interpreting the command line, the shell will fork()/exec() the necessary processes tying stdout to stdin using a pipe.  This is usually done iteratively for as many commands as you have "piped" together.  In linux, pipes are part of the VFS whose entry can be found http://lxr.linux.no/linux+v3.12.6/fs/pipe.c#L997. Here a little example showing how to create two processes, and hook stdout of the first process up to stdin of the second.  Make sure you understand that a pipe is a kernel structure that has a read end and a write end.  Anything written to the write end is able to be read from the read end.  The ends of the pipe are file descriptors.  Because file descriptors live in the task control block, they persist through a fork() or exec() call.  Additionally, the two halves of the pipe can be "duped" onto other file descriptors.  As you will see, the basic mechanism uses these attributes to basically do the following: Creates a pipe. Creates two new processes. Dupes the write half of the pipe to stdout of the first process. Dupes the read half of the pipe to stdin of the second process. Exec()s new images for the two processes. Now, anything the first process writes to stdout will be sent to the pipe, and may be read from stdin on the second process. // Basic pipe example illustrating a pipe between two processes. #include <errno.h> #include <stdio.h> #include <unistd.h> int main(int argc, char* argv[]) { pid_t pid1, pid2; int pipefd[2]; // The two commands we'll execute. In this simple example, we will pipe // the output of `ls` into `wc`, and count the number of lines present. char *argv1[] = {"ls", "-l", "-h", NULL}; char *argv2[] = {"wc", "-l", NULL}; // Create a pipe. pipe(pipefd); // Create our first process. pid1 = fork(); if (pid1 == 0) { // Hook stdout up to the write end of the pipe and close the read end of // the pipe which is no longer needed by this process. dup2(pipefd[1], STDOUT_FILENO); close(pipefd[0]); // Exec `ls -l -h`. If the exec fails, notify the user and exit. Note // that the execvp variant first searches the $PATH before calling execve. execvp(argv1[0], argv1); perror("exec"); return 1; } // Create our second process. pid2 = fork(); if (pid2 == 0) { // Hook stdin up to the read end of the pipe and close the write end of // the pipe which is no longer needed by this process. dup2(pipefd[0], STDIN_FILENO); close(pipefd[1]); // Similarly, exec `wc -l`. execvp(argv2[0], argv2); perror("exec"); return 1; } // Close both ends of the pipe. The respective read/write ends of the pipe // persist in the two processes created above (and happen to be tying stdout // of the first processes to stdin of the second). close(pipefd[0]); close(pipefd[1]); // Wait for everything to finish and exit. waitpid(pid1); waitpid(pid2); return 0; } Note that, for brevity, I left out some error checking code.  Things like fork() and pipe() can fail, and you would usually wrap them in a block that looks like: if (pid=fork() < 0) { perror("Could not create process"); exit(1); } This isn't super interesting, but know that for anything other than example code, you would probably want to catch system level errors.

>>"and what is the actual code which unix use for piping." If you want to know how the Linux pipe works, refer to the usage and implementation details in chapter 4 of http://books.google.co.in/books?id=CoQ_AQAAIAAJ&q=unix+network+programming+volume+2&dq=unix+network+programming+volume+2&hl=en&sa=X&ei=LBYvUsilNob8iAec3oDIDQ&ved=0CC8Q6AEwAA by Richard Stevens. The pipe() is a system call whose interface is defined in unistd.h. It's used to allow for passing information between 2 processes which are related to each other with destructive read and flow control (producer-consumer) semantics. Usually an application will fork() and the parent and child processes will communicate with each other using pipe(). There is a related interface called popen() defined in stdio.h which also creates a pipe like: FILE *popen(const char* command, const char* type); Here command is a shell command line to the executable of the process to whom your program will pass the data to via the pipe(), like /bin/grep, and type is either 'r' or 'w'. So for the shell pipe, bash will fork a process again after cat and pass the 'command' as the path to the grep binary whereis grepwhereis grep as the process with whom the data has to be shared, and internally it will use the pipe() again. Detailed example of the same is at https://github.com/githcx/unpv2/blob/master/pipe/mainpopen.c (not mine). >>"so I want to write the code for this pipe command in c/python." I expect that Python wrappers will invoke the same pipe via glibc.

in case of C: 1)  as you are writing program in c you should know the basics of linux / unix and some sort of system programming for creating pipe in c programming language you need to create child and parent process to handle the command which you are going to pipe for that go through http://www.csl.mtu.edu/cs4411.ck/www/NOTES/process/fork/create.html http://www.tutorialspoint.com/unix_system_calls/fork.htm and http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html 2) For crating pipes using c language use http://www.tldp.org/LDP/lpg/node11.html http://www.gnu.org/software/libc/manual/html_node/Creating-a-Pipe.html and good one http://www.cs.cf.ac.uk/Dave/C/node23.html . in case of python: 1) Just like C use python with shell programming http://www.cyberciti.biz/faq/python-execute-unix-linux-command-examples/ http://stackoverflow.com/questions/89228/calling-an-external-command-in-python 2) for pipes http://docs.python.org/2/library/pipes.html

Only for  bash: Anon fifo [1] :- a)  Process-substitution [2] diff <(A <file1) <(A <file2) tee >(A1 >out1) >(A2 >out2) < is not same as pipe. [3] A :-  some command. b)  Co-process [4] If it's not a read only filesystem & you can use named pipes: mkfifo my_pipe '''example''' ls < my_pipe > grep x & '''gives pid''' strace -e trace=open -p $pid -s 80 -o output.txt cat output.txt [1] [2] [3] [4]