Created
May 25, 2026 20:44
-
-
Save copyleftdev/7600185f6bcc14947305ab89b9bec5c2 to your computer and use it in GitHub Desktop.
parallel — run many independent tasks concurrently with job control
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # parallel — GNU parallel: xargs -P with sane defaults | |
| # sudo apt install parallel | |
| # Basic concurrent execution: | |
| cat tasks.txt | parallel <command> {} | |
| ls *.sh | parallel bash {} | |
| # Control concurrency: | |
| cat tasks.txt | parallel -j4 <command> {} # max 4 jobs | |
| cat tasks.txt | parallel -j0 <command> {} # one per CPU core | |
| # Keep output ordered and labeled: | |
| cat tasks.txt | parallel --tag <command> {} | |
| # Progress bar: | |
| cat tasks.txt | parallel --progress <command> {} | |
| # Retry failed jobs: | |
| cat tasks.txt | parallel --retries 3 <command> {} | |
| # Dry run: | |
| cat tasks.txt | parallel --dry-run <command> {} | |
| # Batched inference — pipe stdin blocks to each worker: | |
| cat data.jsonl | parallel -j4 --pipe --block 10k inference-tool | |
| # Replaces: for item in list; do cmd "$item" & done; wait | |
| # With: cat list | parallel cmd {} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment