shell命令行并行神器 - parallel

shell命令行并行神奇 - parallel

概述

GNU parallel 是一个 shell 工具，用于使用一台或多台计算机并行执行作业。作业可以是单个命令或必须为输入中的每一行运行的小脚本。典型的输入是文件列表、主机列表、用户列表、URL 列表或表列表。作业也可以是从管道读取的命令。 GNU parallel 然后可以拆分输入并将其通过管道并行传输到命令中。

• 在 shell 中编写循环，你会发现 GNU parallel 可以取代大部分循环，并通过并行运行多个作业来使它们运行得更快。

• 对于每一行输入，GNU parallel 将以该行作为参数执行命令。如果没有给出命令，则执行输入行。多条线路将并行运行。

• GNU parallel 确保命令的输出与顺序运行命令的输出相同。这使得使用 GNU parallel 的输出作为其他程序的输入成为可能。
  

基本语法

熟悉xargs的同学对这个应该理解起来很快

1、生成五个文件并重定向输入

seq 5 | parallel seq {} '>' example.{}
# 回忆一下for 循环怎么写来着
# for i in `seq 5`;do echo `seq $i` > example-for.$i;done

2、parallel的输入

::: 后面跟的是其从命令行的输入

• parallel echo ::: 1 2 3 4 5

输出是

1
2
3
4
5

• parallel wc ::: example.*

输入是文件名

1 1 2 example.1
2 2 4 example.2
3 3 6 example.3
4 4 8 example.4
5 5 10 example.5

wc 默认输出解释

wc example.3
3 3 6 example.3
#行数 单词数 字节数 文件名

• parallel echo ::: S M L ::: Green Red

多个::: 输入，输出是排列组合

S Green
S Red
M Green
M Red
L Green
L Red

• find example.* -print | parallel echo File

parallel从标准输入读取

File example.1
File example.2
File example.3
File example.4
File example.5

3、和命令行的结合

# parallel echo counting lines';' wc -l ::: example.*
counting lines
1 example.1
counting lines
2 example.2
counting lines
3 example.3
counting lines
4 example.4
counting lines
5 example.5

用{}进行字符替换，这个是不是和xargs 很像

parallel echo test lines';' wc -l ::: example.*
test example.1
1 example.1
test example.2
2 example.2
test example.4
4 example.4
test example.3
3 example.3
test example.5
5 example.5

当有多个输入的时候，使用{1} {2}

例如需要分别统计example.*中的行数和字节数

# parallel echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
count -l in example.1
1 example.1
count -l in example.2
2 example.2
count -l in example.3
3 example.3
count -l in example.4
4 example.4
count -l in example.5
5 example.5
count -c in example.1
2 example.1
count -c in example.2
4 example.2
count -c in example.3
6 example.3
count -c in example.4
8 example.4
count -c in example.5
10 example.5

–dry-run 测试

# parallel --dry-run echo count {1} {2} ';' wc {1} {2} ::: -c -l ::: example.*
# 看这个结果已经不是顺序得了
echo count -c example.1 ; wc -c example.1
echo count -c example.2 ; wc -c example.2
echo count -c example.3 ; wc -c example.3
echo count -c example.5 ; wc -c example.5
echo count -c example.4 ; wc -c example.4
echo count -l example.1 ; wc -l example.1
echo count -l example.2 ; wc -l example.2
echo count -l example.3 ; wc -l example.3
echo count -l example.4 ; wc -l example.4

4、输出

5、并行数量

当然这个是并行的，并行数设置多少合适呢？

默认值是和你的os 的cores相同。一般为了限制parallel占据所有的cpu资源，建议使用 --jobs限制其并发数，作为脚本的参数输入比较常见

–jobs 0 竟可能多的并行

测试

# 并行为1，理论上就是5 4 3 2 1 =15 s
time parallel --jobs 1 sleep {}';' echo {} done ::: 5 4 3 1 2
# 并行为0，取决于最慢的那个sleep
time parallel --jobs 0 sleep {}';' echo {} done ::: 5 4 3 1 2

如果是五个job

Job slot 1: 55555
Job slot 2: 4444
Job slot 3: 333
Job slot 4: 1
Job slot 5: 22

6、处理大文本数据

将数据块传递给标准输入上

#seq 1000000 | parallel --pipe wc
 165668  165668 1048571
 149796  149796 1048572
 149796  149796 1048572
 149796  149796 1048572
 149796  149796 1048572
 149796  149796 1048572
  85352   85352  597465

大约 1 MB 的块传递给每个作业

1mb的行数、字符数、字节数

实战 并发docker run

并行启动dokcer 容器进行redis key迁移，效能大幅度提升。

通过以下脚本可以体会到 parallel的魅力:

• 代替了shell中的循环
• 线程数量可控制(原生shell循环做不到)
• 多线程输出可保证顺序(原生shell循环做不到)

#!/bin/bash
# date 2023年2月9日17:57:20
# author ninesun
# desc parallel docker run  
set -e
set -o pipefail
# 获取到程序的绝对路径
SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
# parallel 并行数量
JOBS=${JOBS:-5}

ERRORS="$(pwd)/errors"
INFO="$(pwd)/info"

dockerrun() {
	f=$1
    docker rm -f redis-img-${f}
    # echo ${f}
    docker run  --name redis-img-${f} 10.50.10.185/harbortest/redis-mig:1.2 python3 redisMigrate.py 10.50.10.45 19000 10.50.10.170 7100 \
     ${f} ::: "${files[@]}" >  $INFO/dockerrun.log

}
echo
echo

main(){
	# get the indexfile
	IFS=$'\n'
	mapfile -t files < <(find ./ -name "st*.txt.*" -o -name "line*.txt.*" |sed 's|./||'| sort)
	unset IFS
	# docker run all jobs
	echo "Running in parallel with ${JOBS} jobs."
	# 开启$jobs 个 /opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun xxx.txt
	parallel --tag --verbose --ungroup -j"${JOBS}" "$SCRIPT" dockerrun {1} ::: "${files[@]}"

	if [[ ! -f "$ERRORS" ]]; then
		echo "No errors, hooray!"
	else
		echo "[ERROR] Some images did not build correctly, see below." >&2
		echo "These images failed: $(cat "$ERRORS")" >&2
		exit 1
	fi
}

run(){
	args=$*
	f=$1

	if [[ "$f" == "" ]]; then
		main "$args"
	else
		$args
	fi
}

run "$@"

10个并发测试

./mig-v2.sh



Running in parallel with 10 jobs.
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence the citation notice: run 'parallel --bibtex'.

/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.000
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.001
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.002
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.003


lineurl.txt.000
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.004
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.005


lineurl.txt.001


lineurl.txt.002
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.006
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.007
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.008


lineurl.txt.004


lineurl.txt.003
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.009


lineurl.txt.008


lineurl.txt.005


lineurl.txt.007


lineurl.txt.006


lineurl.txt.009
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.010


lineurl.txt.010
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.000


startline.txt.000
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.001


startline.txt.001
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.002


startline.txt.002
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.003


startline.txt.003
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.004


startline.txt.004
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.005


startline.txt.005
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.006


startline.txt.006
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.007


startline.txt.007
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.008


startline.txt.008
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.009


startline.txt.009
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun startline.txt.010


startline.txt.010
No errors, hooray!

10个线程 压测性能

2g2c 本地拉起的虚拟机

10mins迁移完成

 ]# bash -x mig-v2.sh
  set -e
  set -o pipefail
    dirname mig-v2.sh
   cd .
   pwd
   basename mig-v2.sh
  SCRIPT=/opt/redis-mig/redis_key_mig/mig-v2.sh
  JOBS=1
   pwd
  ERRORS=/opt/redis-mig/redis_key_mig/errors
   pwd
  INFO=/opt/redis-mig/redis_key_mig/info
  echo

  echo

  run
  args=
  f=
  [[ '' == '' ]]
  main ''
  IFS='
'
  mapfile -t files
   find ./ -name 'st*.txt.*' -o -name 'line*.txt.*'
   sed 's|./||'
   sort
  unset IFS
  echo

  echo 'Running in parallel with 1 jobs.'
Running in parallel with 1 jobs.
  parallel --tag --verbose --ungroup -j1 /opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun '{1}' ::: lineurl.txt.000 lineurl.txt.001 lineurl.txt.002 lineurl.txt.003 lineurl.txt.004 lineurl.txt.005 lineurl.txt.006 lineurl.txt.007 lineurl.txt.008 lineurl.txt.009 lineurl.txt.010 startline.txt.000 startline.txt.001 startline.txt.002 startline.txt.003 startline.txt.004 startline.txt.005 startline.txt.006 startline.txt.007 startline.txt.008 startline.txt.009 startline.txt.010
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence the citation notice: run 'parallel --bibtex'.

/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.000


lineurl.txt.000
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.001


lineurl.txt.001
/opt/redis-mig/redis_key_mig/mig-v2.sh dockerrun lineurl.txt.002

参考

GNU_Parallel_2018.pdf

https://www.gnu.org/software/parallel/

这篇好文章是转载于：学新通技术网