backports: add threaded Coccinelle spatch support
Turns out Coccinelle has threaded support, it lets you
break down the target code you need to parse for a spatch
file into separate buckets, each bucket handling a subset of
the target files needed to be evaluated. This adds support
for that and shaves down the amount of time it gets to
generate target code by 97.19%.
Using:
mcgrof@drvbp1 ~/backports (git::master)$ time \
./gentree.py --verbose \
--clean \
--refresh \
/home/mcgrof/linux-next/ \
/home/mcgrof/build/next-
20140311
Before this patch:
real 19m34.867s
user 18m43.492s
sys 0m21.244s
After this patch by using num_cpus threads:
real 2m42.012s
user 21m53.616s
sys 0m22.748s
After this patch by using 3 * num_cpus threads:
real 1m33.316s
user 24m6.080s
sys 0m28.264s
Using num_cpus threads alone was giving 2m42.012s, its why I
experimented in increasing this. I also experimented with
6 * num_cpus threads as well but the amount of time spent
running didn't reduce, lets keep this at 3 * num_cpus for now
and we can experiment and empirically find the best setting for
any system or just make this configurable.
We had SmPL patch integration first committed via
86d99c9d4 back then we were able to scale to a reasonable
similar time:
mcgrof@drvbp1 ~/backports (git::cocci-added)$ time \
./gentree.py --verbsose \
--clean \
--refresh \
/home/mcgrof/linux-next
/home/mcgrof/build/next-
20131113
real 1m1.967s
user 0m49.304s
sys 0m11.880s
For history's purpose here's our run time without Coccinelle on
commit
338c4388cc:
mcgrof@drvbp1 ~/backports (git::before-smpl)$ time \
./gentree.py --verbose \
--clean \
--refresh \
/home/mcgrof/linux-next/ \
/home/mcgrof/build/next-
20131029
real 0m29.639s
user 0m17.508s
sys 0m11.568s
This new changes sets us back on track with the timing for when we first
introduced Coccinelle support we just have a lot 5 Coccinelle patches
now. I hope this shows we should be able to scale with Coccinelle much more
efficiently for collateral evolutions that sprinkle through tons of
files.
Screenshots available using htop before [0] and after the patch [1]
while coccinelle is running, this shows how the system is using
resources much more efficiently when doing spatch evaluation for
patch generation on target code.
[0] http://drvbp1.linux-foundation.org/~mcgrof/images/coccinelle-backports/before-threaded-cocci.png
[1] http://drvbp1.linux-foundation.org/~mcgrof/images/coccinelle-backports/after-threaded-cocci.png
Cc: Peter Senna <peter.senna@gmail.com>
Cc: Julia Lawall <julia.lawall@lip6.fr>
Cc: Gilles Muller <Gilles.Muller@lip6.fr>
Signed-off-by: Luis R. Rodriguez <mcgrof@do-not-panic.com>