Profiling

For simpler programs, you can resort to using cProfile and vprof. However, these fail when running multithreading/multiprocessing programs since these profilers only profile the parent process, not the child processes.

Multiprocessing

py-spy can be used to profile individual processes.

Here's my process when using it:

Launch program (put a import time; time.sleep(60) right after the child processes are launched to give yourself time to put in the py-spy commands. If it's a long program, remember to use tmux.
In another terminal session, launch htop.
Identify the PID's of the child processes.
Open $n$ more terminal sessions and put py-spy record -o profile.svg --pid 12345
Optional: You can also monitor the processes in real-time by running py-spy top --pid 12345

PreviousPickle NextRegex

Last updated 2 years ago

Multiprocessing

py-spy can be used to profile individual processes.

Here's my process when using it:

Launch program (put a import time; time.sleep(60) right after the child processes are launched to give yourself time to put in the py-spy commands. If it's a long program, remember to use tmux.

In another terminal session, launch htop.

Identify the PID's of the child processes.

Open $n$ more terminal sessions and put py-spy record -o profile.svg --pid 12345

Optional: You can also monitor the processes in real-time by running py-spy top --pid 12345