VizTracer supports concurrency tracing, including asyncio, multi-thread and multi-process.
asyncio module natively. However, you can use
--log_async to make the report clearer.
Under the rug, asyncio is a single-thread program that’s scheduled by Python built-ins. With
--log_async, you can visualize
different tasks as “threads”, which could separate the real work from the underlying structure, and give you a more intuitive
understanding of how different tasks consume the runtime.
viztracer --log_async my_script.py
VizTracer supports python native
threading module without the need to do any modification to your code.
VizTracer before you create threads and it will just work.
If you are using multi-thread via other mechanism, for example, PyQt thread, VizTracer can’t support it out of the box.
However, you can notice VizTracer that you are in a separate thread and enable tracing in that thread with
from viztracer import get_tracer
# This will tell VizTracer to trace the thread
subprocess. You need to make sure the main process exits after subprocesses finish.
This will generate an HTML file for all processes. There are a couple of things you need to be aware though.
VizTracer patches subprocess module(to be more specific,
subprocess.Popen) to make this work like a magic. However, it will only patch
when the args passed to
subprocess.Popen is a list(
subprocess.Popen(["python", "subscript.py"])) and the first argument starts with
python. This covers most of the cases, but if you do have a situation that can’t be solved, you can raise an issue and we can talk
multiprocessing and concurrent.futures
concurrent.futures, and it will make the main process wait for all the other processes to finish
so the report can include all processes. You can skip the waiting using Ctrl+C.
This feature is available on all platforms and for both
However, on Windows,
multiprocessing.Pool won’t work with VizTracer because there’s no way to gracefully catch the exit of the process
os.fork, with some caveats.
On Python3.8+, it works well, the main process will wait for
forked processes to finish. You can even use
os.exec() and its other forms after you fork the process. Of course
VizTracer only records what happens before
os.exec(), you need generic multi process support to record
what happens after.
On Python3.6/3.7, VizTracer is not able to wait for the forked process to finish. It would be user’s responsibility to wait for the forked process to finish if they want to see both processes in the report.
loky implemented the
viztracer initializer. You can log
just as easy as builtin
generic multi process support
VizTracer has a simple instrumentation for all the third party libraries to integrate VizTracer to their multi process code.
First, your main process has to be executed by
viztracer. Inline VizTracer won’t work. In your program, you need
get_tracer().init_kwargs, which is a
Dict that can be easily serializable with
pickle or other libraries.
Then, pass this argument to your sub-process, and instantiate a VizTracer object with it
# init_kwargs is the argument from main process
tracer = VizTracer(**init_kwargs)
And you are good to go. The main process should collect the data from sub-processes automatically and put together a report.
You can generate json reports from different processes and combine them manually as well. It is recommended to use
--pid_suffix so the report will be saved as a json file ending with the pid of the process. You can specify your own file name using
viztracer --pid_suffix single_process.py
viztracer -o process1.json single_process.py
You can specify the output directory if you want to
viztracer --pid_suffix --output_dir ./temp_dir single_process.py
json files, you need to combine them
viztracer --combine ./temp_dir/*.json
This will generate the HTML report with all the process info. You can specify
--output_file when using