Fix virtual environment details in knot_benchmark

[AlexWaygood]

Following #13227, pyright is looking for a virtual environment in a directory called '--threads', and no such directory exists. This PR fixes that, which slows pyright down again when checking black from 251.9ms to 1.751s on my machine

[AlexWaygood]

I think this is a fairer comparison between mypy/pyright/redknot... but I'm still seeing lots of errors regarding missing imports from both mypy and pyright if I run uv run benchmark --project black -vv locally.

Mypy:

src/blackd/middlewares.py:3:1: error: Cannot find implementation or library stub for module named "aiohttp.web_request"  [import-not-found]
src/blackd/middlewares.py:4:1: error: Cannot find implementation or library stub for module named "aiohttp.web_response"  [import-not-found]
src/black/output.py:11:1: error: Cannot find implementation or library stub for module named "click"  [import-not-found]
src/black/report.py:9:1: error: Cannot find implementation or library stub for module named "click"  [import-not-found]
src/black/cache.py:12:1: error: Cannot find implementation or library stub for module named "platformdirs"  [import-not-found]
src/black/files.py:21:1: error: Cannot find implementation or library stub for module named "packaging.specifiers"  [import-not-found]
src/black/files.py:22:1: error: Cannot find implementation or library stub for module named "packaging.version"  [import-not-found]
src/black/files.py:34:1: error: Cannot find implementation or library stub for module named "tomli"  [import-not-found]

Pyright:

  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:20:6 - warning: Import "mypy_extensions" could not be resolved from source (reportMissingModuleSource)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:21:6 - error: Import "packaging.specifiers" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:22:6 - error: Import "packaging.version" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:23:6 - error: Import "pathspec" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:24:6 - error: Import "pathspec.patterns.gitwildmatch" could not be resolved (reportMissingImports)
  /private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmp1lb8nym4/src/black/files.py:42:12 - warning: Import "colorama" could not be resolved from source (reportMissingModuleSource)

So something else may be going wrong somewhere...

[MichaReiser]

Thanks. I was just about to revert the commit because I don't see any performance improvements by just using the max number of threads (but e.g. 4 threads is faster on my machine for black).

So something else may be going wrong somewhere...

This is what I pointed out in the original PR summary and asked for help:

I'm a bit surprised that I e.g. see an error for Black. I would appreciate it if someone could verify that the diagnostics are reasonable.

#13026

[AlexWaygood]

This is what I pointed out in the original PR summary and asked for help:

Yeah, it's odd! I checked the output of the tools at the time and I don't remember seeing anything like this. I remember everything looking similar to what I'd expect at the time. I'm looking into it now.

Adding the --verbose flag to pyright's invocation in the script indicates that it is successfully finding the virtual environment:

Could not resolve source for 'mypy_extensions' in file '/private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src/black/nodes.py'
  Looking in root directory of execution environment 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92'
  Attempting to resolve using root path 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92'
  Looking in extraPath 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src'
  Attempting to resolve using root path 'file:///private/var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/src'
  Finding python search paths
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib'; looking for site-packages
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/site-packages', so looking for python subdirectory
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib64'
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib'; looking for site-packages
  Did not find 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib/site-packages', so looking for python subdirectory
  Found path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/Lib/python3.12/site-packages'
  Found the following 'site-packages' dirs
    file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages
  Looking in python search path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'
  Attempting to resolve using root path 'file:///var/folders/gd/1czdxc454j15y8gns2krv8vc0000gn/T/tmpesdbtb92/venv/lib/python3.12/site-packages'

[MichaReiser]

Yeah it's not that all packages are missing, just some. The packages that are missing are also missing in the dependencies arrays (which I took from mypy-primer)

[AlexWaygood]

The packages that are missing are also missing in the dependencies arrays (which I took from mypy-primer)

In the logs I posted above, I see mypy errors about not being able to resolve click, and pyright errors about not being able to resolve packaging. Both of those should be py.typed packages we have installed into the virtual environment, no?

ruff/scripts/knot_benchmark/src/benchmark/projects.py

Lines 96 to 109 in c2aac5f

	Project(
	name="black",
	repository="https://github.com/psf/black",
	revision="c20423249e9d8dfb8581eebbfc67a13984ee45e9",
	include=["src"],
	dependencies=[
	"aiohttp",
	"click",
	"pathspec",
	"tomli",
	"platformdirs",
	"packaging",
	],
	),

[MichaReiser]

This is the only error that I get when running mypy over black

Benchmark 1: mypy
src/blackd/__init__.py:91:22: error: List item 0 has incompatible type "Callable[[Request, Callable[[Request], Awaitable[StreamResponse]]], Awaitable[StreamResponse]]"; expected "Middleware"  [list-item]
Warning: unused section(s) in pyproject.toml: module = ['tests.data.*']
Found 1 error in 1 file (checked 40 source files)

[AlexWaygood]

(We debugged this offline -- it seems like something changed in uv between v0.3 and v0.4? We're not sure what, but specifying --python for the uv pip install fixes things, and is more explicit anyway 😄)

[codspeed-hq]

CodSpeed Performance Report

Merging #13228 will degrade performances by 4.45%

_{Comparing alex/redknot-bench (61ee689) with main (c2aac5f)}

Summary

❌ 1 regressions
✅ 31 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	alex/redknot-bench	Change
❌	linter/default-rules[pydantic/types.py]	1.8 ms	1.9 ms	-4.45%