![]() ![]() I have a feeling that this may need to be solved one layer up, in the virtual environment abstraction. I believe reflinks require explicit system calls (like for example provides) and it's very much tied to which filesystem you have the files on). This is different thing than reflinks altogether IMHO. Optimization coming from avoiding going through multiple userkernel syscalls and using userspace buffers for that. Reflink is a feature of "some" Filesystem (nice explanation here) and the shutil Python 3.8 implementation just mentions "fast-copy" operation done in the kernel rather than using user-space. ![]() ![]() “fast-copy” means that the copying operation occurs within the kernel,Īvoiding the use of userspace buffers in Python as in “outfd.write(infd.read())”. “fast-copy” syscalls in order to copy the file more efficiently Starting from Python 3.8, all functions involving a file copy ( I agree to follow the PSF Code of Conduct.Given that platforms generally use shared memory-maps for shared object files, if it's done right this could additionally reduce the memory footprint of python interpreters in different virtualenvs with large C extensions loaded. pth file shenanigans but I feel like those all have significant drawbacks. You could get a similar reduction effect by setting up an import hook, using zipimport, or doing some kind of. (My ~/Library/Caches/pip is only 256M, and presumably all those virtualenvs contain multiple full, uncompressed copies of it!) Alternative Solutions While there may be other bottlenecks, this would also reduce disk usage by an order of magnitude. ![]() Rather than unpacking and duplicating all the data in wheels, pip could store the cache unpacked, so all the files are already on the filesystem, and then clone them into place on copy-on-write filesystems rather than copying them. It also takes up a lot of space my ~/.virtualenvs/ is almost 3 gigabytes, and this is a relatively new machine and that isn't even counting my ~/.local/pipx, which is another 434M. What's the problem this feature will solve?Ĭreating a new virtual environment in a modern Python project can be quite slow, sometimes on the order of tens of seconds even on very high-end hardware, once you have a lot of dependencies. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |