Jul 10, 20:34:43 Worker assigned successfully. Starting soon. Jul 10, 20:34:43 Started container background Jul 10, 20:34:43 Created container background Jul 10, 20:34:43 Container image "quay.io/vessl-ai/kernels-toolbox:0.1.140" already present on machine Jul 10, 20:34:43 Successfully assigned pezghftdj35o/model-service-revision-juz3lnevo7jf-65fccb4f6b-8gddl to gke-managed-cluster--nvidia-l4-1-spot-5b50b8c9-kjgr 8gddl Jul 10, 20:34:43 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB] 8gddl Jul 10, 20:34:43 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB] 8gddl Jul 10, 20:34:43 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB] 8gddl Jul 10, 20:34:43 Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8788 kB] 8gddl Jul 10, 20:34:43 Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages.diff/Index [10.6 kB] 8gddl Jul 10, 20:34:44 Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages T-2024-04-23-2036.10-F-2023-11-06-2008.27.pdiff [8935 B] 8gddl Jul 10, 20:34:44 Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages T-2024-04-23-2036.10-F-2023-11-06-2008.27.pdiff [8935 B] 8gddl Jul 10, 20:34:44 Get:7 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [168 kB] 8gddl Jul 10, 20:34:45 Fetched 9229 kB in 1s (6668 kB/s) 8gddl Jul 10, 20:34:45 Reading package lists... Done 8gddl Jul 10, 20:34:45 Building dependency tree... Done 8gddl Jul 10, 20:34:45 Reading state information... Done 8gddl Jul 10, 20:34:45 47 packages can be upgraded. Run 'apt list --upgradable' to see them. 8gddl Jul 10, 20:34:45 N: Repository 'http://deb.debian.org/debian bookworm InRelease' changed its 'Version' value from '12.1' to '12.6' 8gddl Jul 10, 20:34:46 Reading package lists... Done 8gddl Jul 10, 20:34:46 Building dependency tree... Done 8gddl Jul 10, 20:34:46 Reading state information... Done 8gddl Jul 10, 20:34:46 The following additional packages will be installed: 8gddl Jul 10, 20:34:46 libcurl3-gnutls libcurl4 libjq1 libonig5 8gddl Jul 10, 20:34:46 The following NEW packages will be installed: 8gddl Jul 10, 20:34:46 jq libjq1 libonig5 8gddl Jul 10, 20:34:46 The following packages will be upgraded: 8gddl Jul 10, 20:34:46 curl libcurl3-gnutls libcurl4 8gddl Jul 10, 20:34:46 3 upgraded, 3 newly installed, 0 to remove and 44 not upgraded. 8gddl Jul 10, 20:34:46 Need to get 1477 kB of archives. 8gddl Jul 10, 20:34:46 After this operation, 1182 kB of additional disk space will be used. 8gddl Jul 10, 20:34:46 Get:1 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u6 [314 kB] 8gddl Jul 10, 20:34:46 Get:2 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u6 [390 kB] 8gddl Jul 10, 20:34:46 Get:3 http://deb.debian.org/debian bookworm/main amd64 libonig5 amd64 6.9.8-1 [188 kB] 8gddl Jul 10, 20:34:46 Get:4 http://deb.debian.org/debian bookworm/main amd64 libjq1 amd64 1.6-2.1 [135 kB] 8gddl Jul 10, 20:34:46 Get:5 http://deb.debian.org/debian bookworm/main amd64 jq amd64 1.6-2.1 [64.9 kB] 8gddl Jul 10, 20:34:46 Get:6 http://deb.debian.org/debian bookworm/main amd64 libcurl3-gnutls amd64 7.88.1-10+deb12u6 [385 kB] 8gddl Jul 10, 20:34:46 Fetched 1477 kB in 0s (19.3 MB/s) 8gddl Jul 10, 20:34:46 debconf: delaying package configuration, since apt-utils is not installed 8gddl Jul 10, 20:34:47 (Reading database ... 12850 files and directories currently installed.) 8gddl Jul 10, 20:34:47 Preparing to unpack .../0-curl_7.88.1-10+deb12u6_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking curl (7.88.1-10+deb12u6) over (7.88.1-10+deb12u1) ... 8gddl Jul 10, 20:34:47 Preparing to unpack .../1-libcurl4_7.88.1-10+deb12u6_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking libcurl4:amd64 (7.88.1-10+deb12u6) over (7.88.1-10+deb12u1) ... 8gddl Jul 10, 20:34:47 Selecting previously unselected package libonig5:amd64. 8gddl Jul 10, 20:34:47 Preparing to unpack .../2-libonig5_6.9.8-1_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking libonig5:amd64 (6.9.8-1) ... 8gddl Jul 10, 20:34:47 Selecting previously unselected package libjq1:amd64. 8gddl Jul 10, 20:34:47 Preparing to unpack .../3-libjq1_1.6-2.1_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking libjq1:amd64 (1.6-2.1) ... 8gddl Jul 10, 20:34:47 Selecting previously unselected package jq. 8gddl Jul 10, 20:34:47 Preparing to unpack .../4-jq_1.6-2.1_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking jq (1.6-2.1) ... 8gddl Jul 10, 20:34:47 Preparing to unpack .../5-libcurl3-gnutls_7.88.1-10+deb12u6_amd64.deb ... 8gddl Jul 10, 20:34:47 Unpacking libcurl3-gnutls:amd64 (7.88.1-10+deb12u6) over (7.88.1-10+deb12u1) ... 8gddl Jul 10, 20:34:47 Setting up libcurl3-gnutls:amd64 (7.88.1-10+deb12u6) ... 8gddl Jul 10, 20:34:47 Setting up libcurl4:amd64 (7.88.1-10+deb12u6) ... 8gddl Jul 10, 20:34:47 Setting up curl (7.88.1-10+deb12u6) ... 8gddl Jul 10, 20:34:47 Setting up libonig5:amd64 (6.9.8-1) ... 8gddl Jul 10, 20:34:47 Setting up libjq1:amd64 (1.6-2.1) ... 8gddl Jul 10, 20:34:47 Setting up jq (1.6-2.1) ... 8gddl Jul 10, 20:34:47 Processing triggers for libc-bin (2.36-9+deb12u1) ... 8gddl Jul 10, 20:34:47 % Total % Received % Xferd Average Speed Time Time Time Current 8gddl Jul 10, 20:34:47 Dload Upload Total Spent Left Speed 8gddl Jul 10, 20:34:47 100 243 100 241 100 2 468 3 --:--:-- --:--:-- --:--:-- 471 8gddl Jul 10, 20:34:47 Installing VESSL CLI... 8gddl Jul 10, 20:35:02 vessl, version 0.1.170 8gddl Jul 10, 20:35:02 ++ which git-lfs 8gddl Jul 10, 20:35:02 + [[ '' == '' ]] 8gddl Jul 10, 20:35:02 % Total % Received % Xferd Average Speed Time Time Time Current 8gddl Jul 10, 20:35:02 Dload Upload Total Spent Left Speed 8gddl Jul 10, 20:35:02 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 8gddl Jul 10, 20:35:02 100 4816k 100 4816k 0 0 8976k 0 --:--:-- --:--:-- --:--:-- 109M Jul 10, 20:35:03 Container image "quay.io/vessl-ai/torch:2.1.0-cuda12.2-r3" already present on machine Jul 10, 20:35:03 Created container model-service-revision 8gddl Jul 10, 20:35:03 Git LFS initialized. 8gddl Jul 10, 20:35:03 Git LFS initialized. Jul 10, 20:35:04 Started container run-sidecar Jul 10, 20:35:04 Started container fluentbit Jul 10, 20:35:04 Created container fluentbit Jul 10, 20:35:04 Started container model-service-revision Jul 10, 20:35:04 Container image "quay.io/vessl-ai/fluent-bit:2.1.6" already present on machine Jul 10, 20:35:04 Container image "quay.io/vessl-ai/sidecar:v0.0.1-rc3" already present on machine Jul 10, 20:35:04 Created container run-sidecar 8gddl Jul 10, 20:35:04 ++ which curl 8gddl Jul 10, 20:35:04 + [[ /usr/bin/curl == '' ]] 8gddl Jul 10, 20:35:05 Installing VESSL CLI... 8gddl Jul 10, 20:35:20 vessl, version 0.1.170 8gddl Jul 10, 20:35:21 + mkdir -p /root 8gddl Jul 10, 20:35:21 + cd /root 8gddl Jul 10, 20:35:21 + pip install ray 8gddl Jul 10, 20:35:21 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com 8gddl Jul 10, 20:35:22 Collecting ray 8gddl Jul 10, 20:35:22 Downloading ray-2.31.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (13 kB) 8gddl Jul 10, 20:35:22 Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from ray) (8.1.6) 8gddl Jul 10, 20:35:22 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from ray) (3.12.4) 8gddl Jul 10, 20:35:22 Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/dist-packages (from ray) (4.19.1) 8gddl Jul 10, 20:35:22 Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray) (1.0.5) 8gddl Jul 10, 20:35:22 Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from ray) (24.1) 8gddl Jul 10, 20:35:22 Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.10/dist-packages (from ray) (4.24.4) 8gddl Jul 10, 20:35:22 Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from ray) (6.0.1) 8gddl Jul 10, 20:35:22 Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/dist-packages (from ray) (1.3.1) 8gddl Jul 10, 20:35:22 Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/dist-packages (from ray) (1.4.0) 8gddl Jul 10, 20:35:22 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from ray) (2.31.0) 8gddl Jul 10, 20:35:22 Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema->ray) (23.1.0) 8gddl Jul 10, 20:35:22 Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema->ray) (2023.7.1) 8gddl Jul 10, 20:35:22 Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema->ray) (0.30.2) 8gddl Jul 10, 20:35:22 Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema->ray) (0.10.4) 8gddl Jul 10, 20:35:22 Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->ray) (3.2.0) 8gddl Jul 10, 20:35:22 Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->ray) (3.4) 8gddl Jul 10, 20:35:22 Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->ray) (1.26.16) 8gddl Jul 10, 20:35:22 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->ray) (2023.7.22) 8gddl Jul 10, 20:35:22 Downloading ray-2.31.0-cp310-cp310-manylinux2014_x86_64.whl (66.2 MB) 8gddl Jul 10, 20:35:22 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.2/66.2 MB 276.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:23 Installing collected packages: ray 8gddl Jul 10, 20:35:27 Successfully installed ray-2.31.0 8gddl Jul 10, 20:35:27 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 8gddl Jul 10, 20:35:27 + pip install autoawq==0.2.4 8gddl Jul 10, 20:35:28 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com 8gddl Jul 10, 20:35:28 Collecting autoawq==0.2.4 8gddl Jul 10, 20:35:28 Downloading autoawq-0.2.4-cp310-cp310-manylinux2014_x86_64.whl.metadata (16 kB) 8gddl Jul 10, 20:35:28 Requirement already satisfied: torch>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from autoawq==0.2.4) (2.1.0a0+32f93b1) 8gddl Jul 10, 20:35:28 Collecting transformers<=4.38.2,>=4.35.0 (from autoawq==0.2.4) 8gddl Jul 10, 20:35:28 Downloading transformers-4.38.2-py3-none-any.whl.metadata (130 kB) 8gddl Jul 10, 20:35:28 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.7/130.7 kB 9.1 MB/s eta 0:00:00 8gddl Jul 10, 20:35:29 Collecting tokenizers>=0.12.1 (from autoawq==0.2.4) 8gddl Jul 10, 20:35:29 Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) 8gddl Jul 10, 20:35:29 Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from autoawq==0.2.4) (4.8.0) 8gddl Jul 10, 20:35:29 Collecting accelerate (from autoawq==0.2.4) 8gddl Jul 10, 20:35:29 Downloading accelerate-0.32.1-py3-none-any.whl.metadata (18 kB) 8gddl Jul 10, 20:35:29 Collecting datasets (from autoawq==0.2.4) 8gddl Jul 10, 20:35:29 Downloading datasets-2.20.0-py3-none-any.whl.metadata (19 kB) 8gddl Jul 10, 20:35:29 Collecting zstandard (from autoawq==0.2.4) 8gddl Jul 10, 20:35:29 Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB) 8gddl Jul 10, 20:35:30 Collecting autoawq-kernels (from autoawq==0.2.4) 8gddl Jul 10, 20:35:30 Downloading autoawq_kernels-0.0.6-cp310-cp310-manylinux2014_x86_64.whl.metadata (2.0 kB) 8gddl Jul 10, 20:35:30 Collecting huggingface-hub<1.0,>=0.16.4 (from tokenizers>=0.12.1->autoawq==0.2.4) 8gddl Jul 10, 20:35:30 Downloading huggingface_hub-0.23.4-py3-none-any.whl.metadata (12 kB) 8gddl Jul 10, 20:35:30 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.2.4) (3.12.4) 8gddl Jul 10, 20:35:30 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.2.4) (1.12) 8gddl Jul 10, 20:35:30 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.2.4) (2.6.3) 8gddl Jul 10, 20:35:30 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.2.4) (3.1.2) 8gddl Jul 10, 20:35:30 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.2.4) (2023.6.0) 8gddl Jul 10, 20:35:30 Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (1.22.2) 8gddl Jul 10, 20:35:30 Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (24.1) 8gddl Jul 10, 20:35:30 Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (6.0.1) 8gddl Jul 10, 20:35:30 Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (2023.10.3) 8gddl Jul 10, 20:35:30 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (2.31.0) 8gddl Jul 10, 20:35:30 Collecting tokenizers>=0.12.1 (from autoawq==0.2.4) 8gddl Jul 10, 20:35:30 Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) 8gddl Jul 10, 20:35:30 Collecting safetensors>=0.4.1 (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) 8gddl Jul 10, 20:35:30 Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB) 8gddl Jul 10, 20:35:30 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (4.66.1) 8gddl Jul 10, 20:35:30 Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate->autoawq==0.2.4) (5.9.4) 8gddl Jul 10, 20:35:30 Collecting pyarrow>=15.0.0 (from datasets->autoawq==0.2.4) 8gddl Jul 10, 20:35:30 Downloading pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB) 8gddl Jul 10, 20:35:31 Collecting pyarrow-hotfix (from datasets->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB) 8gddl Jul 10, 20:35:31 Collecting dill<0.3.9,>=0.3.0 (from datasets->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB) 8gddl Jul 10, 20:35:31 Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->autoawq==0.2.4) (1.5.3) 8gddl Jul 10, 20:35:31 Collecting requests (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) 8gddl Jul 10, 20:35:31 Collecting tqdm>=4.27 (from transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading tqdm-4.66.4-py3-none-any.whl.metadata (57 kB) 8gddl Jul 10, 20:35:31 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 232.6 MB/s eta 0:00:00 8gddl Jul 10, 20:35:31 Collecting xxhash (from datasets->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) 8gddl Jul 10, 20:35:31 Collecting multiprocess (from datasets->autoawq==0.2.4) 8gddl Jul 10, 20:35:31 Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB) 8gddl Jul 10, 20:35:31 Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->autoawq==0.2.4) (3.8.5) 8gddl Jul 10, 20:35:31 Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (23.1.0) 8gddl Jul 10, 20:35:31 Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (3.2.0) 8gddl Jul 10, 20:35:31 Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (6.0.4) 8gddl Jul 10, 20:35:31 Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (4.0.3) 8gddl Jul 10, 20:35:31 Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (1.9.2) 8gddl Jul 10, 20:35:31 Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (1.4.0) 8gddl Jul 10, 20:35:31 Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq==0.2.4) (1.3.1) 8gddl Jul 10, 20:35:32 Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (3.4) 8gddl Jul 10, 20:35:32 Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (1.26.16) 8gddl Jul 10, 20:35:32 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers<=4.38.2,>=4.35.0->autoawq==0.2.4) (2023.7.22) 8gddl Jul 10, 20:35:32 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=2.0.1->autoawq==0.2.4) (2.1.3) 8gddl Jul 10, 20:35:32 Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->autoawq==0.2.4) (2.8.2) 8gddl Jul 10, 20:35:32 Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->autoawq==0.2.4) (2023.3) 8gddl Jul 10, 20:35:32 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=2.0.1->autoawq==0.2.4) (1.3.0) 8gddl Jul 10, 20:35:32 Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets->autoawq==0.2.4) (1.16.0) 8gddl Jul 10, 20:35:32 Downloading autoawq-0.2.4-cp310-cp310-manylinux2014_x86_64.whl (80 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.8/80.8 kB 257.8 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading transformers-4.38.2-py3-none-any.whl (8.5 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 MB 183.7 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 325.5 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading accelerate-0.32.1-py3-none-any.whl (314 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 314.1/314.1 kB 280.9 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading autoawq_kernels-0.0.6-cp310-cp310-manylinux2014_x86_64.whl (33.4 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.4/33.4 MB 296.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading datasets-2.20.0-py3-none-any.whl (547 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 246.7 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 245.5 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading dill-0.3.8-py3-none-any.whl (116 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 231.6 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading huggingface_hub-0.23.4-py3-none-any.whl (402 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 402.6/402.6 kB 255.9 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 289.9 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading requests-2.32.3-py3-none-any.whl (64 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 237.1 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 344.7 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading tqdm-4.66.4-py3-none-any.whl (78 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 251.9 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 298.4 MB/s eta 0:00:00 8gddl Jul 10, 20:35:32 Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB) 8gddl Jul 10, 20:35:32 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) 8gddl Jul 10, 20:35:32 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 301.7 MB/s eta 0:00:00 8gddl Jul 10, 20:35:33 Installing collected packages: zstandard, xxhash, tqdm, safetensors, requests, pyarrow-hotfix, pyarrow, dill, multiprocess, huggingface-hub, tokenizers, autoawq-kernels, accelerate, transformers, datasets, autoawq 8gddl Jul 10, 20:35:34 Attempting uninstall: tqdm 8gddl Jul 10, 20:35:34 Found existing installation: tqdm 4.66.1 8gddl Jul 10, 20:35:34 Uninstalling tqdm-4.66.1: 8gddl Jul 10, 20:35:34 Successfully uninstalled tqdm-4.66.1 8gddl Jul 10, 20:35:34 Attempting uninstall: requests 8gddl Jul 10, 20:35:34 Found existing installation: requests 2.31.0 8gddl Jul 10, 20:35:34 Uninstalling requests-2.31.0: 8gddl Jul 10, 20:35:34 Successfully uninstalled requests-2.31.0 8gddl Jul 10, 20:35:34 Attempting uninstall: pyarrow 8gddl Jul 10, 20:35:34 Found existing installation: pyarrow 11.0.0 8gddl Jul 10, 20:35:34 Uninstalling pyarrow-11.0.0: 8gddl Jul 10, 20:35:34 Successfully uninstalled pyarrow-11.0.0 8gddl Jul 10, 20:35:42 Successfully installed accelerate-0.32.1 autoawq-0.2.4 autoawq-kernels-0.0.6 datasets-2.20.0 dill-0.3.8 huggingface-hub-0.23.4 multiprocess-0.70.16 pyarrow-16.1.0 pyarrow-hotfix-0.6 requests-2.32.3 safetensors-0.4.3 tokenizers-0.15.2 tqdm-4.66.4 transformers-4.38.2 xxhash-3.4.1 zstandard-0.22.0 8gddl Jul 10, 20:35:42 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 8gddl Jul 10, 20:35:43 + pip install vllm==v0.4.1 fastapi==0.109.2 8gddl Jul 10, 20:35:43 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com 8gddl Jul 10, 20:35:44 Collecting vllm==v0.4.1 8gddl Jul 10, 20:35:44 Downloading vllm-0.4.1-cp310-cp310-manylinux1_x86_64.whl.metadata (8.9 kB) 8gddl Jul 10, 20:35:44 Collecting fastapi==0.109.2 8gddl Jul 10, 20:35:44 Downloading fastapi-0.109.2-py3-none-any.whl.metadata (25 kB) 8gddl Jul 10, 20:35:44 Requirement already satisfied: cmake>=3.21 in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (3.27.6) 8gddl Jul 10, 20:35:44 Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (1.11.1.1) 8gddl Jul 10, 20:35:44 Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (5.9.4) 8gddl Jul 10, 20:35:44 Collecting sentencepiece (from vllm==v0.4.1) 8gddl Jul 10, 20:35:44 Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB) 8gddl Jul 10, 20:35:44 Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (1.22.2) 8gddl Jul 10, 20:35:44 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (2.32.3) 8gddl Jul 10, 20:35:44 Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (8.0.0) 8gddl Jul 10, 20:35:44 Collecting transformers>=4.40.0 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:44 Downloading transformers-4.42.3-py3-none-any.whl.metadata (43 kB) 8gddl Jul 10, 20:35:44 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.6/43.6 kB 10.6 MB/s eta 0:00:00 8gddl Jul 10, 20:35:45 Collecting tokenizers>=0.19.1 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) 8gddl Jul 10, 20:35:45 Requirement already satisfied: uvicorn[standard] in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (0.24.0.post1) 8gddl Jul 10, 20:35:45 Requirement already satisfied: pydantic>=2.0 in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (2.4.2) 8gddl Jul 10, 20:35:45 Requirement already satisfied: prometheus-client>=0.18.0 in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (0.20.0) 8gddl Jul 10, 20:35:45 Collecting tiktoken==0.6.0 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading tiktoken-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) 8gddl Jul 10, 20:35:45 Collecting lm-format-enforcer==0.9.8 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading lm_format_enforcer-0.9.8-py3-none-any.whl.metadata (14 kB) 8gddl Jul 10, 20:35:45 Collecting outlines==0.0.34 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading outlines-0.0.34-py3-none-any.whl.metadata (13 kB) 8gddl Jul 10, 20:35:45 Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (4.8.0) 8gddl Jul 10, 20:35:45 Requirement already satisfied: filelock>=3.10.4 in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (3.12.4) 8gddl Jul 10, 20:35:45 Requirement already satisfied: ray>=2.9 in /usr/local/lib/python3.10/dist-packages (from vllm==v0.4.1) (2.31.0) 8gddl Jul 10, 20:35:45 Collecting nvidia-ml-py (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading nvidia_ml_py-12.555.43-py3-none-any.whl.metadata (8.6 kB) 8gddl Jul 10, 20:35:45 Collecting vllm-nccl-cu12<2.19,>=2.18 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:45 Downloading vllm_nccl_cu12-2.18.1.0.4.0.tar.gz (6.2 kB) 8gddl Jul 10, 20:35:47 Preparing metadata (setup.py) ... done 8gddl Jul 10, 20:35:48 Collecting torch==2.2.1 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:48 Downloading torch-2.2.1-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB) 8gddl Jul 10, 20:35:48 Collecting xformers==0.0.25 (from vllm==v0.4.1) 8gddl Jul 10, 20:35:48 Downloading xformers-0.0.25-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB) 8gddl Jul 10, 20:35:48 Collecting starlette<0.37.0,>=0.36.3 (from fastapi==0.109.2) 8gddl Jul 10, 20:35:48 Downloading starlette-0.36.3-py3-none-any.whl.metadata (5.9 kB) 8gddl Jul 10, 20:35:48 Collecting interegular>=0.3.2 (from lm-format-enforcer==0.9.8->vllm==v0.4.1) 8gddl Jul 10, 20:35:48 Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB) 8gddl Jul 10, 20:35:48 Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lm-format-enforcer==0.9.8->vllm==v0.4.1) (24.1) 8gddl Jul 10, 20:35:48 Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from lm-format-enforcer==0.9.8->vllm==v0.4.1) (6.0.1) 8gddl Jul 10, 20:35:48 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (3.1.2) 8gddl Jul 10, 20:35:48 Collecting lark (from outlines==0.0.34->vllm==v0.4.1) 8gddl Jul 10, 20:35:48 Downloading lark-1.1.9-py3-none-any.whl.metadata (1.9 kB) 8gddl Jul 10, 20:35:48 Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (1.5.8) 8gddl Jul 10, 20:35:48 Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (1.6.0) 8gddl Jul 10, 20:35:48 Collecting diskcache (from outlines==0.0.34->vllm==v0.4.1) 8gddl Jul 10, 20:35:48 Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB) 8gddl Jul 10, 20:35:48 Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (1.11.1) 8gddl Jul 10, 20:35:48 Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (0.57.1+1.g5fba9aa8f) 8gddl Jul 10, 20:35:48 Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (1.3.2) 8gddl Jul 10, 20:35:48 Requirement already satisfied: referencing in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (0.30.2) 8gddl Jul 10, 20:35:48 Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm==v0.4.1) (4.19.1) 8gddl Jul 10, 20:35:48 Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken==0.6.0->vllm==v0.4.1) (2023.10.3) 8gddl Jul 10, 20:35:48 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.2.1->vllm==v0.4.1) (1.12) 8gddl Jul 10, 20:35:48 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.2.1->vllm==v0.4.1) (2.6.3) 8gddl Jul 10, 20:35:48 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.2.1->vllm==v0.4.1) (2023.6.0) 8gddl Jul 10, 20:35:50 Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:50 Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:50 Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) 8gddl Jul 10, 20:35:51 Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) 8gddl Jul 10, 20:35:51 Collecting nvidia-nccl-cu12==2.19.3 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB) 8gddl Jul 10, 20:35:51 Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB) 8gddl Jul 10, 20:35:51 Collecting triton==2.2.0 (from torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB) 8gddl Jul 10, 20:35:51 Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.1->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB) 8gddl Jul 10, 20:35:51 Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->vllm==v0.4.1) (0.5.0) 8gddl Jul 10, 20:35:51 Requirement already satisfied: pydantic-core==2.10.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->vllm==v0.4.1) (2.10.1) 8gddl Jul 10, 20:35:51 Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==v0.4.1) (8.1.6) 8gddl Jul 10, 20:35:51 Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==v0.4.1) (1.0.5) 8gddl Jul 10, 20:35:51 Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==v0.4.1) (4.24.4) 8gddl Jul 10, 20:35:51 Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==v0.4.1) (1.3.1) 8gddl Jul 10, 20:35:51 Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==v0.4.1) (1.4.0) 8gddl Jul 10, 20:35:51 Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->vllm==v0.4.1) (3.2.0) 8gddl Jul 10, 20:35:51 Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->vllm==v0.4.1) (3.4) 8gddl Jul 10, 20:35:51 Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->vllm==v0.4.1) (1.26.16) 8gddl Jul 10, 20:35:51 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->vllm==v0.4.1) (2023.7.22) 8gddl Jul 10, 20:35:51 Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.37.0,>=0.36.3->fastapi==0.109.2) (3.7.1) 8gddl Jul 10, 20:35:51 Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers>=0.19.1->vllm==v0.4.1) (0.23.4) 8gddl Jul 10, 20:35:51 Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.0->vllm==v0.4.1) (0.4.3) 8gddl Jul 10, 20:35:51 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.0->vllm==v0.4.1) (4.66.4) 8gddl Jul 10, 20:35:51 Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]->vllm==v0.4.1) (0.14.0) 8gddl Jul 10, 20:35:51 Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB) 8gddl Jul 10, 20:35:51 Requirement already satisfied: python-dotenv>=0.13 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]->vllm==v0.4.1) (1.0.0) 8gddl Jul 10, 20:35:51 Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm==v0.4.1) 8gddl Jul 10, 20:35:51 Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) 8gddl Jul 10, 20:35:52 Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm==v0.4.1) 8gddl Jul 10, 20:35:52 Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) 8gddl Jul 10, 20:35:52 Collecting websockets>=10.4 (from uvicorn[standard]->vllm==v0.4.1) 8gddl Jul 10, 20:35:52 Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) 8gddl Jul 10, 20:35:52 Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.37.0,>=0.36.3->fastapi==0.109.2) (1.3.0) 8gddl Jul 10, 20:35:52 Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.37.0,>=0.36.3->fastapi==0.109.2) (1.1.3) 8gddl Jul 10, 20:35:52 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->outlines==0.0.34->vllm==v0.4.1) (2.1.3) 8gddl Jul 10, 20:35:52 Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema->outlines==0.0.34->vllm==v0.4.1) (23.1.0) 8gddl Jul 10, 20:35:52 Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema->outlines==0.0.34->vllm==v0.4.1) (2023.7.1) 8gddl Jul 10, 20:35:52 Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema->outlines==0.0.34->vllm==v0.4.1) (0.10.4) 8gddl Jul 10, 20:35:52 Requirement already satisfied: llvmlite<0.41,>=0.40.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->outlines==0.0.34->vllm==v0.4.1) (0.40.1) 8gddl Jul 10, 20:35:52 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.2.1->vllm==v0.4.1) (1.3.0) 8gddl Jul 10, 20:35:52 Downloading vllm-0.4.1-cp310-cp310-manylinux1_x86_64.whl (84.0 MB) 8gddl Jul 10, 20:35:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.0/84.0 MB 299.4 MB/s eta 0:00:00 8gddl Jul 10, 20:35:52 Downloading fastapi-0.109.2-py3-none-any.whl (92 kB) 8gddl Jul 10, 20:35:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.1/92.1 kB 278.5 MB/s eta 0:00:00 8gddl Jul 10, 20:35:52 Downloading lm_format_enforcer-0.9.8-py3-none-any.whl (40 kB) 8gddl Jul 10, 20:35:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 kB 235.5 MB/s eta 0:00:00 8gddl Jul 10, 20:35:52 Downloading outlines-0.0.34-py3-none-any.whl (76 kB) 8gddl Jul 10, 20:35:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.5/76.5 kB 259.9 MB/s eta 0:00:00 8gddl Jul 10, 20:35:52 Downloading tiktoken-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB) 8gddl Jul 10, 20:35:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 346.4 MB/s eta 0:00:00 8gddl Jul 10, 20:35:52 Downloading torch-2.2.1-cp310-cp310-manylinux1_x86_64.whl (755.5 MB) 8gddl Jul 10, 20:35:55 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 755.5/755.5 MB 272.8 MB/s eta 0:00:00 8gddl Jul 10, 20:35:55 Downloading xformers-0.0.25-cp310-cp310-manylinux2014_x86_64.whl (222.5 MB) 8gddl Jul 10, 20:35:56 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.5/222.5 MB 266.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:56 Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB) 8gddl Jul 10, 20:35:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 270.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:58 Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB) 8gddl Jul 10, 20:35:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 273.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:58 Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB) 8gddl Jul 10, 20:35:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 148.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:58 Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB) 8gddl Jul 10, 20:35:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 344.3 MB/s eta 0:00:00 8gddl Jul 10, 20:35:58 Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB) 8gddl Jul 10, 20:36:01 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 257.7 MB/s eta 0:00:00 8gddl Jul 10, 20:36:01 Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB) 8gddl Jul 10, 20:36:01 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 274.5 MB/s eta 0:00:00 8gddl Jul 10, 20:36:01 Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB) 8gddl Jul 10, 20:36:01 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 252.9 MB/s eta 0:00:00 8gddl Jul 10, 20:36:01 Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB) 8gddl Jul 10, 20:36:02 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 257.0 MB/s eta 0:00:00 8gddl Jul 10, 20:36:02 Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB) 8gddl Jul 10, 20:36:03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 275.6 MB/s eta 0:00:00 8gddl Jul 10, 20:36:03 Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB) 8gddl Jul 10, 20:36:03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.0/166.0 MB 311.6 MB/s eta 0:00:00 8gddl Jul 10, 20:36:03 Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB) 8gddl Jul 10, 20:36:03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 270.4 MB/s eta 0:00:00 8gddl Jul 10, 20:36:03 Downloading triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (167.9 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.9/167.9 MB 313.0 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading starlette-0.36.3-py3-none-any.whl (71 kB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.5/71.5 kB 197.7 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 296.2 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading transformers-4.42.3-py3-none-any.whl (9.3 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 242.3 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading nvidia_ml_py-12.555.43-py3-none-any.whl (39 kB) 8gddl Jul 10, 20:36:04 Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 340.5 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.4/341.4 kB 209.1 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading interegular-0.3.3-py37-none-any.whl (23 kB) 8gddl Jul 10, 20:36:04 Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 293.2 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 254.2 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.2/130.2 kB 292.7 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading diskcache-5.6.3-py3-none-any.whl (45 kB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 214.0 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading lark-1.1.9-py3-none-any.whl (111 kB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.7/111.7 kB 186.1 MB/s eta 0:00:00 8gddl Jul 10, 20:36:04 Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl (21.3 MB) 8gddl Jul 10, 20:36:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 325.0 MB/s eta 0:00:00 8gddl Jul 10, 20:36:14 Building wheels for collected packages: vllm-nccl-cu12 8gddl Jul 10, 20:36:15 Building wheel for vllm-nccl-cu12 (setup.py) ... done 8gddl Jul 10, 20:36:15 Created wheel for vllm-nccl-cu12: filename=vllm_nccl_cu12-2.18.1.0.4.0-py3-none-any.whl size=5451 sha256=e22657b55f367f5020576b66356a7cfdb1b0243670712172850438295359670e 8gddl Jul 10, 20:36:15 Stored in directory: /tmp/pip-ephem-wheel-cache-9yzaklf6/wheels/d1/28/b5/e99e6ea84b08c0bf19a218d408316e55e02ff725d3616fb79d 8gddl Jul 10, 20:36:15 Successfully built vllm-nccl-cu12 8gddl Jul 10, 20:36:16 Installing collected packages: vllm-nccl-cu12, sentencepiece, nvidia-ml-py, websockets, uvloop, triton, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lark, interegular, httptools, diskcache, watchfiles, tiktoken, starlette, nvidia-cusparse-cu12, nvidia-cudnn-cu12, tokenizers, nvidia-cusolver-cu12, lm-format-enforcer, fastapi, transformers, torch, xformers, outlines, vllm 8gddl Jul 10, 20:36:16 Attempting uninstall: triton 8gddl Jul 10, 20:36:16 Found existing installation: triton 2.1.0+e621604 8gddl Jul 10, 20:36:16 Uninstalling triton-2.1.0+e621604: 8gddl Jul 10, 20:36:18 Successfully uninstalled triton-2.1.0+e621604 8gddl Jul 10, 20:36:32 Attempting uninstall: starlette 8gddl Jul 10, 20:36:32 Found existing installation: starlette 0.27.0 8gddl Jul 10, 20:36:32 Uninstalling starlette-0.27.0: 8gddl Jul 10, 20:36:32 Successfully uninstalled starlette-0.27.0 8gddl Jul 10, 20:36:44 Attempting uninstall: tokenizers 8gddl Jul 10, 20:36:44 Found existing installation: tokenizers 0.15.2 8gddl Jul 10, 20:36:44 Uninstalling tokenizers-0.15.2: 8gddl Jul 10, 20:36:44 Successfully uninstalled tokenizers-0.15.2 8gddl Jul 10, 20:36:46 Attempting uninstall: fastapi 8gddl Jul 10, 20:36:46 Found existing installation: fastapi 0.104.1 8gddl Jul 10, 20:36:46 Uninstalling fastapi-0.104.1: 8gddl Jul 10, 20:36:46 Successfully uninstalled fastapi-0.104.1 8gddl Jul 10, 20:36:46 Attempting uninstall: transformers 8gddl Jul 10, 20:36:46 Found existing installation: transformers 4.38.2 8gddl Jul 10, 20:36:46 Uninstalling transformers-4.38.2: 8gddl Jul 10, 20:36:46 Successfully uninstalled transformers-4.38.2 8gddl Jul 10, 20:36:52 Attempting uninstall: torch 8gddl Jul 10, 20:36:52 Found existing installation: torch 2.1.0a0+32f93b1 8gddl Jul 10, 20:36:52 Uninstalling torch-2.1.0a0+32f93b1: 8gddl Jul 10, 20:36:56 Successfully uninstalled torch-2.1.0a0+32f93b1 8gddl Jul 10, 20:37:20 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 8gddl Jul 10, 20:37:20 autoawq 0.2.4 requires transformers<=4.38.2,>=4.35.0, but you have transformers 4.42.3 which is incompatible. 8gddl Jul 10, 20:37:20 torchdata 0.7.0a0 requires torch==2.1.0a0+32f93b1, but you have torch 2.2.1 which is incompatible. 8gddl Jul 10, 20:37:20 torchtext 0.16.0a0 requires torch==2.1.0a0+32f93b1, but you have torch 2.2.1 which is incompatible. 8gddl Jul 10, 20:37:20 torchvision 0.16.0a0 requires torch==2.1.0a0+32f93b1, but you have torch 2.2.1 which is incompatible. 8gddl Jul 10, 20:37:20 vessl 0.1.170 requires fastapi<0.105.0,>=0.104.0, but you have fastapi 0.109.2 which is incompatible. 8gddl Jul 10, 20:37:20 Successfully installed diskcache-5.6.3 fastapi-0.109.2 httptools-0.6.1 interegular-0.3.3 lark-1.1.9 lm-format-enforcer-0.9.8 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-ml-py-12.555.43 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.5.82 nvidia-nvtx-cu12-12.1.105 outlines-0.0.34 sentencepiece-0.2.0 starlette-0.36.3 tiktoken-0.6.0 tokenizers-0.19.1 torch-2.2.1 transformers-4.42.3 triton-2.2.0 uvloop-0.19.0 vllm-0.4.1 vllm-nccl-cu12-2.18.1.0.4.0 watchfiles-0.22.0 websockets-12.0 xformers-0.0.25 8gddl Jul 10, 20:37:20 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 8gddl Jul 10, 20:37:21 + pip uninstall -y transformer-engine 8gddl Jul 10, 20:37:21 Found existing installation: transformer-engine 0.12.0+170797 8gddl Jul 10, 20:37:21 Uninstalling transformer-engine-0.12.0+170797: 8gddl Jul 10, 20:37:23 Successfully uninstalled transformer-engine-0.12.0+170797 8gddl Jul 10, 20:37:23 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 8gddl Jul 10, 20:37:23 + pip install flash-attn==2.5.7 8gddl Jul 10, 20:37:24 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com 8gddl Jul 10, 20:37:24 Collecting flash-attn==2.5.7 8gddl Jul 10, 20:37:24 Downloading flash_attn-2.5.7.tar.gz (2.5 MB) 8gddl Jul 10, 20:37:24 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 61.4 MB/s eta 0:00:00 8gddl Jul 10, 20:37:28 Preparing metadata (setup.py) ... done 8gddl Jul 10, 20:37:28 Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.7) (2.2.1) 8gddl Jul 10, 20:37:28 Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.7) (0.7.0) 8gddl Jul 10, 20:37:28 Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.7) (24.1) 8gddl Jul 10, 20:37:28 Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.7) (1.11.1.1) 8gddl Jul 10, 20:37:28 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (3.12.4) 8gddl Jul 10, 20:37:28 Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (4.8.0) 8gddl Jul 10, 20:37:28 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (1.12) 8gddl Jul 10, 20:37:28 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (2.6.3) 8gddl Jul 10, 20:37:28 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (3.1.2) 8gddl Jul 10, 20:37:28 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (2023.6.0) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.105) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.105) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.105) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (8.9.2.26) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.3.1) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (11.0.2.54) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (10.3.2.106) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (11.4.5.107) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.0.106) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (2.19.3) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (12.1.105) 8gddl Jul 10, 20:37:28 Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.7) (2.2.0) 8gddl Jul 10, 20:37:28 Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->flash-attn==2.5.7) (12.5.82) 8gddl Jul 10, 20:37:28 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->flash-attn==2.5.7) (2.1.3) 8gddl Jul 10, 20:37:28 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->flash-attn==2.5.7) (1.3.0) 8gddl Jul 10, 20:37:28 Building wheels for collected packages: flash-attn 8gddl Jul 10, 20:37:31 Building wheel for flash-attn (setup.py) ... done 8gddl Jul 10, 20:37:32 Created wheel for flash-attn: filename=flash_attn-2.5.7-cp310-cp310-linux_x86_64.whl size=120853563 sha256=bbe6f77fd0899f8a125a5bdcf734b660c4c88e81c9b51c7ce98ebeba44dc6fa0 8gddl Jul 10, 20:37:32 Stored in directory: /tmp/pip-ephem-wheel-cache-fqm52w8c/wheels/13/96/ed/bcac89c56b606421f99b45b16a94db5d0f2b6b4eaf8bac4d01 8gddl Jul 10, 20:37:32 Successfully built flash-attn 8gddl Jul 10, 20:37:32 Installing collected packages: flash-attn 8gddl Jul 10, 20:37:32 Attempting uninstall: flash-attn 8gddl Jul 10, 20:37:32 Found existing installation: flash-attn 2.0.4 8gddl Jul 10, 20:37:32 Uninstalling flash-attn-2.0.4: 8gddl Jul 10, 20:37:35 Successfully uninstalled flash-attn-2.0.4 8gddl Jul 10, 20:37:38 Successfully installed flash-attn-2.5.7 8gddl Jul 10, 20:37:38 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 8gddl Jul 10, 20:37:38 + python -m vllm.entrypoints.openai.api_server --model casperhansen/llama-3-8b-instruct-awq 8gddl Jul 10, 20:37:42 INFO 07-10 11:37:42 api_server.py:151] vLLM API server version 0.4.1 8gddl Jul 10, 20:37:42 INFO 07-10 11:37:42 api_server.py:152] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, served_model_name=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='casperhansen/llama-3-8b-instruct-awq', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=5, disable_log_stats=False, quantization=None, enforce_eager=False, max_context_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', max_cpu_loras=None, device='auto', image_input_type=None, image_token_id=None, image_input_shape=None, image_feature_size=None, scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_max_model_len=None, model_loader_extra_config=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None) 8gddl Jul 10, 20:37:42 config.json: 100% 975/975 [00:00<00:00, 8.15MB/s] 8gddl Jul 10, 20:37:42 WARNING 07-10 11:37:42 config.py:169] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models. 8gddl Jul 10, 20:37:42 INFO 07-10 11:37:42 llm_engine.py:98] Initializing an LLM engine (v0.4.1) with config: model='casperhansen/llama-3-8b-instruct-awq', speculative_config=None, tokenizer='casperhansen/llama-3-8b-instruct-awq', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0) 8gddl Jul 10, 20:37:42 tokenizer_config.json: 100% 50.9k/50.9k [00:00<00:00, 833kB/s] 8gddl Jul 10, 20:37:43 tokenizer.json: 100% 9.08M/9.08M [00:00<00:00, 29.1MB/s] 8gddl Jul 10, 20:37:43 special_tokens_map.json: 100% 296/296 [00:00<00:00, 1.62MB/s] 8gddl Jul 10, 20:37:43 Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 8gddl Jul 10, 20:37:43 generation_config.json: 100% 152/152 [00:00<00:00, 1.12MB/s] 8gddl Jul 10, 20:37:43 INFO 07-10 11:37:43 utils.py:608] Found nccl from library /root/.config/vllm/nccl/cu12/libnccl.so.2.18.1 8gddl Jul 10, 20:37:44 INFO 07-10 11:37:44 selector.py:28] Using FlashAttention backend. 8gddl Jul 10, 20:37:45 INFO 07-10 11:37:45 weight_utils.py:193] Using model weights format ['*.safetensors'] 8gddl Jul 10, 20:37:45 model-00001-of-00002.safetensors: 0% 0.00/4.68G [00:00' + message['role'] + '<|end_header_id|> 8gddl Jul 10, 20:38:15 INFO 07-10 11:38:15 serving_chat.py:344] 8gddl Jul 10, 20:38:15 INFO 07-10 11:38:15 serving_chat.py:344] '+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|> 8gddl Jul 10, 20:38:15 INFO 07-10 11:38:15 serving_chat.py:344] 8gddl Jul 10, 20:38:15 INFO 07-10 11:38:15 serving_chat.py:344] ' }} 8gddl Jul 10, 20:38:15 Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 8gddl Jul 10, 20:38:15 INFO: Started server process [417] 8gddl Jul 10, 20:38:15 INFO: Waiting for application startup. 8gddl Jul 10, 20:38:15 INFO: Application startup complete. 8gddl Jul 10, 20:38:15 INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) 8gddl Jul 10, 20:38:17 INFO: 10.162.141.10:44518 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:38:17 INFO: 10.162.141.10:44518 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:38:25 INFO 07-10 11:38:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:38:35 INFO 07-10 11:38:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:38:45 INFO 07-10 11:38:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:38:55 INFO 07-10 11:38:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:05 INFO 07-10 11:39:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:15 INFO 07-10 11:39:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:17 INFO: 10.162.141.10:48106 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:39:17 INFO: 10.162.141.10:48106 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:39:25 INFO 07-10 11:39:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:35 INFO 07-10 11:39:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:45 INFO 07-10 11:39:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:39:55 INFO 07-10 11:39:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:05 INFO 07-10 11:40:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:15 INFO 07-10 11:40:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:17 INFO: 10.162.141.10:34690 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:40:17 INFO: 10.162.141.10:34690 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:40:25 INFO 07-10 11:40:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:35 INFO 07-10 11:40:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:45 INFO 07-10 11:40:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:40:55 INFO 07-10 11:40:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:05 INFO 07-10 11:41:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:15 INFO 07-10 11:41:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:17 INFO: 10.162.141.10:57506 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:41:17 INFO: 10.162.141.10:57506 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:41:25 INFO 07-10 11:41:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:35 INFO 07-10 11:41:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:45 INFO 07-10 11:41:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:41:55 INFO 07-10 11:41:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:05 INFO 07-10 11:42:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:15 INFO 07-10 11:42:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:17 INFO: 10.162.141.10:58704 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:42:17 INFO: 10.162.141.10:58704 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:42:25 INFO 07-10 11:42:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:35 INFO 07-10 11:42:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:45 INFO 07-10 11:42:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:42:55 INFO 07-10 11:42:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:05 INFO 07-10 11:43:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:15 INFO 07-10 11:43:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:17 INFO: 10.162.141.10:56348 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:43:17 INFO: 10.162.141.10:56348 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:43:25 INFO 07-10 11:43:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:35 INFO 07-10 11:43:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:45 INFO 07-10 11:43:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:43:55 INFO 07-10 11:43:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:05 INFO 07-10 11:44:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:15 INFO 07-10 11:44:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:17 INFO: 10.162.141.10:46700 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:44:17 INFO: 10.162.141.10:46700 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:44:25 INFO 07-10 11:44:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:35 INFO 07-10 11:44:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:45 INFO 07-10 11:44:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:44:55 INFO 07-10 11:44:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:05 INFO 07-10 11:45:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:15 INFO 07-10 11:45:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:17 INFO: 10.162.141.10:43042 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:45:17 INFO: 10.162.141.10:43042 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:45:25 INFO 07-10 11:45:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:35 INFO 07-10 11:45:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:45 INFO 07-10 11:45:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:45:55 INFO 07-10 11:45:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:05 INFO 07-10 11:46:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:15 INFO 07-10 11:46:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:17 INFO: 10.162.141.10:48104 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:46:17 INFO: 10.162.141.10:48104 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:46:25 INFO 07-10 11:46:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:35 INFO 07-10 11:46:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:45 INFO 07-10 11:46:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:46:55 INFO 07-10 11:46:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:05 INFO 07-10 11:47:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:15 INFO 07-10 11:47:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:17 INFO: 10.162.141.10:54046 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:47:17 INFO: 10.162.141.10:54046 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:47:25 INFO 07-10 11:47:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:35 INFO 07-10 11:47:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:45 INFO 07-10 11:47:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:47:55 INFO 07-10 11:47:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:05 INFO 07-10 11:48:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:15 INFO 07-10 11:48:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:17 INFO: 10.162.141.10:35948 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:48:17 INFO: 10.162.141.10:35948 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:48:25 INFO 07-10 11:48:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:35 INFO 07-10 11:48:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:45 INFO 07-10 11:48:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:48:55 INFO 07-10 11:48:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:05 INFO 07-10 11:49:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:15 INFO 07-10 11:49:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:17 INFO: 10.162.141.10:54306 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:49:17 INFO: 10.162.141.10:54306 - "GET /metrics/ HTTP/1.1" 200 OK 8gddl Jul 10, 20:49:25 INFO 07-10 11:49:25 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:35 INFO 07-10 11:49:35 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:45 INFO 07-10 11:49:45 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:49:55 INFO 07-10 11:49:55 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:50:05 INFO 07-10 11:50:05 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:50:15 INFO 07-10 11:50:15 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0% 8gddl Jul 10, 20:50:17 INFO: 10.162.141.10:57632 - "GET /metrics HTTP/1.1" 307 Temporary Redirect 8gddl Jul 10, 20:50:17 INFO: 10.162.141.10:57632 - "GET /metrics/ HTTP/1.1" 200 OK

VESSL Hub to VESSL Serve