Jul 10, 19:53:11 workload awaiting to start Jul 10, 19:53:11 Workload status changed to pending Jul 10, 19:53:16 0/22 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 19 node(s) didn't match Pod's node affinity/selector, 2 Insufficient cpu, 2 Insufficient ephemeral-storage, 2 Insufficient memory, 2 Insufficient nvidia.com/gpu. preemption: 0/22 nodes are available: 2 No preemption victims found for incoming pod, 20 Preemption is not helpful for scheduling.. Jul 10, 19:53:40 0/22 nodes are available: 19 node(s) didn't match Pod's node affinity/selector, 2 Insufficient ephemeral-storage, 2 Insufficient memory, 3 Insufficient cpu, 3 Insufficient nvidia.com/gpu. preemption: 0/22 nodes are available: 1 Insufficient nvidia.com/gpu, 19 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.. Jul 10, 19:55:13 0/23 nodes are available: 19 node(s) didn't match Pod's node affinity/selector, 2 Insufficient ephemeral-storage, 2 Insufficient memory, 3 Insufficient cpu, 4 Insufficient nvidia.com/gpu. preemption: 0/23 nodes are available: 1 Insufficient nvidia.com/gpu, 19 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.. Jul 10, 19:58:28 0/23 nodes are available: 19 node(s) didn't match Pod's node affinity/selector, 2 Insufficient ephemeral-storage, 2 Insufficient memory, 3 Insufficient cpu, 3 Insufficient nvidia.com/gpu. Jul 10, 19:58:38 0/23 nodes are available: 19 node(s) didn't match Pod's node affinity/selector, 2 Insufficient ephemeral-storage, 2 Insufficient memory, 2 Insufficient nvidia.com/gpu, 4 Insufficient cpu. preemption: not eligible due to a terminating pod on the nominated node.. Jul 10, 19:58:48 Worker assigned successfully. Starting soon. Jul 10, 19:58:48 Successfully assigned pezghftdj35o/run-execution-ju6uc69ws77n-0 to gke-managed-cluster--nvidia-l4-1-spot-5b50b8c9-pmw6 Jul 10, 19:58:49 Pulling image "quay.io/vessl-ai/torch:2.3.0-cuda12.4-r4" Jul 10, 20:02:54 Successfully pulled image "quay.io/vessl-ai/torch:2.3.0-cuda12.4-r4" in 4m5.719353678s (4m5.7193744s including waiting) Jul 10, 20:02:55 Started container run-execution Jul 10, 20:02:55 Created container run-execution Jul 10, 20:02:55 Pulling image "quay.io/vessl-ai/sidecar:v0.0.1-rc3" Jul 10, 20:02:56 ++ which curl Jul 10, 20:02:56 + [[ /usr/bin/curl == '' ]] Jul 10, 20:02:57 ++ which git Jul 10, 20:02:57 + [[ /usr/bin/git == '' ]] Jul 10, 20:02:57 ++ which git-lfs Jul 10, 20:02:57 + [[ /usr/bin/git-lfs == '' ]] Jul 10, 20:02:57 Installing VESSL CLI... Jul 10, 20:03:00 Started container run-sidecar Jul 10, 20:03:00 Pulling image "quay.io/vessl-ai/fluent-bit:2.1.6" Jul 10, 20:03:00 Successfully pulled image "quay.io/vessl-ai/sidecar:v0.0.1-rc3" in 4.682520527s (4.682546823s including waiting) Jul 10, 20:03:00 Created container run-sidecar Jul 10, 20:03:03 Successfully pulled image "quay.io/vessl-ai/fluent-bit:2.1.6" in 3.287599479s (3.287627519s including waiting) Jul 10, 20:03:03 Created container fluentbit Jul 10, 20:03:03 workload reported alive as pod run-execution-ju6uc69ws77n-0 Jul 10, 20:03:03 Workload status changed to initializing Jul 10, 20:03:04 Pulling image "quay.io/vessl-ai/kernels-toolbox:0.1.140" Jul 10, 20:03:04 Started container fluentbit Jul 10, 20:03:12 vessl, version 0.1.170 Jul 10, 20:03:16 Created container ttyd-sidecar Jul 10, 20:03:16 Successfully pulled image "quay.io/vessl-ai/kernels-toolbox:0.1.140" in 12.672604464s (12.672623106s including waiting) Jul 10, 20:03:16 Started container ttyd-sidecar Jul 10, 20:03:16 hint: Using 'master' as the name for the initial branch. This default branch name Jul 10, 20:03:16 hint: is subject to change. To configure the initial branch name to use in all Jul 10, 20:03:16 hint: of your new repositories, which will suppress this warning, call: Jul 10, 20:03:16 hint: Jul 10, 20:03:16 hint: git config --global init.defaultBranch Jul 10, 20:03:16 hint: Jul 10, 20:03:16 hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and Jul 10, 20:03:16 hint: 'development'. The just-created branch can be renamed via this command: Jul 10, 20:03:16 hint: Jul 10, 20:03:16 hint: git branch -m Jul 10, 20:03:16 Initialized empty Git repository in /mnt/vessl/ju6uc69ws77n-emptydir-workdir-jdmrp/pl7ieup4bi1d/root/code/.git/ Jul 10, 20:03:17 remote: Enumerating objects: 5279, done. Jul 10, 20:03:17 remote: Counting objects: 100% (689/689), done. Jul 10, 20:03:17 remote: Compressing objects: 100% (372/372), done. Jul 10, 20:03:23 remote: Total 5279 (delta 382), reused 495 (delta 307), pack-reused 4590 Jul 10, 20:03:23 Receiving objects: 100% (5279/5279), 215.86 MiB | 35.82 MiB/s, done. Jul 10, 20:03:24 Resolving deltas: 100% (2428/2428), done. Jul 10, 20:03:24 From https://github.com/vessl-ai/examples Jul 10, 20:03:24 * [new branch] Mnist_streamlit -> origin/Mnist_streamlit Jul 10, 20:03:24 * [new branch] cifar-resnet -> origin/cifar-resnet Jul 10, 20:03:24 * [new branch] expo-2024 -> origin/expo-2024 Jul 10, 20:03:24 * [new branch] finetuning-llm -> origin/finetuning-llm Jul 10, 20:03:24 * [new branch] instant-style-bentoml -> origin/instant-style-bentoml Jul 10, 20:03:24 * [new branch] llm-portal -> origin/llm-portal Jul 10, 20:03:24 * [new branch] main -> origin/main Jul 10, 20:03:24 * [new branch] seokju -> origin/seokju Jul 10, 20:03:24 * [new branch] vssl-8001 -> origin/vssl-8001 Jul 10, 20:03:24 * [new branch] vssl-8196 -> origin/vssl-8196 Jul 10, 20:03:24 * [new branch] vssl-8368 -> origin/vssl-8368 Jul 10, 20:03:24 * [new branch] vssl-8425 -> origin/vssl-8425 Jul 10, 20:03:24 * [new branch] vssl-8476 -> origin/vssl-8476 Jul 10, 20:03:24 * [new branch] vssl-8554 -> origin/vssl-8554 Jul 10, 20:03:24 * [new branch] vssl-9004 -> origin/vssl-9004 Jul 10, 20:03:24 * [new branch] vssl-9344 -> origin/vssl-9344 Jul 10, 20:03:27 Updating files: 100% (1403/1403), done. Jul 10, 20:03:27 HEAD is now at 494bc3c Pipeline examples (#49) Jul 10, 20:03:28 [74G[ OK ]g OpenBSD Secure Shell server sshd [80G Jul 10, 20:03:28 workload reported alive as pod run-execution-ju6uc69ws77n-0 Jul 10, 20:03:28 Workload status changed to running Jul 10, 20:03:28 + mkdir -p /root/code/finetune-llms Jul 10, 20:03:28 + cd /root/code/finetune-llms Jul 10, 20:03:28 + pip uninstall -y transformer-engine flash-attn torch Jul 10, 20:03:29 Found existing installation: transformer-engine 1.4.0+0fbc76a Jul 10, 20:03:29 Uninstalling transformer-engine-1.4.0+0fbc76a: Jul 10, 20:03:31 Successfully uninstalled transformer-engine-1.4.0+0fbc76a Jul 10, 20:03:31 Found existing installation: flash-attn 2.4.2 Jul 10, 20:03:31 Uninstalling flash-attn-2.4.2: Jul 10, 20:03:33 Successfully uninstalled flash-attn-2.4.2 Jul 10, 20:03:33 Found existing installation: torch 2.3.0a0+40ec155e58.nv24.3 Jul 10, 20:03:33 Uninstalling torch-2.3.0a0+40ec155e58.nv24.3: Jul 10, 20:03:37 Successfully uninstalled torch-2.3.0a0+40ec155e58.nv24.3 Jul 10, 20:03:40 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Jul 10, 20:03:40 + pip install -r requirements.txt Jul 10, 20:03:41 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Jul 10, 20:03:41 Collecting transformers>=4.40.2 (from -r requirements.txt (line 1)) Jul 10, 20:03:41 Downloading transformers-4.42.3-py3-none-any.whl.metadata (43 kB) Jul 10, 20:03:41 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.6/43.6 kB 4.9 MB/s eta 0:00:00 Jul 10, 20:03:42 Collecting datasets>=0.29.0 (from -r requirements.txt (line 2)) Jul 10, 20:03:42 Downloading datasets-2.20.0-py3-none-any.whl.metadata (19 kB) Jul 10, 20:03:42 Collecting trl>=0.8.6 (from -r requirements.txt (line 3)) Jul 10, 20:03:42 Downloading trl-0.9.6-py3-none-any.whl.metadata (12 kB) Jul 10, 20:03:42 Collecting peft>=0.10.0 (from -r requirements.txt (line 4)) Jul 10, 20:03:42 Downloading peft-0.11.1-py3-none-any.whl.metadata (13 kB) Jul 10, 20:03:42 Collecting accelerate>=0.30.0 (from -r requirements.txt (line 5)) Jul 10, 20:03:42 Downloading accelerate-0.32.1-py3-none-any.whl.metadata (18 kB) Jul 10, 20:03:42 Collecting bitsandbytes>=0.43.0 (from -r requirements.txt (line 6)) Jul 10, 20:03:42 Downloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl.metadata (2.2 kB) Jul 10, 20:03:42 Requirement already satisfied: vessl>=0.1.164 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 7)) (0.1.170) Jul 10, 20:03:42 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (3.13.1) Jul 10, 20:03:42 Collecting huggingface-hub<1.0,>=0.23.2 (from transformers>=4.40.2->-r requirements.txt (line 1)) Jul 10, 20:03:42 Downloading huggingface_hub-0.23.4-py3-none-any.whl.metadata (12 kB) Jul 10, 20:03:42 Requirement already satisfied: numpy<2.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (1.24.4) Jul 10, 20:03:42 Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (24.1) Jul 10, 20:03:42 Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (6.0.1) Jul 10, 20:03:42 Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (2023.12.25) Jul 10, 20:03:42 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (2.31.0) Jul 10, 20:03:43 Collecting safetensors>=0.4.1 (from transformers>=4.40.2->-r requirements.txt (line 1)) Jul 10, 20:03:43 Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB) Jul 10, 20:03:43 Collecting tokenizers<0.20,>=0.19 (from transformers>=4.40.2->-r requirements.txt (line 1)) Jul 10, 20:03:43 Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) Jul 10, 20:03:43 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.40.2->-r requirements.txt (line 1)) (4.66.2) Jul 10, 20:03:43 Collecting pyarrow>=15.0.0 (from datasets>=0.29.0->-r requirements.txt (line 2)) Jul 10, 20:03:43 Downloading pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB) Jul 10, 20:03:43 Collecting pyarrow-hotfix (from datasets>=0.29.0->-r requirements.txt (line 2)) Jul 10, 20:03:43 Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB) Jul 10, 20:03:43 Collecting dill<0.3.9,>=0.3.0 (from datasets>=0.29.0->-r requirements.txt (line 2)) Jul 10, 20:03:43 Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB) Jul 10, 20:03:43 Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=0.29.0->-r requirements.txt (line 2)) (1.5.3) Jul 10, 20:03:44 Collecting requests (from transformers>=4.40.2->-r requirements.txt (line 1)) Jul 10, 20:03:44 Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Jul 10, 20:03:44 Collecting tqdm>=4.27 (from transformers>=4.40.2->-r requirements.txt (line 1)) Jul 10, 20:03:44 Downloading tqdm-4.66.4-py3-none-any.whl.metadata (57 kB) Jul 10, 20:03:44 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 147.9 MB/s eta 0:00:00 Jul 10, 20:03:44 Collecting xxhash (from datasets>=0.29.0->-r requirements.txt (line 2)) Jul 10, 20:03:44 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) Jul 10, 20:03:44 Collecting multiprocess (from datasets>=0.29.0->-r requirements.txt (line 2)) Jul 10, 20:03:44 Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB) Jul 10, 20:03:44 Requirement already satisfied: fsspec<=2024.5.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.5.0,>=2023.1.0->datasets>=0.29.0->-r requirements.txt (line 2)) (2024.2.0) Jul 10, 20:03:44 Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=0.29.0->-r requirements.txt (line 2)) (3.9.3) Jul 10, 20:03:44 Collecting torch>=1.4.0 (from trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:44 Downloading torch-2.3.1-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB) Jul 10, 20:03:45 Collecting tyro>=0.5.11 (from trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading tyro-0.8.5-py3-none-any.whl.metadata (8.2 kB) Jul 10, 20:03:45 Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft>=0.10.0->-r requirements.txt (line 4)) (5.9.4) Jul 10, 20:03:45 Requirement already satisfied: boto3>=1.18.9 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.34.84) Jul 10, 20:03:45 Requirement already satisfied: brotli<2.0.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.1.0) Jul 10, 20:03:45 Requirement already satisfied: click>=8.0.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (8.1.7) Jul 10, 20:03:45 Requirement already satisfied: cloudpickle<2.0.0,>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.6.0) Jul 10, 20:03:45 Requirement already satisfied: docker>=6.0.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (7.0.0) Jul 10, 20:03:45 Requirement already satisfied: fastapi<0.105.0,>=0.104.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.104.1) Jul 10, 20:03:45 Requirement already satisfied: google-auth<3.0.0,>=2.28.2 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (2.28.2) Jul 10, 20:03:45 Requirement already satisfied: google-cloud-storage<3.0.0,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (2.16.0) Jul 10, 20:03:45 Requirement already satisfied: halo<0.0.32,>=0.0.31 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.0.31) Jul 10, 20:03:45 Requirement already satisfied: humanfriendly<11.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (10.0) Jul 10, 20:03:45 Requirement already satisfied: inquirer>=3.1.3 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (3.2.4) Jul 10, 20:03:45 Requirement already satisfied: kubernetes<25.0.0,>=24.2.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (24.2.0) Jul 10, 20:03:45 Requirement already satisfied: paramiko<3.0.0,>=2.11.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (2.12.0) Jul 10, 20:03:45 Requirement already satisfied: prometheus-client<0.21.0,>=0.20.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.20.0) Jul 10, 20:03:45 Requirement already satisfied: pynvml<12.0.0,>=11.0.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (11.4.1) Jul 10, 20:03:45 Requirement already satisfied: python-dotenv<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.0.1) Jul 10, 20:03:45 Requirement already satisfied: requests-futures>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.0.1) Jul 10, 20:03:45 Requirement already satisfied: schema>=0.7.4 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.7.5) Jul 10, 20:03:45 Requirement already satisfied: sentry-sdk>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.45.0) Jul 10, 20:03:45 Requirement already satisfied: shortuuid>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.0.13) Jul 10, 20:03:45 Requirement already satisfied: six<2.0.0,>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.16.0) Jul 10, 20:03:45 Requirement already satisfied: sshpubkeys<4.0.0,>=3.3.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (3.3.1) Jul 10, 20:03:45 Requirement already satisfied: tabulate<0.10.0,>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.9.0) Jul 10, 20:03:45 Requirement already satisfied: terminaltables>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (3.1.10) Jul 10, 20:03:45 Requirement already satisfied: timeago<2.0.0,>=1.0.15 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (1.0.16) Jul 10, 20:03:45 Requirement already satisfied: toml<0.11.0,>=0.10.1 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.10.2) Jul 10, 20:03:45 Requirement already satisfied: uvicorn>=0.16.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (0.29.0) Jul 10, 20:03:45 Requirement already satisfied: watchdog<4.0.0,>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from vessl>=0.1.164->-r requirements.txt (line 7)) (3.0.0) Jul 10, 20:03:45 Requirement already satisfied: botocore<1.35.0,>=1.34.84 in /usr/local/lib/python3.10/dist-packages (from boto3>=1.18.9->vessl>=0.1.164->-r requirements.txt (line 7)) (1.34.84) Jul 10, 20:03:45 Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from boto3>=1.18.9->vessl>=0.1.164->-r requirements.txt (line 7)) (1.0.1) Jul 10, 20:03:45 Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from boto3>=1.18.9->vessl>=0.1.164->-r requirements.txt (line 7)) (0.10.1) Jul 10, 20:03:45 Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.10/dist-packages (from docker>=6.0.1->vessl>=0.1.164->-r requirements.txt (line 7)) (1.26.18) Jul 10, 20:03:45 Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.10/dist-packages (from fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (3.7.1) Jul 10, 20:03:45 Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.6.3) Jul 10, 20:03:45 Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.10/dist-packages (from fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (0.27.0) Jul 10, 20:03:45 Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (4.10.0) Jul 10, 20:03:45 Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (1.3.1) Jul 10, 20:03:45 Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (23.2.0) Jul 10, 20:03:45 Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (1.4.1) Jul 10, 20:03:45 Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (6.0.5) Jul 10, 20:03:45 Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (1.9.4) Jul 10, 20:03:45 Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=0.29.0->-r requirements.txt (line 2)) (4.0.3) Jul 10, 20:03:45 Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.0,>=2.28.2->vessl>=0.1.164->-r requirements.txt (line 7)) (5.3.3) Jul 10, 20:03:45 Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.0,>=2.28.2->vessl>=0.1.164->-r requirements.txt (line 7)) (0.3.0) Jul 10, 20:03:45 Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.0,>=2.28.2->vessl>=0.1.164->-r requirements.txt (line 7)) (4.9) Jul 10, 20:03:45 Requirement already satisfied: google-api-core<3.0.0dev,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.18.0) Jul 10, 20:03:45 Requirement already satisfied: google-cloud-core<3.0dev,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.4.1) Jul 10, 20:03:45 Requirement already satisfied: google-resumable-media>=2.6.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.7.0) Jul 10, 20:03:45 Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.5.0) Jul 10, 20:03:45 Requirement already satisfied: log-symbols>=0.0.14 in /usr/local/lib/python3.10/dist-packages (from halo<0.0.32,>=0.0.31->vessl>=0.1.164->-r requirements.txt (line 7)) (0.0.14) Jul 10, 20:03:45 Requirement already satisfied: spinners>=0.0.24 in /usr/local/lib/python3.10/dist-packages (from halo<0.0.32,>=0.0.31->vessl>=0.1.164->-r requirements.txt (line 7)) (0.0.24) Jul 10, 20:03:45 Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from halo<0.0.32,>=0.0.31->vessl>=0.1.164->-r requirements.txt (line 7)) (2.4.0) Jul 10, 20:03:45 Requirement already satisfied: colorama>=0.3.9 in /usr/local/lib/python3.10/dist-packages (from halo<0.0.32,>=0.0.31->vessl>=0.1.164->-r requirements.txt (line 7)) (0.4.6) Jul 10, 20:03:45 Requirement already satisfied: blessed>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (1.20.0) Jul 10, 20:03:45 Requirement already satisfied: editor>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (1.6.6) Jul 10, 20:03:45 Requirement already satisfied: readchar>=3.0.6 in /usr/local/lib/python3.10/dist-packages (from inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (4.0.6) Jul 10, 20:03:45 Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.10/dist-packages (from kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2024.2.2) Jul 10, 20:03:45 Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.10/dist-packages (from kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.9.0.post0) Jul 10, 20:03:45 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.10/dist-packages (from kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (68.2.2) Jul 10, 20:03:45 Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.10/dist-packages (from kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.7.0) Jul 10, 20:03:45 Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.10/dist-packages (from kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.3.1) Jul 10, 20:03:45 Requirement already satisfied: bcrypt>=3.1.3 in /usr/local/lib/python3.10/dist-packages (from paramiko<3.0.0,>=2.11.0->vessl>=0.1.164->-r requirements.txt (line 7)) (4.1.2) Jul 10, 20:03:45 Requirement already satisfied: cryptography>=2.5 in /usr/local/lib/python3.10/dist-packages (from paramiko<3.0.0,>=2.11.0->vessl>=0.1.164->-r requirements.txt (line 7)) (42.0.5) Jul 10, 20:03:45 Requirement already satisfied: pynacl>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from paramiko<3.0.0,>=2.11.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.5.0) Jul 10, 20:03:45 Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.40.2->-r requirements.txt (line 1)) (3.3.2) Jul 10, 20:03:45 Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.40.2->-r requirements.txt (line 1)) (3.6) Jul 10, 20:03:45 Requirement already satisfied: contextlib2>=0.5.5 in /usr/local/lib/python3.10/dist-packages (from schema>=0.7.4->vessl>=0.1.164->-r requirements.txt (line 7)) (21.6.0) Jul 10, 20:03:45 Requirement already satisfied: ecdsa>=0.13 in /usr/local/lib/python3.10/dist-packages (from sshpubkeys<4.0.0,>=3.3.1->vessl>=0.1.164->-r requirements.txt (line 7)) (0.19.0) Jul 10, 20:03:45 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) (1.12) Jul 10, 20:03:45 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) (2.6.3) Jul 10, 20:03:45 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) (3.1.3) Jul 10, 20:03:45 Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:45 Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:45 Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Jul 10, 20:03:45 Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Jul 10, 20:03:45 Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:45 Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:46 Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:46 Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:46 Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Jul 10, 20:03:46 Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Jul 10, 20:03:46 Collecting nvidia-nccl-cu12==2.20.5 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB) Jul 10, 20:03:46 Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB) Jul 10, 20:03:46 Collecting triton==2.3.1 (from torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading triton-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB) Jul 10, 20:03:46 Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:46 Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB) Jul 10, 20:03:47 Collecting docstring-parser>=0.16 (from tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:47 Downloading docstring_parser-0.16-py3-none-any.whl.metadata (3.0 kB) Jul 10, 20:03:47 Requirement already satisfied: rich>=11.1.0 in /usr/local/lib/python3.10/dist-packages (from tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) (13.7.1) Jul 10, 20:03:47 Collecting shtab>=1.5.6 (from tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) Jul 10, 20:03:47 Downloading shtab-1.7.1-py3-none-any.whl.metadata (7.3 kB) Jul 10, 20:03:47 Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.16.0->vessl>=0.1.164->-r requirements.txt (line 7)) (0.14.0) Jul 10, 20:03:47 Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=0.29.0->-r requirements.txt (line 2)) (2024.1) Jul 10, 20:03:47 Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.3.1) Jul 10, 20:03:47 Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.2.0) Jul 10, 20:03:47 Requirement already satisfied: wcwidth>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.19.0->inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (0.2.13) Jul 10, 20:03:47 Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/dist-packages (from cryptography>=2.5->paramiko<3.0.0,>=2.11.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.16.0) Jul 10, 20:03:47 Requirement already satisfied: runs in /usr/local/lib/python3.10/dist-packages (from editor>=1.6.0->inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (1.2.2) Jul 10, 20:03:47 Requirement already satisfied: xmod in /usr/local/lib/python3.10/dist-packages (from editor>=1.6.0->inquirer>=3.1.3->vessl>=0.1.164->-r requirements.txt (line 7)) (1.8.1) Jul 10, 20:03:47 Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.63.0) Jul 10, 20:03:47 Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5 in /usr/local/lib/python3.10/dist-packages (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (4.24.4) Jul 10, 20:03:47 Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/dist-packages (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage<3.0.0,>=2.15.0->vessl>=0.1.164->-r requirements.txt (line 7)) (1.23.0) Jul 10, 20:03:47 Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.0,>=2.28.2->vessl>=0.1.164->-r requirements.txt (line 7)) (0.5.1) Jul 10, 20:03:47 Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (0.6.0) Jul 10, 20:03:47 Requirement already satisfied: pydantic-core==2.16.3 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<0.105.0,>=0.104.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.16.3) Jul 10, 20:03:47 Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=11.1.0->tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) (3.0.0) Jul 10, 20:03:47 Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=11.1.0->tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) (2.17.2) Jul 10, 20:03:47 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) (2.1.5) Jul 10, 20:03:47 Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib->kubernetes<25.0.0,>=24.2.0->vessl>=0.1.164->-r requirements.txt (line 7)) (3.2.2) Jul 10, 20:03:47 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.4.0->trl>=0.8.6->-r requirements.txt (line 3)) (1.3.0) Jul 10, 20:03:47 Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12->cryptography>=2.5->paramiko<3.0.0,>=2.11.0->vessl>=0.1.164->-r requirements.txt (line 7)) (2.21) Jul 10, 20:03:47 Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=11.1.0->tyro>=0.5.11->trl>=0.8.6->-r requirements.txt (line 3)) (0.1.2) Jul 10, 20:03:47 Downloading transformers-4.42.3-py3-none-any.whl (9.3 MB) Jul 10, 20:03:47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 173.5 MB/s eta 0:00:00 Jul 10, 20:03:47 Downloading datasets-2.20.0-py3-none-any.whl (547 kB) Jul 10, 20:03:47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 310.0 MB/s eta 0:00:00 Jul 10, 20:03:47 Downloading trl-0.9.6-py3-none-any.whl (245 kB) Jul 10, 20:03:47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 245.8/245.8 kB 297.2 MB/s eta 0:00:00 Jul 10, 20:03:47 Downloading peft-0.11.1-py3-none-any.whl (251 kB) Jul 10, 20:03:47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 251.6/251.6 kB 297.9 MB/s eta 0:00:00 Jul 10, 20:03:47 Downloading accelerate-0.32.1-py3-none-any.whl (314 kB) Jul 10, 20:03:47 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 314.1/314.1 kB 313.0 MB/s eta 0:00:00 Jul 10, 20:03:47 Downloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl (119.8 MB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.8/119.8 MB 299.1 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading dill-0.3.8-py3-none-any.whl (116 kB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 270.1 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading huggingface_hub-0.23.4-py3-none-any.whl (402 kB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 402.6/402.6 kB 320.5 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading pyarrow-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 309.5 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading requests-2.32.3-py3-none-any.whl (64 kB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 238.6 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 329.3 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) Jul 10, 20:03:48 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 314.8 MB/s eta 0:00:00 Jul 10, 20:03:48 Downloading torch-2.3.1-cp310-cp310-manylinux1_x86_64.whl (779.1 MB) Jul 10, 20:03:51 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 779.1/779.1 MB 302.0 MB/s eta 0:00:00 Jul 10, 20:03:51 Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB) Jul 10, 20:03:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 305.2 MB/s eta 0:00:00 Jul 10, 20:03:52 Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB) Jul 10, 20:03:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 278.9 MB/s eta 0:00:00 Jul 10, 20:03:52 Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB) Jul 10, 20:03:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 293.3 MB/s eta 0:00:00 Jul 10, 20:03:52 Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB) Jul 10, 20:03:52 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 318.8 MB/s eta 0:00:00 Jul 10, 20:03:52 Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB) Jul 10, 20:03:55 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 285.4 MB/s eta 0:00:00 Jul 10, 20:03:55 Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB) Jul 10, 20:03:55 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 297.4 MB/s eta 0:00:00 Jul 10, 20:03:55 Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB) Jul 10, 20:03:56 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 295.6 MB/s eta 0:00:00 Jul 10, 20:03:56 Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB) Jul 10, 20:03:56 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 297.4 MB/s eta 0:00:00 Jul 10, 20:03:56 Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB) Jul 10, 20:03:57 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 272.8 MB/s eta 0:00:00 Jul 10, 20:03:57 Downloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.2/176.2 MB 271.1 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 270.3 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading triton-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (168.1 MB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.1/168.1 MB 299.4 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading tqdm-4.66.4-py3-none-any.whl (78 kB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 257.3 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading tyro-0.8.5-py3-none-any.whl (103 kB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.4/103.4 kB 266.8 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 265.8 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB) Jul 10, 20:03:58 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 301.0 MB/s eta 0:00:00 Jul 10, 20:03:58 Downloading docstring_parser-0.16-py3-none-any.whl (36 kB) Jul 10, 20:03:58 Downloading shtab-1.7.1-py3-none-any.whl (14 kB) Jul 10, 20:03:58 Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl (21.3 MB) Jul 10, 20:03:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 275.7 MB/s eta 0:00:00 Jul 10, 20:04:09 Installing collected packages: xxhash, triton, tqdm, shtab, safetensors, requests, pyarrow-hotfix, pyarrow, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, docstring-parser, dill, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, huggingface-hub, tyro, tokenizers, nvidia-cusolver-cu12, transformers, torch, datasets, bitsandbytes, accelerate, trl, peft Jul 10, 20:04:11 Attempting uninstall: tqdm Jul 10, 20:04:11 Found existing installation: tqdm 4.66.2 Jul 10, 20:04:11 Uninstalling tqdm-4.66.2: Jul 10, 20:04:11 Successfully uninstalled tqdm-4.66.2 Jul 10, 20:04:11 Attempting uninstall: requests Jul 10, 20:04:11 Found existing installation: requests 2.31.0 Jul 10, 20:04:11 Uninstalling requests-2.31.0: Jul 10, 20:04:11 Successfully uninstalled requests-2.31.0 Jul 10, 20:04:11 Attempting uninstall: pyarrow Jul 10, 20:04:11 Found existing installation: pyarrow 14.0.1 Jul 10, 20:04:11 Uninstalling pyarrow-14.0.1: Jul 10, 20:04:12 Successfully uninstalled pyarrow-14.0.1 Jul 10, 20:05:07 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. Jul 10, 20:05:07 torchtext 0.17.0a0 requires torch==2.3.0a0+40ec155e58, but you have torch 2.3.1 which is incompatible. Jul 10, 20:05:07 torchvision 0.18.0a0 requires torch==2.3.0a0+40ec155e58.nv24.03, but you have torch 2.3.1 which is incompatible. Jul 10, 20:05:07 Successfully installed accelerate-0.32.1 bitsandbytes-0.43.1 datasets-2.20.0 dill-0.3.8 docstring-parser-0.16 huggingface-hub-0.23.4 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.5.82 nvidia-nvtx-cu12-12.1.105 peft-0.11.1 pyarrow-16.1.0 pyarrow-hotfix-0.6 requests-2.32.3 safetensors-0.4.3 shtab-1.7.1 tokenizers-0.19.1 torch-2.3.1 tqdm-4.66.4 transformers-4.42.3 triton-2.3.1 trl-0.9.6 tyro-0.8.5 xxhash-3.4.1 Jul 10, 20:05:07 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Jul 10, 20:05:08 + pip install packaging ninja einops wheel xformers Jul 10, 20:05:09 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Jul 10, 20:05:09 Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (24.1) Jul 10, 20:05:09 Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (1.11.1.1) Jul 10, 20:05:09 Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (0.7.0) Jul 10, 20:05:09 Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (0.42.0) Jul 10, 20:05:09 Collecting xformers Jul 10, 20:05:09 Downloading xformers-0.0.27-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB) Jul 10, 20:05:09 Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from xformers) (1.24.4) Jul 10, 20:05:09 Requirement already satisfied: torch==2.3.1 in /usr/local/lib/python3.10/dist-packages (from xformers) (2.3.1) Jul 10, 20:05:09 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (3.13.1) Jul 10, 20:05:09 Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (4.10.0) Jul 10, 20:05:09 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (1.12) Jul 10, 20:05:09 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (2.6.3) Jul 10, 20:05:09 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (3.1.3) Jul 10, 20:05:09 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (2024.2.0) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.105) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.105) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.105) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (8.9.2.26) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.3.1) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (11.0.2.54) Jul 10, 20:05:09 Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (10.3.2.106) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (11.4.5.107) Jul 10, 20:05:09 Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.0.106) Jul 10, 20:05:09 Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (2.20.5) Jul 10, 20:05:09 Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (12.1.105) Jul 10, 20:05:09 Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->xformers) (2.3.1) Jul 10, 20:05:09 Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.1->xformers) (12.5.82) Jul 10, 20:05:09 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.3.1->xformers) (2.1.5) Jul 10, 20:05:09 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.3.1->xformers) (1.3.0) Jul 10, 20:05:09 Downloading xformers-0.0.27-cp310-cp310-manylinux2014_x86_64.whl (164.1 MB) Jul 10, 20:05:10 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.1/164.1 MB 310.2 MB/s eta 0:00:00 Jul 10, 20:05:11 Installing collected packages: xformers Jul 10, 20:05:14 Successfully installed xformers-0.0.27 Jul 10, 20:05:14 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Jul 10, 20:05:14 + pip install flash-attn==2.5.8 --no-build-isolation Jul 10, 20:05:15 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Jul 10, 20:05:15 Collecting flash-attn==2.5.8 Jul 10, 20:05:15 Downloading flash_attn-2.5.8.tar.gz (2.5 MB) Jul 10, 20:05:15 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 44.4 MB/s eta 0:00:00 Jul 10, 20:05:19 Preparing metadata (setup.py) ... done Jul 10, 20:05:19 Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.8) (2.3.1) Jul 10, 20:05:19 Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.8) (0.7.0) Jul 10, 20:05:19 Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.8) (24.1) Jul 10, 20:05:19 Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.8) (1.11.1.1) Jul 10, 20:05:19 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (3.13.1) Jul 10, 20:05:19 Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (4.10.0) Jul 10, 20:05:19 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (1.12) Jul 10, 20:05:19 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (2.6.3) Jul 10, 20:05:19 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (3.1.3) Jul 10, 20:05:19 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (2024.2.0) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.105) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.105) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.105) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (8.9.2.26) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.3.1) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (11.0.2.54) Jul 10, 20:05:19 Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (10.3.2.106) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (11.4.5.107) Jul 10, 20:05:19 Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.0.106) Jul 10, 20:05:19 Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (2.20.5) Jul 10, 20:05:19 Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (12.1.105) Jul 10, 20:05:19 Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.8) (2.3.1) Jul 10, 20:05:19 Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->flash-attn==2.5.8) (12.5.82) Jul 10, 20:05:20 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->flash-attn==2.5.8) (2.1.5) Jul 10, 20:05:20 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->flash-attn==2.5.8) (1.3.0) Jul 10, 20:05:20 Building wheels for collected packages: flash-attn Jul 10, 20:05:23 Building wheel for flash-attn (setup.py) ... done Jul 10, 20:05:24 Created wheel for flash-attn: filename=flash_attn-2.5.8-cp310-cp310-linux_x86_64.whl size=120922965 sha256=13454dd3d37cf173649bd389b84614b8072fb283f8f2fd23a65ab66caafc304b Jul 10, 20:05:24 Stored in directory: /tmp/pip-ephem-wheel-cache-d6emtel8/wheels/9b/5b/2b/dea8af4e954161c49ef1941938afcd91bb93689371ed12a226 Jul 10, 20:05:24 Successfully built flash-attn Jul 10, 20:05:25 Installing collected packages: flash-attn Jul 10, 20:05:27 Successfully installed flash-attn-2.5.8 Jul 10, 20:05:27 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Jul 10, 20:05:28 + python main.py --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct --dataset_name kyujinpy/OpenOrca-KO --output_dir outputs --max_seq_length 2048 --num_train_epochs 1 --logging_steps 5 --bf16 True --packing True --learning_rate 2e-4 --lr_scheduler_type cosine --weight_decay 1e-4 --warmup_ratio 0.0 --max_grad_norm 1.0 --per_device_train_batch_size 2 --per_device_eval_batch_size 2 --gradient_accumulation_steps 4 --gradient_checkpointing True --use_flash_attn True --upload_model True --repository_name Llama-3-ko-finetuned --peft_type LORA --task_type CAUSAL_LM --lora_r 8 --lora_alpha 16 --load_in_4bit True --bnb_4bit_quant_type nf4 --bnb_4bit_use_double_quant True --bnb_4bit_compute_dtype float16 Jul 10, 20:05:35 config.json: 100% 654/654 [00:00<00:00, 4.93MB/s] Jul 10, 20:05:36 model.safetensors.index.json: 100% 23.9k/23.9k [00:00<00:00, 87.4MB/s] Jul 10, 20:05:36 Downloading shards: 0% 0/4 [00:00

VESSL Hub to VESSL Run