openweb-prac.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"machine_shape":"hm","authorship_tag":"ABX9TyPNnmr7aNVnXmIv5Ao2bqG4"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU","gpuClass":"premium"},"cells":[{"cell_type":"code","execution_count":1,"metadata":{"id":"g770QvyqEMy-","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1682824120370,"user_tz":-540,"elapsed":3226,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"4d356838-295d-4cef-b08d-2dbfe4b4d18c"},"outputs":[{"output_type":"stream","name":"stdout","text":["Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"]}],"source":["from google.colab import drive\n","drive.mount('/content/drive')"]},{"cell_type":"code","source":["cd \"/content/drive/My Drive/\""],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-YIr86mLEg7w","executionInfo":{"status":"ok","timestamp":1682824122988,"user_tz":-540,"elapsed":3,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"6e4a49b2-eb64-44cd-e9af-93d783ed3226"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["/content/drive/My Drive\n"]}]},{"cell_type":"code","source":["!git clone https://github.com/karpathy/nanoGPT.git"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jZ63c-z-EyT-","executionInfo":{"status":"ok","timestamp":1682824129882,"user_tz":-540,"elapsed":512,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"c87d855e-af2c-4dd2-9330-46d7ea56c891"},"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["fatal: destination path 'nanoGPT' already exists and is not an empty directory.\n"]}]},{"cell_type":"code","source":["cd nanoGPT"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"x9SIw1jPFAlX","executionInfo":{"status":"ok","timestamp":1682824132055,"user_tz":-540,"elapsed":756,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"b5179591-c376-4f1b-8b1e-b7d3b7b367f4"},"execution_count":4,"outputs":[{"output_type":"stream","name":"stdout","text":["/content/drive/My Drive/nanoGPT\n"]}]},{"cell_type":"code","source":["pip install tiktoken"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mI8tjQ29s8dO","executionInfo":{"status":"ok","timestamp":1682824137879,"user_tz":-540,"elapsed":4413,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"6d85c95b-0e10-497e-a733-867760139293"},"execution_count":5,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (0.3.3)\n","Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2.27.1)\n","Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2022.10.31)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (1.26.15)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.4)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2022.12.7)\n"]}]},{"cell_type":"code","source":["pip install datasets"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"b-UH8BFXtN9N","executionInfo":{"status":"ok","timestamp":1682824201310,"user_tz":-540,"elapsed":4129,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"39aea0ac-331c-43a4-eb6d-a51de7e7cbdb"},"execution_count":6,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.12.0)\n","Requirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.14.1)\n","Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.4.0)\n","Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.2.0)\n","Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.8.4)\n","Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.65.0)\n","Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0)\n","Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.18.0)\n","Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.6)\n","Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (9.0.0)\n","Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.14)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.1)\n","Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.22.4)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3)\n","Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.27.1)\n","Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.2)\n","Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.4)\n","Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)\n","Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.2)\n","Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n","Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.0.12)\n","Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.3)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (4.5.0)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.12.0)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2022.12.7)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (3.4)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (1.26.15)\n","Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2022.7.1)\n","Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n","Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n"]}]},{"cell_type":"code","source":["!python data/openwebtext/prepare.py"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"71tJ7QsGFEJw","outputId":"f79bdd41-2665-4f49-d396-27b7ab3d680a","executionInfo":{"status":"ok","timestamp":1682828646477,"user_tz":-540,"elapsed":4441476,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}}},"execution_count":7,"outputs":[{"output_type":"stream","name":"stdout","text":["Found cached dataset openwebtext (/root/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/6f68e85c16ccc770c0dd489f4008852ea9633604995addd0cd76e293aed9e521)\n","100% 1/1 [00:11<00:00, 11.69s/it]\n","Loading cached split indices for dataset at /root/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/6f68e85c16ccc770c0dd489f4008852ea9633604995addd0cd76e293aed9e521/cache-20d0093e379632f2.arrow and /root/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/6f68e85c16ccc770c0dd489f4008852ea9633604995addd0cd76e293aed9e521/cache-e1e5af05f4aa07f1.arrow\n","Loading cached processed dataset at /root/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/6f68e85c16ccc770c0dd489f4008852ea9633604995addd0cd76e293aed9e521/cache-485a409bda7785b2_*_of_00008.arrow\n","Loading cached processed dataset at /root/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/6f68e85c16ccc770c0dd489f4008852ea9633604995addd0cd76e293aed9e521/cache-075eb64fd92d0537_*_of_00008.arrow\n","writing /content/drive/MyDrive/nanoGPT/data/openwebtext/train.bin...\n","100% 8009762/8009762 [1:13:34<00:00, 1814.55it/s]\n","writing /content/drive/MyDrive/nanoGPT/data/openwebtext/val.bin...\n","100% 4007/4007 [00:02<00:00, 1885.94it/s]\n"]}]},{"cell_type":"code","source":["pip install wandb"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"iyro28hFbzm4","executionInfo":{"status":"ok","timestamp":1682829187629,"user_tz":-540,"elapsed":12575,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"2264d018-acf1-4a2b-a5bb-16a9e0a30290"},"execution_count":9,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting wandb\n","  Downloading wandb-0.15.0-py3-none-any.whl (2.0 MB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m49.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting GitPython!=3.1.29,>=1.0.0\n","  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m25.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from wandb) (6.0)\n","Requirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (2.27.1)\n","Collecting docker-pycreds>=0.4.0\n","  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n","Collecting sentry-sdk>=1.0.0\n","  Downloading sentry_sdk-1.21.1-py2.py3-none-any.whl (201 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m201.7/201.7 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (3.20.3)\n","Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb) (1.4.4)\n","Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (5.9.5)\n","Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.10/dist-packages (from wandb) (8.1.3)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb) (67.7.2)\n","Collecting setproctitle\n","  Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n","Collecting pathtools\n","  Downloading pathtools-0.1.2.tar.gz (11 kB)\n","  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from docker-pycreds>=0.4.0->wandb) (1.16.0)\n","Collecting gitdb<5,>=4.0.1\n","  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (1.26.15)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (2.0.12)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (2022.12.7)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.0.0->wandb) (3.4)\n","Collecting smmap<6,>=3.0.1\n","  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n","Building wheels for collected packages: pathtools\n","  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8807 sha256=aeb919b957bcc9d95197776ba9b39ffaf38d7c1d73604e50078977f0d3a513f3\n","  Stored in directory: /root/.cache/pip/wheels/e7/f3/22/152153d6eb222ee7a56ff8617d80ee5207207a8c00a7aab794\n","Successfully built pathtools\n","Installing collected packages: pathtools, smmap, setproctitle, sentry-sdk, docker-pycreds, gitdb, GitPython, wandb\n","Successfully installed GitPython-3.1.31 docker-pycreds-0.4.0 gitdb-4.0.10 pathtools-0.1.2 sentry-sdk-1.21.1 setproctitle-1.3.2 smmap-5.0.0 wandb-0.15.0\n"]}]},{"cell_type":"code","source":["!python train.py config/train_gpt2.py # change dtype to float16 in train.py"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"6bkRPGU0FHyt","executionInfo":{"status":"ok","timestamp":1682833099299,"user_tz":-540,"elapsed":3909699,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"1a5c8b23-417c-4339-ea02-53f1dabb39e4"},"execution_count":10,"outputs":[{"output_type":"stream","name":"stdout","text":["Overriding config with config/train_gpt2.py:\n","# config for training GPT-2 (124M) down to very nice loss of ~2.85 on 1 node of 8X A100 40GB\n","# launch as the following (e.g. in a screen session) and wait ~5 days:\n","# $ torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py\n","\n","wandb_log = True\n","wandb_project = 'owt'\n","wandb_run_name='gpt2-124M'\n","\n","# these make the total batch size be ~0.5M\n","# 12 batch size * 1024 block size * 5 gradaccum * 8 GPUs = 491,520\n","batch_size = 12\n","block_size = 1024\n","gradient_accumulation_steps = 5\n","\n","# this makes total number of tokens be 300B\n","max_iters = 600000\n","lr_decay_iters = 600000\n","\n","# eval stuff\n","eval_interval = 1000\n","eval_iters = 200\n","log_interval = 10\n","\n","# weight decay\n","weight_decay = 1e-1\n","\n","tokens per iteration will be: 61,440\n","Initializing a new model from scratch\n","defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency)\n","number of parameters: 123.59M\n","using fused AdamW: True\n","compiling the model... (takes a ~minute)\n","\u001b[34m\u001b[1mwandb\u001b[0m: (1) Create a W&B account\n","\u001b[34m\u001b[1mwandb\u001b[0m: (2) Use an existing W&B account\n","\u001b[34m\u001b[1mwandb\u001b[0m: (3) Don't visualize my results\n","\u001b[34m\u001b[1mwandb\u001b[0m: Enter your choice: 3\n","\u001b[34m\u001b[1mwandb\u001b[0m: You chose \"Don't visualize my results\"\n","\u001b[34m\u001b[1mwandb\u001b[0m: Tracking run with wandb version 0.15.0\n","\u001b[34m\u001b[1mwandb\u001b[0m: W&B syncing is set to \u001b[1m`offline`\u001b[0m in this directory.  \n","\u001b[34m\u001b[1mwandb\u001b[0m: Run \u001b[1m`wandb online`\u001b[0m or set \u001b[1mWANDB_MODE=online\u001b[0m to enable cloud syncing.\n","step 0: train loss 10.9885, val loss 10.9897\n","iter 0: loss 10.9933, time 41380.17ms, mfu -100.00%\n","iter 10: loss 10.3945, time 427.56ms, mfu 39.37%\n","iter 20: loss 9.7840, time 427.07ms, mfu 39.37%\n","iter 30: loss 9.5147, time 428.39ms, mfu 39.36%\n","iter 40: loss 9.2241, time 428.09ms, mfu 39.36%\n","iter 50: loss 9.0063, time 428.63ms, mfu 39.35%\n","iter 60: loss 8.8773, time 428.10ms, mfu 39.35%\n","iter 70: loss 8.5903, time 428.04ms, mfu 39.35%\n","iter 80: loss 8.4633, time 428.15ms, mfu 39.34%\n","iter 90: loss 8.2469, time 428.51ms, mfu 39.34%\n","iter 100: loss 8.1014, time 428.08ms, mfu 39.33%\n","iter 110: loss 8.0945, time 428.41ms, mfu 39.33%\n","iter 120: loss 7.7050, time 427.98ms, mfu 39.33%\n","iter 130: loss 7.4627, time 429.16ms, mfu 39.32%\n","iter 140: loss 7.5593, time 428.25ms, mfu 39.32%\n","iter 150: loss 7.1342, time 427.89ms, mfu 39.32%\n","iter 160: loss 7.0447, time 428.98ms, mfu 39.31%\n","iter 170: loss 7.0497, time 429.23ms, mfu 39.30%\n","iter 180: loss 7.0297, time 427.96ms, mfu 39.31%\n","iter 190: loss 6.9281, time 429.52ms, mfu 39.29%\n","iter 200: loss 6.7482, time 429.55ms, mfu 39.28%\n","iter 210: loss 6.9311, time 429.09ms, mfu 39.28%\n","iter 220: loss 6.4906, time 429.03ms, mfu 39.27%\n","iter 230: loss 6.5814, time 428.24ms, mfu 39.28%\n","iter 240: loss 6.3486, time 429.82ms, mfu 39.26%\n","iter 250: loss 6.6301, time 429.51ms, mfu 39.26%\n","iter 260: loss 6.6438, time 429.77ms, mfu 39.25%\n","iter 270: loss 6.5446, time 428.02ms, mfu 39.26%\n","iter 280: loss 6.4338, time 430.88ms, mfu 39.24%\n","iter 290: loss 6.3512, time 430.76ms, mfu 39.22%\n","iter 300: loss 6.3811, time 428.71ms, mfu 39.22%\n","iter 310: loss 6.5188, time 430.59ms, mfu 39.21%\n","iter 320: loss 6.5734, time 430.29ms, mfu 39.20%\n","iter 330: loss 6.4931, time 429.65ms, mfu 39.20%\n","iter 340: loss 6.2903, time 430.33ms, mfu 39.19%\n","iter 350: loss 6.4876, time 430.53ms, mfu 39.18%\n","iter 360: loss 6.3452, time 429.79ms, mfu 39.18%\n","iter 370: loss 6.2723, time 429.36ms, mfu 39.18%\n","iter 380: loss 6.2526, time 430.53ms, mfu 39.17%\n","iter 390: loss 6.0021, time 429.78ms, mfu 39.17%\n","iter 400: loss 6.1885, time 430.00ms, mfu 39.17%\n","iter 410: loss 6.2201, time 430.22ms, mfu 39.17%\n","iter 420: loss 6.0175, time 428.97ms, mfu 39.17%\n","iter 430: loss 6.4431, time 429.69ms, mfu 39.17%\n","iter 440: loss 6.0694, time 430.50ms, mfu 39.17%\n","iter 450: loss 5.9703, time 429.23ms, mfu 39.17%\n","iter 460: loss 6.2788, time 430.29ms, mfu 39.17%\n","iter 470: loss 5.9339, time 430.34ms, mfu 39.16%\n","iter 480: loss 6.0564, time 429.81ms, mfu 39.16%\n","iter 490: loss 5.9708, time 430.87ms, mfu 39.15%\n","iter 500: loss 5.8272, time 429.35ms, mfu 39.16%\n","iter 510: loss 6.0802, time 429.82ms, mfu 39.16%\n","iter 520: loss 5.6170, time 429.64ms, mfu 39.16%\n","iter 530: loss 5.9851, time 429.47ms, mfu 39.16%\n","iter 540: loss 5.8363, time 430.09ms, mfu 39.16%\n","iter 550: loss 5.8784, time 429.29ms, mfu 39.16%\n","iter 560: loss 5.7326, time 430.06ms, mfu 39.16%\n","iter 570: loss 5.6330, time 429.73ms, mfu 39.16%\n","iter 580: loss 5.7412, time 430.35ms, mfu 39.16%\n","iter 590: loss 5.7453, time 429.59ms, mfu 39.16%\n","iter 600: loss 5.8106, time 430.73ms, mfu 39.15%\n","iter 610: loss 5.9872, time 429.82ms, mfu 39.15%\n","iter 620: loss 6.0173, time 429.99ms, mfu 39.15%\n","iter 630: loss 5.7007, time 428.99ms, mfu 39.16%\n","iter 640: loss 5.4480, time 429.39ms, mfu 39.16%\n","iter 650: loss 5.7752, time 430.45ms, mfu 39.16%\n","iter 660: loss 5.6415, time 429.85ms, mfu 39.16%\n","iter 670: loss 5.8776, time 430.32ms, mfu 39.15%\n","iter 680: loss 5.7201, time 430.49ms, mfu 39.15%\n","iter 690: loss 5.7122, time 429.65ms, mfu 39.15%\n","iter 700: loss 5.7090, time 430.04ms, mfu 39.15%\n","iter 710: loss 5.6648, time 430.74ms, mfu 39.14%\n","iter 720: loss 5.5001, time 429.10ms, mfu 39.15%\n","iter 730: loss 5.4995, time 429.94ms, mfu 39.15%\n","iter 740: loss 5.3742, time 429.58ms, mfu 39.15%\n","iter 750: loss 5.6947, time 429.68ms, mfu 39.16%\n","iter 760: loss 5.5282, time 430.27ms, mfu 39.15%\n","iter 770: loss 5.8595, time 429.66ms, mfu 39.16%\n","iter 780: loss 5.2636, time 429.87ms, mfu 39.16%\n","iter 790: loss 5.3372, time 429.70ms, mfu 39.16%\n","iter 800: loss 5.5402, time 430.04ms, mfu 39.16%\n","iter 810: loss 5.3590, time 429.46ms, mfu 39.16%\n","iter 820: loss 5.4057, time 429.38ms, mfu 39.16%\n","iter 830: loss 5.4199, time 430.56ms, mfu 39.16%\n","iter 840: loss 5.4496, time 430.01ms, mfu 39.16%\n","iter 850: loss 5.2964, time 430.11ms, mfu 39.15%\n","iter 860: loss 5.3898, time 430.89ms, mfu 39.14%\n","iter 870: loss 5.5053, time 429.33ms, mfu 39.15%\n","iter 880: loss 5.2897, time 430.00ms, mfu 39.15%\n","iter 890: loss 5.4652, time 429.30ms, mfu 39.16%\n","iter 900: loss 5.2399, time 430.39ms, mfu 39.15%\n","iter 910: loss 5.2541, time 429.89ms, mfu 39.15%\n","iter 920: loss 5.3270, time 429.68ms, mfu 39.15%\n","iter 930: loss 5.3631, time 429.83ms, mfu 39.15%\n","iter 940: loss 5.1608, time 430.59ms, mfu 39.15%\n","iter 950: loss 4.9652, time 429.71ms, mfu 39.15%\n","iter 960: loss 5.2237, time 429.70ms, mfu 39.15%\n","iter 970: loss 5.1508, time 430.59ms, mfu 39.15%\n","iter 980: loss 4.8678, time 430.81ms, mfu 39.14%\n","iter 990: loss 5.1935, time 430.65ms, mfu 39.13%\n","step 1000: train loss 5.1938, val loss 5.1942\n","saving checkpoint to out\n","iter 1000: loss 5.1255, time 14122.77ms, mfu 35.34%\n","iter 1010: loss 5.1333, time 428.46ms, mfu 35.73%\n","iter 1020: loss 5.2649, time 429.11ms, mfu 36.08%\n","iter 1030: loss 5.1717, time 428.83ms, mfu 36.40%\n","iter 1040: loss 5.0471, time 429.59ms, mfu 36.68%\n","iter 1050: loss 4.8815, time 430.42ms, mfu 36.92%\n","iter 1060: loss 5.1638, time 429.64ms, mfu 37.15%\n","iter 1070: loss 4.9778, time 429.17ms, mfu 37.35%\n","iter 1080: loss 5.0797, time 430.04ms, mfu 37.53%\n","iter 1090: loss 5.2028, time 430.87ms, mfu 37.69%\n","iter 1100: loss 5.1263, time 430.15ms, mfu 37.83%\n","iter 1110: loss 4.9397, time 430.24ms, mfu 37.96%\n","iter 1120: loss 5.1658, time 430.46ms, mfu 38.07%\n","iter 1130: loss 4.9960, time 430.10ms, mfu 38.18%\n","iter 1140: loss 5.0728, time 429.79ms, mfu 38.28%\n","iter 1150: loss 5.0225, time 429.74ms, mfu 38.37%\n","iter 1160: loss 4.9467, time 430.00ms, mfu 38.45%\n","iter 1170: loss 4.9360, time 430.19ms, mfu 38.51%\n","iter 1180: loss 5.1218, time 429.91ms, mfu 38.58%\n","iter 1190: loss 4.9008, time 430.41ms, mfu 38.63%\n","iter 1200: loss 5.0604, time 430.53ms, mfu 38.68%\n","iter 1210: loss 4.8625, time 430.64ms, mfu 38.72%\n","iter 1220: loss 4.9501, time 431.09ms, mfu 38.75%\n","iter 1230: loss 4.7849, time 429.17ms, mfu 38.80%\n","iter 1240: loss 4.9112, time 429.65ms, mfu 38.84%\n","iter 1250: loss 4.9391, time 430.83ms, mfu 38.86%\n","iter 1260: loss 4.8360, time 430.82ms, mfu 38.88%\n","iter 1270: loss 4.8269, time 430.79ms, mfu 38.90%\n","iter 1280: loss 4.8174, time 430.97ms, mfu 38.92%\n","iter 1290: loss 4.7158, time 430.28ms, mfu 38.94%\n","iter 1300: loss 4.8719, time 430.15ms, mfu 38.96%\n","iter 1310: loss 4.9227, time 430.12ms, mfu 38.97%\n","iter 1320: loss 4.6384, time 430.68ms, mfu 38.98%\n","iter 1330: loss 4.7256, time 429.40ms, mfu 39.01%\n","iter 1340: loss 4.8095, time 429.75ms, mfu 39.02%\n","iter 1350: loss 4.8076, time 429.60ms, mfu 39.04%\n","iter 1360: loss 4.6654, time 430.32ms, mfu 39.05%\n","iter 1370: loss 4.6989, time 430.02ms, mfu 39.06%\n","iter 1380: loss 4.8548, time 429.71ms, mfu 39.07%\n","iter 1390: loss 4.6826, time 430.18ms, mfu 39.07%\n","iter 1400: loss 4.4797, time 430.50ms, mfu 39.08%\n","iter 1410: loss 4.6318, time 430.16ms, mfu 39.08%\n","iter 1420: loss 4.6940, time 430.21ms, mfu 39.09%\n","iter 1430: loss 4.5733, time 430.65ms, mfu 39.09%\n","iter 1440: loss 4.6382, time 429.66ms, mfu 39.09%\n","iter 1450: loss 4.6542, time 430.18ms, mfu 39.10%\n","iter 1460: loss 4.8211, time 430.37ms, mfu 39.10%\n","iter 1470: loss 4.4842, time 430.51ms, mfu 39.10%\n","iter 1480: loss 4.5745, time 430.64ms, mfu 39.10%\n","iter 1490: loss 4.6571, time 430.23ms, mfu 39.10%\n","iter 1500: loss 4.5444, time 430.45ms, mfu 39.10%\n","iter 1510: loss 4.6002, time 430.10ms, mfu 39.10%\n","iter 1520: loss 4.5025, time 430.69ms, mfu 39.10%\n","iter 1530: loss 4.5700, time 430.39ms, mfu 39.10%\n","iter 1540: loss 4.5111, time 430.56ms, mfu 39.10%\n","iter 1550: loss 4.5138, time 430.31ms, mfu 39.10%\n","iter 1560: loss 4.6001, time 430.15ms, mfu 39.11%\n","iter 1570: loss 4.6571, time 429.87ms, mfu 39.11%\n","iter 1580: loss 4.5136, time 430.36ms, mfu 39.11%\n","iter 1590: loss 4.6931, time 429.68ms, mfu 39.12%\n","iter 1600: loss 4.5991, time 429.23ms, mfu 39.13%\n","iter 1610: loss 4.6278, time 430.29ms, mfu 39.13%\n","iter 1620: loss 4.5269, time 429.76ms, mfu 39.13%\n","iter 1630: loss 4.5810, time 429.98ms, mfu 39.13%\n","iter 1640: loss 4.5997, time 431.25ms, mfu 39.12%\n","iter 1650: loss 4.4027, time 430.44ms, mfu 39.12%\n","iter 1660: loss 4.3599, time 429.77ms, mfu 39.13%\n","iter 1670: loss 4.5042, time 430.06ms, mfu 39.13%\n","iter 1680: loss 4.4655, time 430.46ms, mfu 39.12%\n","iter 1690: loss 4.2869, time 429.91ms, mfu 39.13%\n","iter 1700: loss 4.3433, time 430.98ms, mfu 39.12%\n","iter 1710: loss 4.4102, time 429.78ms, mfu 39.12%\n","iter 1720: loss 4.3998, time 430.92ms, mfu 39.12%\n","iter 1730: loss 4.4760, time 429.56ms, mfu 39.12%\n","iter 1740: loss 4.5034, time 430.44ms, mfu 39.12%\n","iter 1750: loss 4.5550, time 430.60ms, mfu 39.12%\n","iter 1760: loss 4.3183, time 430.03ms, mfu 39.12%\n","iter 1770: loss 4.5261, time 430.54ms, mfu 39.12%\n","iter 1780: loss 4.4570, time 430.15ms, mfu 39.12%\n","iter 1790: loss 4.3928, time 430.45ms, mfu 39.12%\n","iter 1800: loss 4.4095, time 430.26ms, mfu 39.12%\n","iter 1810: loss 4.3238, time 430.49ms, mfu 39.12%\n","iter 1820: loss 4.5351, time 430.55ms, mfu 39.12%\n","iter 1830: loss 4.2964, time 429.90ms, mfu 39.12%\n","iter 1840: loss 4.5134, time 429.65ms, mfu 39.12%\n","iter 1850: loss 4.4559, time 430.77ms, mfu 39.12%\n","iter 1860: loss 4.2754, time 430.47ms, mfu 39.12%\n","iter 1870: loss 4.2577, time 430.26ms, mfu 39.12%\n","iter 1880: loss 4.4877, time 430.81ms, mfu 39.11%\n","iter 1890: loss 4.6911, time 430.13ms, mfu 39.12%\n","iter 1900: loss 4.2964, time 430.56ms, mfu 39.11%\n","iter 1910: loss 4.3840, time 431.07ms, mfu 39.11%\n","iter 1920: loss 4.3040, time 430.03ms, mfu 39.11%\n","iter 1930: loss 4.4627, time 430.16ms, mfu 39.11%\n","iter 1940: loss 4.2999, time 430.20ms, mfu 39.11%\n","iter 1950: loss 4.4011, time 430.53ms, mfu 39.11%\n","iter 1960: loss 4.3231, time 430.40ms, mfu 39.11%\n","iter 1970: loss 4.2026, time 430.43ms, mfu 39.11%\n","iter 1980: loss 4.2636, time 430.78ms, mfu 39.11%\n","iter 1990: loss 4.3838, time 430.34ms, mfu 39.11%\n","step 2000: train loss 4.3212, val loss 4.3467\n","saving checkpoint to out\n","iter 2000: loss 4.4346, time 14414.96ms, mfu 35.31%\n","iter 2010: loss 4.2855, time 431.06ms, mfu 35.69%\n","iter 2020: loss 4.5037, time 429.95ms, mfu 36.03%\n","iter 2030: loss 4.3220, time 429.63ms, mfu 36.35%\n","iter 2040: loss 4.3646, time 430.93ms, mfu 36.62%\n","iter 2050: loss 4.2732, time 429.08ms, mfu 36.88%\n","iter 2060: loss 4.2425, time 428.46ms, mfu 37.12%\n","iter 2070: loss 4.4113, time 430.92ms, mfu 37.32%\n","iter 2080: loss 4.3645, time 429.47ms, mfu 37.50%\n","iter 2090: loss 4.3260, time 430.18ms, mfu 37.67%\n","iter 2100: loss 4.1935, time 429.62ms, mfu 37.82%\n","iter 2110: loss 4.1563, time 430.89ms, mfu 37.94%\n","iter 2120: loss 4.2062, time 429.30ms, mfu 38.07%\n","iter 2130: loss 4.2978, time 430.40ms, mfu 38.17%\n","iter 2140: loss 4.5420, time 430.09ms, mfu 38.27%\n","iter 2150: loss 4.0500, time 430.34ms, mfu 38.35%\n","iter 2160: loss 4.4640, time 430.31ms, mfu 38.43%\n","iter 2170: loss 4.2118, time 430.60ms, mfu 38.50%\n","iter 2180: loss 4.4123, time 430.00ms, mfu 38.56%\n","iter 2190: loss 4.4394, time 430.92ms, mfu 38.61%\n","iter 2200: loss 4.2283, time 430.45ms, mfu 38.66%\n","iter 2210: loss 4.3196, time 430.67ms, mfu 38.70%\n","iter 2220: loss 4.3470, time 429.96ms, mfu 38.75%\n","iter 2230: loss 4.2539, time 430.14ms, mfu 38.79%\n","iter 2240: loss 4.3847, time 430.12ms, mfu 38.82%\n","iter 2250: loss 4.1231, time 429.71ms, mfu 38.86%\n","iter 2260: loss 4.0635, time 430.66ms, mfu 38.88%\n","iter 2270: loss 4.2246, time 430.80ms, mfu 38.90%\n","iter 2280: loss 4.3375, time 429.51ms, mfu 38.93%\n","iter 2290: loss 4.2205, time 430.71ms, mfu 38.94%\n","iter 2300: loss 4.3082, time 430.15ms, mfu 38.96%\n","iter 2310: loss 4.0461, time 430.49ms, mfu 38.98%\n","iter 2320: loss 4.1549, time 430.34ms, mfu 38.99%\n","iter 2330: loss 4.1076, time 430.74ms, mfu 39.00%\n","iter 2340: loss 4.1619, time 431.32ms, mfu 39.00%\n","iter 2350: loss 4.1217, time 430.49ms, mfu 39.01%\n","iter 2360: loss 4.3124, time 430.95ms, mfu 39.02%\n","iter 2370: loss 4.1117, time 430.42ms, mfu 39.02%\n","iter 2380: loss 4.2544, time 429.71ms, mfu 39.04%\n","iter 2390: loss 4.0988, time 429.82ms, mfu 39.05%\n","iter 2400: loss 3.9615, time 431.40ms, mfu 39.05%\n","iter 2410: loss 4.3712, time 430.31ms, mfu 39.06%\n","iter 2420: loss 4.3668, time 430.55ms, mfu 39.06%\n","iter 2430: loss 4.1504, time 430.74ms, mfu 39.06%\n","iter 2440: loss 4.3078, time 430.80ms, mfu 39.06%\n","iter 2450: loss 4.2200, time 429.48ms, mfu 39.08%\n","iter 2460: loss 4.2288, time 430.43ms, mfu 39.08%\n","iter 2470: loss 4.2489, time 431.11ms, mfu 39.07%\n","iter 2480: loss 4.2611, time 430.30ms, mfu 39.08%\n","iter 2490: loss 4.1811, time 430.34ms, mfu 39.08%\n","iter 2500: loss 4.2003, time 429.63ms, mfu 39.09%\n","iter 2510: loss 3.9675, time 430.74ms, mfu 39.09%\n","iter 2520: loss 4.0453, time 431.00ms, mfu 39.09%\n","iter 2530: loss 4.2494, time 430.28ms, mfu 39.09%\n","iter 2540: loss 4.1999, time 431.01ms, mfu 39.09%\n","iter 2550: loss 4.1268, time 430.96ms, mfu 39.08%\n","iter 2560: loss 4.1058, time 430.91ms, mfu 39.08%\n","iter 2570: loss 4.0315, time 431.37ms, mfu 39.08%\n","iter 2580: loss 4.0832, time 431.19ms, mfu 39.07%\n","iter 2590: loss 4.2381, time 429.85ms, mfu 39.08%\n","iter 2600: loss 4.1114, time 431.13ms, mfu 39.08%\n","iter 2610: loss 4.0225, time 430.49ms, mfu 39.08%\n","iter 2620: loss 4.0138, time 430.80ms, mfu 39.08%\n","iter 2630: loss 4.4098, time 431.20ms, mfu 39.07%\n","iter 2640: loss 4.2444, time 429.88ms, mfu 39.08%\n","iter 2650: loss 3.8219, time 429.79ms, mfu 39.09%\n","iter 2660: loss 3.8492, time 430.45ms, mfu 39.09%\n","iter 2670: loss 4.1329, time 430.55ms, mfu 39.09%\n","iter 2680: loss 4.0296, time 429.78ms, mfu 39.10%\n","iter 2690: loss 4.1986, time 430.52ms, mfu 39.10%\n","iter 2700: loss 4.0516, time 430.32ms, mfu 39.10%\n","iter 2710: loss 4.0083, time 430.90ms, mfu 39.10%\n","iter 2720: loss 4.0620, time 431.53ms, mfu 39.09%\n","iter 2730: loss 4.2111, time 430.69ms, mfu 39.09%\n","iter 2740: loss 4.1678, time 430.55ms, mfu 39.09%\n","iter 2750: loss 4.1626, time 431.10ms, mfu 39.08%\n","iter 2760: loss 3.9491, time 430.39ms, mfu 39.09%\n","iter 2770: loss 3.9718, time 430.76ms, mfu 39.09%\n","iter 2780: loss 4.0472, time 430.84ms, mfu 39.08%\n","iter 2790: loss 4.1509, time 430.51ms, mfu 39.09%\n","iter 2800: loss 4.0186, time 430.52ms, mfu 39.09%\n","iter 2810: loss 4.0538, time 429.67ms, mfu 39.10%\n","iter 2820: loss 4.0813, time 430.02ms, mfu 39.10%\n","iter 2830: loss 4.0802, time 431.14ms, mfu 39.09%\n","iter 2840: loss 4.0064, time 430.53ms, mfu 39.09%\n","iter 2850: loss 4.1123, time 430.33ms, mfu 39.10%\n","iter 2860: loss 4.0961, time 430.36ms, mfu 39.10%\n","iter 2870: loss 3.8213, time 430.96ms, mfu 39.09%\n","iter 2880: loss 4.1898, time 430.93ms, mfu 39.09%\n","iter 2890: loss 4.0964, time 430.37ms, mfu 39.09%\n","iter 2900: loss 4.1413, time 429.79ms, mfu 39.10%\n","iter 2910: loss 3.9613, time 430.33ms, mfu 39.10%\n","iter 2920: loss 4.0446, time 430.23ms, mfu 39.10%\n","iter 2930: loss 4.1226, time 430.53ms, mfu 39.10%\n","iter 2940: loss 3.9876, time 430.75ms, mfu 39.10%\n","iter 2950: loss 4.0991, time 431.08ms, mfu 39.10%\n","iter 2960: loss 3.8283, time 430.09ms, mfu 39.10%\n","iter 2970: loss 4.0741, time 430.60ms, mfu 39.10%\n","iter 2980: loss 4.0523, time 430.11ms, mfu 39.10%\n","iter 2990: loss 4.1392, time 430.63ms, mfu 39.10%\n","step 3000: train loss 4.0269, val loss 4.0269\n","saving checkpoint to out\n","iter 3000: loss 4.0251, time 14293.56ms, mfu 35.31%\n","iter 3010: loss 4.0742, time 429.52ms, mfu 35.70%\n","iter 3020: loss 4.2702, time 429.54ms, mfu 36.05%\n","iter 3030: loss 4.0022, time 430.02ms, mfu 36.36%\n","iter 3040: loss 3.9439, time 431.10ms, mfu 36.62%\n","iter 3050: loss 3.9712, time 430.05ms, mfu 36.88%\n","iter 3060: loss 3.9274, time 429.96ms, mfu 37.10%\n","iter 3070: loss 3.9031, time 430.34ms, mfu 37.30%\n","iter 3080: loss 4.0478, time 430.49ms, mfu 37.48%\n","iter 3090: loss 4.0973, time 430.73ms, mfu 37.64%\n","iter 3100: loss 3.8517, time 430.22ms, mfu 37.79%\n","iter 3110: loss 3.8116, time 430.34ms, mfu 37.92%\n","iter 3120: loss 4.0172, time 430.41ms, mfu 38.04%\n","iter 3130: loss 3.8609, time 429.93ms, mfu 38.15%\n","iter 3140: loss 3.7597, time 430.04ms, mfu 38.25%\n","iter 3150: loss 3.9711, time 430.10ms, mfu 38.34%\n","iter 3160: loss 4.0363, time 430.14ms, mfu 38.42%\n","iter 3170: loss 3.8351, time 430.61ms, mfu 38.49%\n","iter 3180: loss 4.1731, time 431.00ms, mfu 38.54%\n","iter 3190: loss 3.9264, time 429.91ms, mfu 38.60%\n","iter 3200: loss 4.0991, time 430.70ms, mfu 38.65%\n","iter 3210: loss 4.1549, time 429.36ms, mfu 38.71%\n","iter 3220: loss 3.7998, time 430.03ms, mfu 38.75%\n","iter 3230: loss 4.0676, time 431.07ms, mfu 38.78%\n","iter 3240: loss 3.9954, time 430.56ms, mfu 38.81%\n","iter 3250: loss 3.9877, time 431.65ms, mfu 38.83%\n","iter 3260: loss 4.2175, time 430.54ms, mfu 38.86%\n","iter 3270: loss 4.0105, time 430.95ms, mfu 38.88%\n","iter 3280: loss 4.0474, time 429.61ms, mfu 38.91%\n","iter 3290: loss 3.6791, time 430.30ms, mfu 38.93%\n","iter 3300: loss 4.0357, time 430.15ms, mfu 38.95%\n","iter 3310: loss 4.0349, time 429.97ms, mfu 38.97%\n","iter 3320: loss 3.9755, time 430.06ms, mfu 38.99%\n","iter 3330: loss 3.8889, time 431.03ms, mfu 38.99%\n","iter 3340: loss 3.9654, time 430.26ms, mfu 39.01%\n","iter 3350: loss 3.7082, time 430.50ms, mfu 39.01%\n","iter 3360: loss 4.0471, time 430.48ms, mfu 39.02%\n","iter 3370: loss 4.0992, time 430.66ms, mfu 39.03%\n","iter 3380: loss 3.7776, time 430.41ms, mfu 39.04%\n","iter 3390: loss 3.9593, time 430.47ms, mfu 39.04%\n","iter 3400: loss 3.9972, time 430.79ms, mfu 39.05%\n","iter 3410: loss 3.8043, time 430.32ms, mfu 39.05%\n","iter 3420: loss 4.2602, time 430.54ms, mfu 39.06%\n","iter 3430: loss 4.1139, time 430.53ms, mfu 39.06%\n","iter 3440: loss 3.9406, time 430.35ms, mfu 39.07%\n","iter 3450: loss 3.9407, time 430.31ms, mfu 39.07%\n","iter 3460: loss 3.8250, time 430.24ms, mfu 39.08%\n","iter 3470: loss 4.0144, time 430.54ms, mfu 39.08%\n","iter 3480: loss 3.8826, time 430.98ms, mfu 39.08%\n","iter 3490: loss 3.8795, time 430.23ms, mfu 39.08%\n","iter 3500: loss 3.9039, time 430.12ms, mfu 39.09%\n","iter 3510: loss 4.0976, time 430.05ms, mfu 39.09%\n","iter 3520: loss 3.8697, time 430.86ms, mfu 39.09%\n","iter 3530: loss 3.9775, time 430.21ms, mfu 39.09%\n","iter 3540: loss 3.9539, time 430.54ms, mfu 39.09%\n","iter 3550: loss 3.7785, time 430.80ms, mfu 39.09%\n","iter 3560: loss 3.9179, time 430.63ms, mfu 39.09%\n","iter 3570: loss 3.7499, time 430.81ms, mfu 39.09%\n","iter 3580: loss 3.9860, time 430.78ms, mfu 39.09%\n","iter 3590: loss 3.7343, time 430.01ms, mfu 39.09%\n","iter 3600: loss 3.8208, time 429.70ms, mfu 39.10%\n","iter 3610: loss 3.8176, time 430.83ms, mfu 39.10%\n","iter 3620: loss 4.0281, time 430.70ms, mfu 39.10%\n","iter 3630: loss 3.9173, time 430.95ms, mfu 39.09%\n","iter 3640: loss 3.8494, time 430.54ms, mfu 39.09%\n","iter 3650: loss 3.8078, time 430.45ms, mfu 39.09%\n","iter 3660: loss 3.8215, time 430.31ms, mfu 39.10%\n","iter 3670: loss 4.0766, time 430.82ms, mfu 39.09%\n","iter 3680: loss 3.9163, time 430.84ms, mfu 39.09%\n","iter 3690: loss 4.1170, time 430.60ms, mfu 39.09%\n","iter 3700: loss 4.0099, time 430.75ms, mfu 39.09%\n","iter 3710: loss 3.9067, time 430.14ms, mfu 39.09%\n","iter 3720: loss 4.0461, time 429.95ms, mfu 39.10%\n","iter 3730: loss 3.7429, time 430.63ms, mfu 39.10%\n","iter 3740: loss 3.7356, time 430.66ms, mfu 39.10%\n","iter 3750: loss 4.0227, time 430.06ms, mfu 39.10%\n","iter 3760: loss 3.8652, time 431.42ms, mfu 39.09%\n","iter 3770: loss 3.7210, time 430.28ms, mfu 39.10%\n","iter 3780: loss 3.8782, time 430.74ms, mfu 39.09%\n","iter 3790: loss 3.8168, time 430.67ms, mfu 39.09%\n","iter 3800: loss 4.0700, time 430.53ms, mfu 39.09%\n","iter 3810: loss 3.7751, time 431.73ms, mfu 39.08%\n","iter 3820: loss 3.9766, time 430.72ms, mfu 39.08%\n","iter 3830: loss 3.9850, time 430.18ms, mfu 39.09%\n","iter 3840: loss 4.0190, time 431.30ms, mfu 39.08%\n","iter 3850: loss 3.9367, time 430.94ms, mfu 39.08%\n","iter 3860: loss 4.0419, time 430.19ms, mfu 39.08%\n","iter 3870: loss 4.1406, time 430.59ms, mfu 39.08%\n","iter 3880: loss 3.8896, time 430.39ms, mfu 39.09%\n","iter 3890: loss 3.8211, time 430.13ms, mfu 39.09%\n","iter 3900: loss 3.9001, time 429.61ms, mfu 39.10%\n","iter 3910: loss 3.7926, time 430.33ms, mfu 39.10%\n","iter 3920: loss 3.7241, time 430.05ms, mfu 39.11%\n","iter 3930: loss 3.8186, time 429.97ms, mfu 39.11%\n","iter 3940: loss 4.1543, time 430.37ms, mfu 39.11%\n","iter 3950: loss 4.0243, time 435.23ms, mfu 39.07%\n","iter 3960: loss 3.8469, time 430.18ms, mfu 39.07%\n","iter 3970: loss 3.7952, time 429.51ms, mfu 39.08%\n","iter 3980: loss 3.7562, time 430.03ms, mfu 39.09%\n","iter 3990: loss 3.8479, time 430.10ms, mfu 39.09%\n","step 4000: train loss 3.8982, val loss 3.8930\n","saving checkpoint to out\n","iter 4000: loss 3.9557, time 14362.13ms, mfu 35.30%\n","iter 4010: loss 3.7603, time 430.37ms, mfu 35.68%\n","iter 4020: loss 3.8360, time 429.92ms, mfu 36.03%\n","iter 4030: loss 3.9772, time 430.07ms, mfu 36.34%\n","iter 4040: loss 3.8391, time 430.21ms, mfu 36.62%\n","iter 4050: loss 3.8917, time 429.63ms, mfu 36.88%\n","iter 4060: loss 3.9037, time 430.38ms, mfu 37.10%\n","iter 4070: loss 3.6569, time 429.58ms, mfu 37.31%\n","iter 4080: loss 3.8497, time 430.59ms, mfu 37.49%\n","iter 4090: loss 3.8020, time 430.17ms, mfu 37.65%\n","iter 4100: loss 3.7222, time 430.75ms, mfu 37.79%\n","iter 4110: loss 3.8155, time 429.85ms, mfu 37.93%\n","iter 4120: loss 3.9465, time 430.86ms, mfu 38.04%\n","iter 4130: loss 3.9084, time 430.85ms, mfu 38.15%\n","iter 4140: loss 3.9158, time 430.52ms, mfu 38.24%\n","iter 4150: loss 3.7416, time 429.97ms, mfu 38.33%\n","iter 4160: loss 3.9770, time 430.48ms, mfu 38.41%\n","iter 4170: loss 3.8912, time 430.48ms, mfu 38.48%\n","iter 4180: loss 3.9326, time 430.28ms, mfu 38.54%\n","iter 4190: loss 3.8898, time 430.37ms, mfu 38.60%\n","iter 4200: loss 3.7981, time 430.04ms, mfu 38.65%\n","iter 4210: loss 3.9099, time 431.00ms, mfu 38.69%\n","iter 4220: loss 3.6674, time 431.40ms, mfu 38.73%\n","iter 4230: loss 3.7601, time 430.90ms, mfu 38.76%\n","iter 4240: loss 3.6579, time 430.68ms, mfu 38.79%\n","iter 4250: loss 3.7966, time 430.91ms, mfu 38.82%\n","iter 4260: loss 3.8537, time 431.03ms, mfu 38.84%\n","iter 4270: loss 3.7745, time 430.71ms, mfu 38.87%\n","iter 4280: loss 3.8255, time 430.53ms, mfu 38.89%\n","iter 4290: loss 3.7904, time 431.17ms, mfu 38.90%\n","iter 4300: loss 3.7462, time 430.53ms, mfu 38.92%\n","iter 4310: loss 3.8309, time 429.91ms, mfu 38.95%\n","iter 4320: loss 3.6827, time 430.38ms, mfu 38.96%\n","iter 4330: loss 3.7148, time 430.86ms, mfu 38.97%\n","iter 4340: loss 3.9774, time 430.35ms, mfu 38.99%\n","iter 4350: loss 3.8173, time 429.74ms, mfu 39.01%\n","iter 4360: loss 3.8576, time 430.33ms, mfu 39.02%\n","iter 4370: loss 3.8226, time 430.48ms, mfu 39.02%\n","iter 4380: loss 4.0281, time 430.97ms, mfu 39.03%\n","iter 4390: loss 3.8114, time 429.95ms, mfu 39.04%\n","iter 4400: loss 3.9535, time 430.29ms, mfu 39.05%\n","iter 4410: loss 3.9001, time 430.70ms, mfu 39.05%\n","iter 4420: loss 3.8214, time 430.60ms, mfu 39.06%\n","iter 4430: loss 3.8056, time 430.75ms, mfu 39.06%\n","iter 4440: loss 3.6727, time 430.40ms, mfu 39.06%\n","iter 4450: loss 3.7226, time 430.51ms, mfu 39.07%\n","iter 4460: loss 3.9009, time 429.99ms, mfu 39.07%\n","iter 4470: loss 3.6715, time 430.52ms, mfu 39.08%\n","iter 4480: loss 3.8685, time 430.21ms, mfu 39.08%\n","iter 4490: loss 3.8286, time 429.72ms, mfu 39.09%\n","iter 4500: loss 3.9607, time 430.19ms, mfu 39.09%\n","iter 4510: loss 3.8760, time 430.50ms, mfu 39.09%\n","iter 4520: loss 3.6868, time 431.61ms, mfu 39.09%\n","iter 4530: loss 3.6694, time 430.62ms, mfu 39.09%\n","iter 4540: loss 3.8687, time 430.96ms, mfu 39.08%\n","iter 4550: loss 3.9498, time 430.25ms, mfu 39.09%\n","iter 4560: loss 3.9012, time 429.98ms, mfu 39.09%\n","iter 4570: loss 3.7427, time 431.58ms, mfu 39.08%\n","iter 4580: loss 3.6813, time 431.28ms, mfu 39.08%\n","iter 4590: loss 3.7432, time 430.02ms, mfu 39.08%\n","iter 4600: loss 3.8680, time 430.68ms, mfu 39.08%\n","iter 4610: loss 3.7938, time 430.53ms, mfu 39.09%\n","iter 4620: loss 3.9618, time 430.90ms, mfu 39.08%\n","iter 4630: loss 4.2376, time 431.11ms, mfu 39.08%\n","iter 4640: loss 3.7449, time 430.04ms, mfu 39.09%\n","iter 4650: loss 3.8623, time 430.63ms, mfu 39.09%\n","iter 4660: loss 3.8063, time 430.65ms, mfu 39.09%\n","iter 4670: loss 3.6638, time 430.91ms, mfu 39.08%\n","iter 4680: loss 3.8028, time 431.32ms, mfu 39.08%\n","iter 4690: loss 3.8576, time 429.92ms, mfu 39.09%\n","iter 4700: loss 3.7776, time 429.81ms, mfu 39.09%\n","iter 4710: loss 3.9950, time 430.31ms, mfu 39.10%\n","iter 4720: loss 3.7446, time 431.10ms, mfu 39.09%\n","iter 4730: loss 3.7681, time 430.17ms, mfu 39.09%\n","iter 4740: loss 3.8252, time 431.21ms, mfu 39.09%\n","iter 4750: loss 3.7548, time 430.41ms, mfu 39.09%\n","iter 4760: loss 3.8095, time 431.02ms, mfu 39.09%\n","iter 4770: loss 3.7998, time 430.05ms, mfu 39.09%\n","iter 4780: loss 3.8311, time 430.85ms, mfu 39.09%\n","iter 4790: loss 3.9539, time 430.46ms, mfu 39.09%\n","iter 4800: loss 3.7681, time 430.47ms, mfu 39.09%\n","iter 4810: loss 3.7200, time 429.94ms, mfu 39.10%\n","iter 4820: loss 3.7457, time 430.34ms, mfu 39.10%\n","iter 4830: loss 3.9482, time 429.80ms, mfu 39.11%\n","iter 4840: loss 3.7633, time 430.04ms, mfu 39.11%\n","iter 4850: loss 3.8201, time 431.15ms, mfu 39.10%\n","iter 4860: loss 3.9112, time 430.74ms, mfu 39.10%\n","iter 4870: loss 3.9116, time 430.41ms, mfu 39.10%\n","iter 4880: loss 3.6792, time 430.55ms, mfu 39.10%\n","iter 4890: loss 3.9863, time 430.61ms, mfu 39.10%\n","iter 4900: loss 3.7701, time 429.99ms, mfu 39.10%\n","iter 4910: loss 3.6922, time 430.02ms, mfu 39.11%\n","iter 4920: loss 3.8746, time 430.25ms, mfu 39.11%\n","iter 4930: loss 3.8245, time 429.81ms, mfu 39.11%\n","iter 4940: loss 3.9603, time 431.25ms, mfu 39.11%\n","iter 4950: loss 3.7439, time 430.38ms, mfu 39.11%\n","iter 4960: loss 3.7256, time 431.02ms, mfu 39.10%\n","iter 4970: loss 3.9357, time 430.95ms, mfu 39.10%\n","iter 4980: loss 3.8584, time 430.67ms, mfu 39.10%\n","iter 4990: loss 3.7772, time 429.86ms, mfu 39.10%\n","step 5000: train loss 3.7933, val loss 3.8162\n","saving checkpoint to out\n","iter 5000: loss 3.9874, time 15963.69ms, mfu 35.30%\n","iter 5010: loss 3.6978, time 428.85ms, mfu 35.69%\n","iter 5020: loss 3.8850, time 429.88ms, mfu 36.04%\n","iter 5030: loss 3.8784, time 429.80ms, mfu 36.35%\n","iter 5040: loss 3.7338, time 430.40ms, mfu 36.63%\n","iter 5050: loss 3.8672, time 429.74ms, mfu 36.88%\n","iter 5060: loss 3.7239, time 429.26ms, mfu 37.11%\n","iter 5070: loss 3.9220, time 430.50ms, mfu 37.31%\n","iter 5080: loss 3.8505, time 429.78ms, mfu 37.50%\n","iter 5090: loss 3.7395, time 430.32ms, mfu 37.66%\n","iter 5100: loss 3.8853, time 430.92ms, mfu 37.80%\n","iter 5110: loss 3.7194, time 430.10ms, mfu 37.93%\n","iter 5120: loss 3.5572, time 430.12ms, mfu 38.05%\n","iter 5130: loss 3.8241, time 430.83ms, mfu 38.16%\n","iter 5140: loss 3.8375, time 430.69ms, mfu 38.25%\n","iter 5150: loss 3.7320, time 430.42ms, mfu 38.33%\n","iter 5160: loss 3.6421, time 430.28ms, mfu 38.41%\n","iter 5170: loss 3.6185, time 430.52ms, mfu 38.48%\n","iter 5180: loss 3.7870, time 430.85ms, mfu 38.54%\n","iter 5190: loss 3.7295, time 431.22ms, mfu 38.59%\n","iter 5200: loss 3.5634, time 430.70ms, mfu 38.64%\n","iter 5210: loss 3.8478, time 430.08ms, mfu 38.69%\n","iter 5220: loss 3.7702, time 430.36ms, mfu 38.73%\n","iter 5230: loss 3.7684, time 429.80ms, mfu 38.77%\n","iter 5240: loss 3.7055, time 435.47ms, mfu 38.76%\n","iter 5250: loss 3.6114, time 429.84ms, mfu 38.80%\n","iter 5260: loss 3.9659, time 430.50ms, mfu 38.83%\n","iter 5270: loss 3.7229, time 430.41ms, mfu 38.86%\n","iter 5280: loss 3.6707, time 430.80ms, mfu 38.88%\n","iter 5290: loss 3.7577, time 431.13ms, mfu 38.90%\n","iter 5300: loss 3.8746, time 430.11ms, mfu 38.92%\n","iter 5310: loss 3.8307, time 430.63ms, mfu 38.94%\n","iter 5320: loss 3.9327, time 431.08ms, mfu 38.95%\n","iter 5330: loss 3.8614, time 430.57ms, mfu 38.96%\n","iter 5340: loss 3.7497, time 430.59ms, mfu 38.98%\n","iter 5350: loss 3.6461, time 430.24ms, mfu 38.99%\n","iter 5360: loss 3.6798, time 431.16ms, mfu 39.00%\n","iter 5370: loss 3.9354, time 430.47ms, mfu 39.01%\n","iter 5380: loss 3.9495, time 430.65ms, mfu 39.01%\n","iter 5390: loss 3.6729, time 430.87ms, mfu 39.02%\n","iter 5400: loss 3.7921, time 430.13ms, mfu 39.03%\n","iter 5410: loss 3.9114, time 430.95ms, mfu 39.03%\n","iter 5420: loss 3.6451, time 430.02ms, mfu 39.04%\n","iter 5430: loss 3.6594, time 430.39ms, mfu 39.05%\n","iter 5440: loss 3.8400, time 430.11ms, mfu 39.06%\n","iter 5450: loss 3.7382, time 430.68ms, mfu 39.06%\n","iter 5460: loss 3.9192, time 430.69ms, mfu 39.06%\n","iter 5470: loss 3.8283, time 430.47ms, mfu 39.07%\n","iter 5480: loss 3.4066, time 430.00ms, mfu 39.08%\n","iter 5490: loss 3.7980, time 430.44ms, mfu 39.08%\n","iter 5500: loss 3.8586, time 429.39ms, mfu 39.09%\n","iter 5510: loss 3.8336, time 429.90ms, mfu 39.10%\n","iter 5520: loss 3.7963, time 430.36ms, mfu 39.10%\n","iter 5530: loss 3.6389, time 430.14ms, mfu 39.10%\n","iter 5540: loss 3.8701, time 430.59ms, mfu 39.10%\n","iter 5550: loss 3.6670, time 430.28ms, mfu 39.10%\n","iter 5560: loss 3.6187, time 429.85ms, mfu 39.11%\n","iter 5570: loss 3.8990, time 430.60ms, mfu 39.11%\n","iter 5580: loss 3.6709, time 430.56ms, mfu 39.11%\n","iter 5590: loss 3.7774, time 430.51ms, mfu 39.10%\n","iter 5600: loss 3.7557, time 430.99ms, mfu 39.10%\n","iter 5610: loss 3.9113, time 430.69ms, mfu 39.10%\n","iter 5620: loss 3.7648, time 430.85ms, mfu 39.09%\n","iter 5630: loss 3.8468, time 430.12ms, mfu 39.10%\n","iter 5640: loss 3.8309, time 430.27ms, mfu 39.10%\n","iter 5650: loss 3.5573, time 430.49ms, mfu 39.10%\n","iter 5660: loss 3.7955, time 430.25ms, mfu 39.10%\n","iter 5670: loss 3.8771, time 430.90ms, mfu 39.10%\n","iter 5680: loss 3.6201, time 430.15ms, mfu 39.10%\n","iter 5690: loss 3.8495, time 430.36ms, mfu 39.10%\n","iter 5700: loss 3.9834, time 430.50ms, mfu 39.10%\n","iter 5710: loss 3.7159, time 430.24ms, mfu 39.11%\n","iter 5720: loss 3.7186, time 430.34ms, mfu 39.11%\n","iter 5730: loss 3.9638, time 431.09ms, mfu 39.10%\n","iter 5740: loss 3.9177, time 430.63ms, mfu 39.10%\n","iter 5750: loss 3.7224, time 430.29ms, mfu 39.10%\n","iter 5760: loss 3.3509, time 430.39ms, mfu 39.10%\n","iter 5770: loss 3.4749, time 429.95ms, mfu 39.11%\n","iter 5780: loss 3.5613, time 430.94ms, mfu 39.10%\n","iter 5790: loss 3.8932, time 430.76ms, mfu 39.10%\n","iter 5800: loss 3.6741, time 429.67ms, mfu 39.11%\n","iter 5810: loss 3.8175, time 430.00ms, mfu 39.11%\n","iter 5820: loss 3.7044, time 430.31ms, mfu 39.11%\n","iter 5830: loss 3.7911, time 431.33ms, mfu 39.10%\n","iter 5840: loss 3.4866, time 430.95ms, mfu 39.10%\n","iter 5850: loss 3.8176, time 430.88ms, mfu 39.09%\n","iter 5860: loss 3.7478, time 429.60ms, mfu 39.10%\n","iter 5870: loss 3.7742, time 431.14ms, mfu 39.10%\n","iter 5880: loss 3.7449, time 430.85ms, mfu 39.09%\n","iter 5890: loss 3.6986, time 429.92ms, mfu 39.10%\n","iter 5900: loss 3.8309, time 430.40ms, mfu 39.10%\n","iter 5910: loss 3.6627, time 430.40ms, mfu 39.10%\n","iter 5920: loss 3.6832, time 429.70ms, mfu 39.11%\n","iter 5930: loss 3.7669, time 430.56ms, mfu 39.11%\n","iter 5940: loss 3.8211, time 430.75ms, mfu 39.10%\n","iter 5950: loss 3.6390, time 430.85ms, mfu 39.10%\n","iter 5960: loss 3.7653, time 430.72ms, mfu 39.10%\n","iter 5970: loss 4.0486, time 429.94ms, mfu 39.10%\n","iter 5980: loss 3.8035, time 430.78ms, mfu 39.10%\n","iter 5990: loss 3.7842, time 430.48ms, mfu 39.10%\n","step 6000: train loss 3.7219, val loss 3.7387\n","saving checkpoint to out\n","iter 6000: loss 3.7124, time 16555.06ms, mfu 35.29%\n","iter 6010: loss 3.8100, time 429.17ms, mfu 35.69%\n","iter 6020: loss 3.8340, time 429.23ms, mfu 36.04%\n","iter 6030: loss 3.8675, time 430.05ms, mfu 36.35%\n","iter 6040: loss 3.8328, time 429.87ms, mfu 36.63%\n","iter 6050: loss 3.6278, time 430.55ms, mfu 36.88%\n","iter 6060: loss 3.7304, time 429.79ms, mfu 37.10%\n","iter 6070: loss 3.7914, time 429.91ms, mfu 37.31%\n","iter 6080: loss 3.9557, time 430.11ms, mfu 37.49%\n","iter 6090: loss 3.6964, time 430.06ms, mfu 37.66%\n","iter 6100: loss 3.8475, time 430.53ms, mfu 37.80%\n","iter 6110: loss 3.7805, time 430.16ms, mfu 37.93%\n","iter 6120: loss 3.8775, time 430.17ms, mfu 38.05%\n","iter 6130: loss 3.6277, time 430.26ms, mfu 38.16%\n","iter 6140: loss 3.5903, time 430.21ms, mfu 38.26%\n","iter 6150: loss 3.8022, time 430.66ms, mfu 38.34%\n","iter 6160: loss 3.6641, time 430.05ms, mfu 38.42%\n","iter 6170: loss 3.8400, time 430.57ms, mfu 38.49%\n","iter 6180: loss 3.6598, time 429.81ms, mfu 38.55%\n","iter 6190: loss 3.6330, time 430.42ms, mfu 38.61%\n","iter 6200: loss 3.6079, time 430.34ms, mfu 38.66%\n","iter 6210: loss 3.6894, time 430.38ms, mfu 38.71%\n","iter 6220: loss 3.6650, time 430.15ms, mfu 38.75%\n","iter 6230: loss 3.5824, time 431.07ms, mfu 38.78%\n","iter 6240: loss 3.5438, time 430.29ms, mfu 38.81%\n","iter 6250: loss 3.8186, time 429.99ms, mfu 38.85%\n","iter 6260: loss 3.7641, time 430.66ms, mfu 38.87%\n","iter 6270: loss 3.8640, time 430.32ms, mfu 38.89%\n","iter 6280: loss 3.7038, time 430.56ms, mfu 38.91%\n","iter 6290: loss 3.6995, time 430.55ms, mfu 38.93%\n","iter 6300: loss 3.6002, time 430.34ms, mfu 38.95%\n","iter 6310: loss 3.6323, time 429.66ms, mfu 38.97%\n","iter 6320: loss 3.6185, time 430.60ms, mfu 38.98%\n","iter 6330: loss 3.6194, time 429.14ms, mfu 39.01%\n","iter 6340: loss 3.9350, time 430.72ms, mfu 39.02%\n","iter 6350: loss 3.6140, time 430.84ms, mfu 39.02%\n","iter 6360: loss 3.7077, time 430.29ms, mfu 39.03%\n","iter 6370: loss 3.6825, time 430.31ms, mfu 39.04%\n","iter 6380: loss 3.7455, time 430.64ms, mfu 39.04%\n","iter 6390: loss 3.8944, time 430.19ms, mfu 39.05%\n","iter 6400: loss 3.7272, time 430.39ms, mfu 39.06%\n","iter 6410: loss 3.7175, time 430.25ms, mfu 39.06%\n","iter 6420: loss 3.6574, time 430.29ms, mfu 39.07%\n","iter 6430: loss 3.8706, time 430.09ms, mfu 39.08%\n","iter 6440: loss 3.7356, time 430.93ms, mfu 39.08%\n","iter 6450: loss 3.6319, time 430.11ms, mfu 39.08%\n","iter 6460: loss 3.6785, time 430.56ms, mfu 39.08%\n","iter 6470: loss 3.6186, time 430.43ms, mfu 39.08%\n","iter 6480: loss 3.9164, time 430.96ms, mfu 39.08%\n","iter 6490: loss 3.5449, time 430.45ms, mfu 39.08%\n","iter 6500: loss 3.8009, time 430.28ms, mfu 39.09%\n","iter 6510: loss 3.7837, time 430.33ms, mfu 39.09%\n","iter 6520: loss 3.5132, time 430.03ms, mfu 39.10%\n","iter 6530: loss 3.5953, time 430.42ms, mfu 39.10%\n","iter 6540: loss 3.7893, time 430.58ms, mfu 39.10%\n","iter 6550: loss 3.7899, time 429.68ms, mfu 39.10%\n","iter 6560: loss 3.8608, time 430.70ms, mfu 39.10%\n","iter 6570: loss 3.8129, time 430.80ms, mfu 39.10%\n","iter 6580: loss 3.5088, time 430.57ms, mfu 39.10%\n","iter 6590: loss 3.8250, time 431.24ms, mfu 39.09%\n","iter 6600: loss 3.6432, time 430.80ms, mfu 39.09%\n","iter 6610: loss 3.7315, time 430.68ms, mfu 39.09%\n","iter 6620: loss 3.8479, time 430.85ms, mfu 39.09%\n","iter 6630: loss 3.6817, time 430.45ms, mfu 39.09%\n","iter 6640: loss 3.5739, time 429.97ms, mfu 39.09%\n","iter 6650: loss 3.9177, time 430.58ms, mfu 39.09%\n","iter 6660: loss 3.9174, time 430.58ms, mfu 39.09%\n","iter 6670: loss 3.6979, time 431.08ms, mfu 39.09%\n","iter 6680: loss 3.6557, time 431.14ms, mfu 39.08%\n","iter 6690: loss 3.5994, time 430.41ms, mfu 39.09%\n","iter 6700: loss 3.6665, time 430.30ms, mfu 39.09%\n","iter 6710: loss 3.7107, time 430.49ms, mfu 39.09%\n","iter 6720: loss 3.7284, time 430.66ms, mfu 39.09%\n","iter 6730: loss 3.7269, time 430.22ms, mfu 39.09%\n","iter 6740: loss 3.8117, time 430.24ms, mfu 39.10%\n","iter 6750: loss 3.5915, time 430.82ms, mfu 39.09%\n","iter 6760: loss 3.8027, time 430.28ms, mfu 39.10%\n","iter 6770: loss 3.6505, time 430.76ms, mfu 39.09%\n","iter 6780: loss 3.8428, time 430.61ms, mfu 39.09%\n","iter 6790: loss 3.6891, time 430.92ms, mfu 39.09%\n","iter 6800: loss 3.6968, time 430.58ms, mfu 39.09%\n","iter 6810: loss 3.7272, time 430.74ms, mfu 39.09%\n","iter 6820: loss 3.6933, time 430.65ms, mfu 39.09%\n","iter 6830: loss 3.7214, time 430.88ms, mfu 39.09%\n","iter 6840: loss 3.7558, time 430.19ms, mfu 39.09%\n","iter 6850: loss 3.8148, time 431.08ms, mfu 39.09%\n","iter 6860: loss 3.3786, time 431.31ms, mfu 39.08%\n","iter 6870: loss 3.7973, time 431.06ms, mfu 39.08%\n","iter 6880: loss 3.4735, time 430.73ms, mfu 39.08%\n","iter 6890: loss 3.8057, time 430.90ms, mfu 39.08%\n","iter 6900: loss 3.6498, time 429.79ms, mfu 39.09%\n","iter 6910: loss 3.7115, time 430.34ms, mfu 39.09%\n","iter 6920: loss 3.6782, time 430.37ms, mfu 39.09%\n","iter 6930: loss 3.7799, time 429.71ms, mfu 39.10%\n","iter 6940: loss 3.6066, time 430.27ms, mfu 39.10%\n","iter 6950: loss 3.6371, time 429.63ms, mfu 39.11%\n","iter 6960: loss 3.4735, time 430.57ms, mfu 39.11%\n","iter 6970: loss 3.6317, time 430.84ms, mfu 39.10%\n","iter 6980: loss 3.5353, time 430.92ms, mfu 39.10%\n","iter 6990: loss 3.6446, time 430.39ms, mfu 39.10%\n","step 7000: train loss 3.6805, val loss 3.6828\n","saving checkpoint to out\n","iter 7000: loss 3.6986, time 14049.60ms, mfu 35.31%\n","iter 7010: loss 3.7259, time 429.86ms, mfu 35.69%\n","iter 7020: loss 3.7046, time 429.85ms, mfu 36.04%\n","iter 7030: loss 3.6104, time 429.74ms, mfu 36.35%\n","iter 7040: loss 3.7314, time 429.20ms, mfu 36.64%\n","iter 7050: loss 3.6377, time 429.51ms, mfu 36.90%\n","iter 7060: loss 3.6731, time 429.67ms, mfu 37.12%\n","iter 7070: loss 3.7091, time 429.80ms, mfu 37.33%\n","iter 7080: loss 3.6334, time 429.58ms, mfu 37.51%\n","iter 7090: loss 3.6785, time 430.26ms, mfu 37.67%\n","iter 7100: loss 3.7074, time 429.25ms, mfu 37.83%\n","iter 7110: loss 3.8828, time 430.59ms, mfu 37.95%\n","iter 7120: loss 3.6561, time 430.10ms, mfu 38.07%\n","iter 7130: loss 3.7491, time 430.87ms, mfu 38.17%\n","iter 7140: loss 3.7356, time 430.53ms, mfu 38.26%\n","iter 7150: loss 3.6037, time 430.55ms, mfu 38.35%\n","iter 7160: loss 3.7762, time 429.89ms, mfu 38.43%\n","iter 7170: loss 3.7095, time 430.51ms, mfu 38.50%\n","iter 7180: loss 3.7142, time 430.53ms, mfu 38.56%\n","iter 7190: loss 3.7956, time 430.35ms, mfu 38.61%\n","iter 7200: loss 3.5583, time 430.21ms, mfu 38.66%\n","iter 7210: loss 3.6821, time 430.29ms, mfu 38.71%\n","iter 7220: loss 3.5782, time 430.44ms, mfu 38.75%\n","iter 7230: loss 3.7801, time 430.62ms, mfu 38.78%\n","iter 7240: loss 3.2894, time 430.06ms, mfu 38.82%\n","iter 7250: loss 3.7514, time 430.13ms, mfu 38.85%\n","iter 7260: loss 4.1995, time 430.05ms, mfu 38.88%\n","iter 7270: loss 3.5563, time 430.54ms, mfu 38.90%\n","iter 7280: loss 3.7265, time 430.03ms, mfu 38.92%\n","iter 7290: loss 3.7779, time 430.36ms, mfu 38.94%\n","iter 7300: loss 3.6165, time 430.09ms, mfu 38.96%\n","iter 7310: loss 3.5439, time 429.98ms, mfu 38.98%\n","iter 7320: loss 3.8439, time 430.66ms, mfu 38.99%\n","iter 7330: loss 3.8353, time 429.76ms, mfu 39.01%\n","iter 7340: loss 3.7597, time 430.21ms, mfu 39.02%\n","iter 7350: loss 3.6476, time 430.57ms, mfu 39.03%\n","iter 7360: loss 3.6666, time 430.03ms, mfu 39.04%\n","iter 7370: loss 3.6803, time 430.64ms, mfu 39.04%\n","iter 7380: loss 3.7796, time 430.66ms, mfu 39.05%\n","iter 7390: loss 3.5207, time 430.70ms, mfu 39.05%\n","iter 7400: loss 3.5622, time 430.37ms, mfu 39.06%\n","iter 7410: loss 3.6578, time 430.45ms, mfu 39.06%\n","iter 7420: loss 3.5887, time 429.65ms, mfu 39.07%\n","iter 7430: loss 3.8107, time 430.96ms, mfu 39.07%\n","iter 7440: loss 3.5649, time 430.24ms, mfu 39.08%\n","iter 7450: loss 3.7135, time 431.23ms, mfu 39.07%\n","iter 7460: loss 3.5465, time 430.33ms, mfu 39.08%\n","iter 7470: loss 3.5661, time 430.43ms, mfu 39.08%\n","iter 7480: loss 3.6840, time 430.14ms, mfu 39.09%\n","iter 7490: loss 3.7627, time 430.33ms, mfu 39.09%\n","iter 7500: loss 3.4306, time 430.02ms, mfu 39.09%\n","iter 7510: loss 3.7785, time 430.01ms, mfu 39.10%\n","iter 7520: loss 3.7198, time 430.75ms, mfu 39.10%\n","iter 7530: loss 3.3831, time 430.78ms, mfu 39.09%\n","iter 7540: loss 3.8191, time 430.41ms, mfu 39.10%\n","iter 7550: loss 3.7563, time 430.29ms, mfu 39.10%\n","iter 7560: loss 3.8311, time 430.31ms, mfu 39.10%\n","iter 7570: loss 3.7755, time 429.97ms, mfu 39.10%\n","iter 7580: loss 3.6113, time 430.47ms, mfu 39.10%\n","iter 7590: loss 3.6722, time 430.66ms, mfu 39.10%\n","iter 7600: loss 3.6772, time 430.68ms, mfu 39.10%\n","iter 7610: loss 3.6301, time 430.44ms, mfu 39.10%\n","iter 7620: loss 3.5830, time 431.03ms, mfu 39.10%\n","iter 7630: loss 3.6590, time 430.60ms, mfu 39.10%\n","iter 7640: loss 3.7055, time 430.45ms, mfu 39.10%\n","iter 7650: loss 3.5430, time 430.50ms, mfu 39.10%\n","iter 7660: loss 3.7088, time 430.35ms, mfu 39.10%\n","iter 7670: loss 3.7300, time 430.62ms, mfu 39.10%\n","iter 7680: loss 3.6587, time 430.73ms, mfu 39.10%\n","iter 7690: loss 3.7770, time 430.47ms, mfu 39.10%\n","iter 7700: loss 3.4358, time 430.07ms, mfu 39.10%\n","iter 7710: loss 3.6760, time 430.61ms, mfu 39.10%\n","iter 7720: loss 3.6449, time 430.64ms, mfu 39.10%\n","iter 7730: loss 3.5846, time 431.10ms, mfu 39.09%\n","iter 7740: loss 3.5266, time 431.13ms, mfu 39.09%\n","iter 7750: loss 3.5448, time 430.73ms, mfu 39.09%\n","iter 7760: loss 3.7028, time 430.22ms, mfu 39.09%\n","iter 7770: loss 3.6750, time 430.60ms, mfu 39.09%\n","iter 7780: loss 3.6679, time 430.59ms, mfu 39.09%\n","iter 7790: loss 3.6191, time 430.95ms, mfu 39.09%\n","iter 7800: loss 3.6983, time 430.19ms, mfu 39.09%\n","iter 7810: loss 3.6173, time 430.83ms, mfu 39.09%\n","iter 7820: loss 3.5907, time 430.34ms, mfu 39.09%\n","iter 7830: loss 3.7805, time 430.96ms, mfu 39.09%\n","iter 7840: loss 3.5637, time 430.39ms, mfu 39.09%\n","iter 7850: loss 3.7611, time 430.23ms, mfu 39.09%\n","iter 7860: loss 3.5836, time 430.56ms, mfu 39.09%\n","iter 7870: loss 3.6745, time 430.19ms, mfu 39.10%\n","iter 7880: loss 3.6911, time 430.23ms, mfu 39.10%\n","iter 7890: loss 3.7991, time 430.59ms, mfu 39.10%\n","iter 7900: loss 3.7996, time 430.80ms, mfu 39.10%\n","iter 7910: loss 3.7214, time 430.49ms, mfu 39.10%\n","iter 7920: loss 3.5986, time 431.51ms, mfu 39.09%\n","iter 7930: loss 3.6531, time 429.76ms, mfu 39.10%\n","iter 7940: loss 3.7988, time 430.53ms, mfu 39.10%\n","iter 7950: loss 3.3372, time 431.03ms, mfu 39.09%\n","iter 7960: loss 3.5630, time 430.48ms, mfu 39.09%\n","iter 7970: loss 3.6788, time 430.80ms, mfu 39.09%\n","iter 7980: loss 3.5176, time 430.85ms, mfu 39.09%\n","iter 7990: loss 3.6613, time 430.64ms, mfu 39.09%\n","step 8000: train loss 3.6379, val loss 3.6468\n","saving checkpoint to out\n","iter 8000: loss 3.6028, time 14034.65ms, mfu 35.30%\n","iter 8010: loss 3.7772, time 430.24ms, mfu 35.68%\n","iter 8020: loss 3.7568, time 429.43ms, mfu 36.03%\n","iter 8030: loss 3.6571, time 428.90ms, mfu 36.35%\n","iter 8040: loss 3.8803, time 429.42ms, mfu 36.64%\n","iter 8050: loss 3.8032, time 429.71ms, mfu 36.89%\n","iter 8060: loss 3.7536, time 430.19ms, mfu 37.12%\n","iter 8070: loss 3.7109, time 429.72ms, mfu 37.32%\n","iter 8080: loss 3.6527, time 429.64ms, mfu 37.51%\n","iter 8090: loss 3.4868, time 430.13ms, mfu 37.67%\n","iter 8100: loss 3.6609, time 429.53ms, mfu 37.82%\n","iter 8110: loss 3.5788, time 429.50ms, mfu 37.96%\n","iter 8120: loss 3.5897, time 430.60ms, mfu 38.07%\n","iter 8130: loss 3.6613, time 430.54ms, mfu 38.17%\n","iter 8140: loss 3.6653, time 430.34ms, mfu 38.27%\n","iter 8150: loss 3.5536, time 430.19ms, mfu 38.35%\n","iter 8160: loss 3.6459, time 429.99ms, mfu 38.43%\n","iter 8170: loss 3.5151, time 429.96ms, mfu 38.50%\n","iter 8180: loss 3.4736, time 429.53ms, mfu 38.57%\n","iter 8190: loss 3.6900, time 430.18ms, mfu 38.63%\n","iter 8200: loss 3.8157, time 430.57ms, mfu 38.68%\n","iter 8210: loss 3.6373, time 430.41ms, mfu 38.72%\n","iter 8220: loss 3.5280, time 429.29ms, mfu 38.77%\n","iter 8230: loss 3.5093, time 430.54ms, mfu 38.80%\n","iter 8240: loss 3.6854, time 430.44ms, mfu 38.83%\n","iter 8250: loss 3.7061, time 430.57ms, mfu 38.86%\n","iter 8260: loss 3.5541, time 430.38ms, mfu 38.88%\n","iter 8270: loss 3.6252, time 430.22ms, mfu 38.91%\n","iter 8280: loss 3.6039, time 429.64ms, mfu 38.93%\n","iter 8290: loss 3.7665, time 430.24ms, mfu 38.95%\n","iter 8300: loss 3.6279, time 431.47ms, mfu 38.96%\n","iter 8310: loss 3.6509, time 431.12ms, mfu 38.97%\n","iter 8320: loss 3.5476, time 430.92ms, mfu 38.98%\n","iter 8330: loss 3.6499, time 430.73ms, mfu 38.99%\n","iter 8340: loss 3.5724, time 430.69ms, mfu 39.00%\n","iter 8350: loss 3.6098, time 430.51ms, mfu 39.01%\n","iter 8360: loss 3.5754, time 430.22ms, mfu 39.02%\n","iter 8370: loss 3.6235, time 430.60ms, mfu 39.03%\n","iter 8380: loss 3.2963, time 429.87ms, mfu 39.04%\n","iter 8390: loss 3.4999, time 430.51ms, mfu 39.04%\n","iter 8400: loss 3.5975, time 429.95ms, mfu 39.06%\n","iter 8410: loss 3.5746, time 430.63ms, mfu 39.06%\n","iter 8420: loss 3.8251, time 430.80ms, mfu 39.06%\n","iter 8430: loss 3.6662, time 429.88ms, mfu 39.07%\n","iter 8440: loss 3.4263, time 430.77ms, mfu 39.07%\n","iter 8450: loss 3.8341, time 430.41ms, mfu 39.07%\n","iter 8460: loss 3.5244, time 430.73ms, mfu 39.07%\n","iter 8470: loss 3.5590, time 431.27ms, mfu 39.07%\n","iter 8480: loss 3.6873, time 430.16ms, mfu 39.08%\n","iter 8490: loss 3.6041, time 430.61ms, mfu 39.08%\n","iter 8500: loss 3.7133, time 430.16ms, mfu 39.08%\n","iter 8510: loss 3.5951, time 430.83ms, mfu 39.08%\n","iter 8520: loss 3.7847, time 430.32ms, mfu 39.08%\n","iter 8530: loss 3.5593, time 430.51ms, mfu 39.09%\n","iter 8540: loss 3.7412, time 431.01ms, mfu 39.08%\n","iter 8550: loss 3.6200, time 430.23ms, mfu 39.09%\n","iter 8560: loss 3.6248, time 430.66ms, mfu 39.09%\n","iter 8570: loss 3.4083, time 430.00ms, mfu 39.09%\n","iter 8580: loss 3.3688, time 430.19ms, mfu 39.10%\n","iter 8590: loss 3.5328, time 431.85ms, mfu 39.08%\n","iter 8600: loss 3.4865, time 430.47ms, mfu 39.09%\n","iter 8610: loss 3.5948, time 430.07ms, mfu 39.09%\n","iter 8620: loss 3.6111, time 429.73ms, mfu 39.10%\n","iter 8630: loss 3.6143, time 430.20ms, mfu 39.10%\n","Process ForkProcess-12:\n","Process ForkProcess-10:\n","Process ForkProcess-11:\n","Process ForkProcess-8:\n","Process ForkProcess-9:\n","Process ForkProcess-6:\n","Process ForkProcess-3:\n","Process ForkProcess-7:\n","Process ForkProcess-2:\n","Process ForkProcess-1:\n","Process ForkProcess-4:\n","Process ForkProcess-5:\n","Error in sys.excepthook:\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/linecache.py\", line 72, in checkcache\n","Traceback (most recent call last):\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","Traceback (most recent call last):\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","KeyboardInterrupt\n","Traceback (most recent call last):\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","KeyboardInterrupt\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 103, in get\n","    res = self._recv_bytes()\n","  File \"/usr/lib/python3.10/multiprocessing/connection.py\", line 216, in recv_bytes\n","    buf = self._recv_bytes(maxlength)\n","  File \"/usr/lib/python3.10/multiprocessing/connection.py\", line 414, in _recv_bytes\n","    buf = self._recv(4)\n","  File \"/usr/lib/python3.10/multiprocessing/connection.py\", line 379, in _recv\n","    chunk = read(handle, remaining)\n","KeyboardInterrupt\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","    stat = os.stat(fullname)\n","KeyboardInterrupt\n","\n","Original exception was:\n","Traceback (most recent call last):\n","  File \"/content/drive/MyDrive/nanoGPT/train.py\", line 295, in <module>\n","Traceback (most recent call last):\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 314, in _bootstrap\n","    self.run()\n","  File \"/usr/lib/python3.10/multiprocessing/process.py\", line 108, in run\n","    self._target(*self._args, **self._kwargs)\n","  File \"/usr/lib/python3.10/concurrent/futures/process.py\", line 240, in _process_worker\n","    call_item = call_queue.get(block=True)\n","  File \"/usr/lib/python3.10/multiprocessing/queues.py\", line 102, in get\n","    with self._rlock:\n","  File \"/usr/lib/python3.10/multiprocessing/synchronize.py\", line 95, in __enter__\n","    return self._semlock.__enter__()\n","KeyboardInterrupt\n","    logits, loss = model(X, Y)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n","    return forward_call(*args, **kwargs)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py\", line 82, in forward\n","    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py\", line 209, in _fn\n","    return fn(*args, **kwargs)\n","  File \"/content/drive/MyDrive/nanoGPT/model.py\", line 177, in forward\n","    def forward(self, idx, targets=None):\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py\", line 209, in _fn\n","    return fn(*args, **kwargs)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 2819, in forward\n","    return compiled_fn(full_args)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 1222, in g\n","    return f(*args)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 2386, in debug_compiled_function\n","    return compiled_function(*args)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 1898, in runtime_wrapper\n","    all_outs = call_func_with_args(\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 1247, in call_func_with_args\n","    out = normalize_as_list(f(args))\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 1222, in g\n","    return f(*args)\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py\", line 506, in apply\n","    return super().apply(*args, **kwargs)  # type: ignore[misc]\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 2151, in forward\n","    fw_outs = call_func_with_args(\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py\", line 1247, in call_func_with_args\n","    out = normalize_as_list(f(args))\n","  File \"/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py\", line 248, in run\n","    return model(new_inputs)\n","  File \"/tmp/torchinductor_root/l7/cl7ae4dx7ltgcfuozdnfv3bfa6v5imv2yu2trza36z5n5iczzqvu.py\", line 1299, in call\n","    buf144 = empty_strided((12288, 768), (768, 1), device='cuda', dtype=torch.bfloat16)\n","KeyboardInterrupt\n","\u001b[34m\u001b[1mwandb\u001b[0m: Waiting for W&B process to finish... \u001b[31m(failed 255).\u001b[0m\n","\u001b[34m\u001b[1mwandb\u001b[0m: \n","\u001b[34m\u001b[1mwandb\u001b[0m: Run history:\n","\u001b[34m\u001b[1mwandb\u001b[0m:       iter ▁▂▃▄▅▅▆▇█\n","\u001b[34m\u001b[1mwandb\u001b[0m:         lr ▁▅███████\n","\u001b[34m\u001b[1mwandb\u001b[0m:        mfu ▁████████\n","\u001b[34m\u001b[1mwandb\u001b[0m: train/loss █▂▂▁▁▁▁▁▁\n","\u001b[34m\u001b[1mwandb\u001b[0m:   val/loss █▂▂▁▁▁▁▁▁\n","\u001b[34m\u001b[1mwandb\u001b[0m: \n","\u001b[34m\u001b[1mwandb\u001b[0m: Run summary:\n","\u001b[34m\u001b[1mwandb\u001b[0m:       iter 8000\n","\u001b[34m\u001b[1mwandb\u001b[0m:         lr 0.0006\n","\u001b[34m\u001b[1mwandb\u001b[0m:        mfu 39.08811\n","\u001b[34m\u001b[1mwandb\u001b[0m: train/loss 3.63786\n","\u001b[34m\u001b[1mwandb\u001b[0m:   val/loss 3.6468\n","\u001b[34m\u001b[1mwandb\u001b[0m: \n","\u001b[34m\u001b[1mwandb\u001b[0m: You can sync this run to the cloud by running:\n","\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[1mwandb sync /content/drive/MyDrive/nanoGPT/wandb/offline-run-20230430_043331-cgy66d0h\u001b[0m\n","\u001b[34m\u001b[1mwandb\u001b[0m: Find logs at: \u001b[35m\u001b[1m./wandb/offline-run-20230430_043331-cgy66d0h/logs\u001b[0m\n","^C\n"]}]},{"cell_type":"code","source":["!python sample.py --out_dir=out"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ol5khpZMONNp","executionInfo":{"status":"ok","timestamp":1682833159336,"user_tz":-540,"elapsed":58352,"user":{"displayName":"Sue Lee","userId":"12052451563593377451"}},"outputId":"4cc15603-9445-4c9c-e20e-5b1e6d6efd76"},"execution_count":11,"outputs":[{"output_type":"stream","name":"stdout","text":["Overriding: out_dir = out\n","number of parameters: 123.59M\n","No meta.pkl found, assuming GPT-2 encodings...\n","\n","\n","The court heard testimony that they have made false statements so far as they are concerned about the possibility of a “very high degree of harm” to their client’s health.\n","\n","“What they did was very good news,” said District 2 appellate judge John Callahan, who spoke with the court. “We have said that as a client, it is actually very difficult for the client to consider a recommendation that such behavior would be tolerated on a consistent basis.”\n","\n","The court heard that the judge stressed that the defendant had more control over the outcome of a preliminary hearing than what the defendant had considered because of the physical harm to the client.\n","\n","The court heard that the defendant had more control over the outcome of a preliminary hearing than what the defendant had considered because of the physical harm to the client.\n","\n","“The defendant had more control over the outcome of the initial hearing,” said the court.\n","\n","Both were still in custody for 15 days, the court heard.\n","\n","“The defendant had more control over the outcome of a preliminary hearing,” the court heard.\n","\n","The judge heard that the defendant had more control over the outcome of a preliminary hearing, the court heard.\n","\n","“The defendant had very strong control over the outcome of a preliminary hearing,” the court heard. “The defendant had more control over the outcome of a preliminary hearing,” the court heard.\n","\n","When asked to describe the defendant’s doctorate, the court heard that the court heard that the prosecution had a strong understanding that the defendant had more control over the outcome of a preliminary hearing.\n","\n","The court heard that trial lawyers said that the sentence, which would have to be read by a defendant, “is of no value to the client.”\n","\n","The court heard that the defendant had a strong understanding that the defendant had more control over the outcome of a preliminary hearing, “which would have to be read by a defendant” and “which an defendant” would have to consider whether the defendant was guilty of one of the charges.\n","\n","We heard that the defendant had more control over the outcome of a preliminary hearing, and that trial lawyers did not have the right to appeal the application.\n","\n","“The defendant’s doctorate was his patient who was not committed to a trial, and he was not harmed\n","---------------\n","\n","\n","From the U.S. military’s perspective, we can’t help but feel empathy for its people. But there are so many other things we can do better, too – what can happen when an enemy has a bad conscience?\n","\n","The question is: Will a people understand what they want?\n","\n","When, when and how we engage, we can reduce or limit the numbers of people who are dependent on the government to understand what our government is doing. But a lot of people are still motivated to participate in the kind of public-identity that comes with the political system.\n","\n","In a sense, people must help – and this is a question that people who are not prepared for, and one of the reasons that I’m hopeful about, or do I think about.\n","\n","“Do you want to kill people?” asked a senior Pentagon veteran, an expert in non-governmental affairs and a friend of mine who was directly involved in a domestic conflict in the beginning.\n","\n","“Are you from the Middle East?” asked the senior, one of the senior military soldiers who was involved in the first U.S. military strike (in Iraq in August 2009) directed by Secretary of Defense Leon Panetta.\n","\n","I was shocked by how uncomfortable the U.S. military has been as a force for many years, and I think that’s why there are so many involved, and the challenge lies in their ability to keep their forces in control of one’s mind. The world is in a transition, and the world is in a transition period, and the battle to keep one’s mind is between them.\n","\n","Just going home, I was a month and a half ago, a month and a half ago, there was a war between Saddam Hussein and the Islamic State group and the Islamic State group, and how we spent our military in the Middle East. We told our chief of staff, Robert S. Kirk, that we’re going to try to protect our military by moving on with the other side of the war. We gave all of this division to the Islamic State group to work with.\n","\n","In the U.S., we weren’t convinced that the Islamic State group in the Middle East was ever going to do this operation.\n","\n","But at the same time, we, with our approach and our ability, in this case, are\n","---------------\n","\n","\n","A second-choice Senate Democrat, Brian J. Rangel of Florida, had hoped to fight for the Republican nomination, but the Republican floor felt more pressure on the nomination.\n","\n","“In the Senate, the Democratic Party has gone in multiple directions, and I don’t see it that way,” Rangel said.\n","\n","Shaman also said she opposed the nomination of Sen. Mike Duffy, whose firm was engaged in bribery and corruption in the early stages of the fight.\n","\n","“That is not the case,” he said. “It is not a criminal case, for the Senate, and what we do are not going to do in this election, and we do not support nor oppose a candidate like Duffy.”<|endoftext|>The Trump administration is among the first to admit the president's ability to exert his influence in this crucial election. In August, Donald Trump's lawyers argued that he was able to influence the election as part of his campaign's transition team, not by a wide margin, but by his own party and by other candidates.\n","\n","Trump administration officials denied the claims in an interview with The MMAFighting.com about the president's progress. House of Representatives Chuck Schumer and Lindsey Graham unveiled a report that said that the president could make a decision about whether he or the president would participate in the transition process.\n","\n","A number of advisers said that Trump's transition team was very \"developing\" to the president, and the president would support the transition in 2016.\n","\n","Republican presidential nominee Hillary Clinton pledged in August to the campaign that the president would \"deceive and have great respect for the American people.\"\n","\n","Reaction reporter Joe Barton contributed to this report.<|endoftext|>Jodi Meli/Getty Images\n","\n","I’m at war with Canada. I want to be a little more optimistic. I’m staying clear. I want to be a little more pessimistic.\n","\n","The Canadian National Intelligence Agency is worried about pulling out the Paris climate accord.\n","\n","I and many other European governments, particularly the United States, have significant problems. Here, we have a tiny sampling of a few countries — including Germany, the United Kingdom, Denmark, Portugal, Brunei, and Indonesia — that is concerned about decreasing global warming and not making the United States any more effective.\n","\n","The United States is the world’s fifth-largest economy, and the United States is the world’\n","---------------\n","\n","\n","Also on HuffPost:\n","\n","10. Who did you describe him in person?\n","\n","I don't know\n","\n","I don't know\n","\n","If you are all over, I hope you want to consider this topic as an issue. I want to talk you over to some of the interesting material we've put out.\n","\n","11. What are the challenges of being a seasoned reporter for The New York Times?\n","\n","I sit down with some of his friends over at The New York Times, and I try to explain how he has some pretty fantastic stories about his work. And I see it from him!\n","\n","13. Can you say what you've read in your mind?\n","\n","I've had a great conversation with a friend over at WKMG. I've been a long time with him as a reporter for The New York Times, too. He's quite a talented citizen, and looks like an awkward guy.\n","\n","14. How's the money going to help him?\n","\n","No, I'll take you over a few minutes and talk about how he's interesting in the editing room.\n","\n","15. What's the point of the conversation?\n","\n","I'm going to talk about the need for him to focus on the situation, and he's the one saying \"She wants to kill me.\"\n","\n","15. He's the biggest fan of his life?\n","\n","I have to deal with his problems with the guy he's actually the biggest fan of. I have to get a clean reporter every Sunday.\n","\n","16. Can you say what you've read in the past?\n","\n","I don't know. He's so good. He's like a normal guy. I have to talk to him a bit. I use him on Monday and Saturday mornings, and I do believe sometimes I'm just just going to keep up with him.\n","\n","17. How do I talk about him on CNN?\n","\n","I don't know. I have a little bit of an obsession with him. There's so much talk about him. The one thing that's interesting about him is that he's very funny. That's when he's talking about it. He's very cool.\n","\n","18. What is the hardest things going on in journalism?\n","\n","I've been in journalism for quite some time. I had a hard time staying completely out of the newsroom. I was one of the most experienced people in journalism as a\n","---------------\n","\n","\n","The company that controls the virus was seen by the public as a more difficult target for the virus.\n","\n","In a statement to The Sun that day, the agency described the virus as its first known target, the Korean National Institutes of Health and the Ministry of Cultural Affairs.\n","\n","\"This particular target will be the most acute and potentially deadly virus in the world,\" the statement said.\n","\n","\"The first known virus in the world was observed with a virus consistent with the virus, but with the case of the disease in particular, it was likely that it would have been a contagious disease.\n","\n","\"The malware can cause a host of other diseases, such as cholera, dori and doriemia, such as cholera, dori, dori and dori, and was a target of transmission.\n","\n","\"The other virus in the world is the virus that causes a host of diseases in North Korea,\" the statement went on.\n","\n","The virus was registered with the Ministry of Cultural Affairs and further confirmed that it had been discovered.\n","\n","\"This is an extremely robust development for the virus,\" the statement added.\n","\n","The Health Ministry, the security agency, and the Ministry of Cultural Affairs of North Korea issued an assessment that the virus had been found in a range of regions.\n","\n","\"The information presented at the conclusion of the report is that a number of factors were in its way related to the virus,\" the announcement said.\n","\n","All reports of the virus were in the public domain.\n","\n","\"The investigators have been told that the virus is a virus that spreads by the virus and that it may cause the disease.\n","\n","\"This conclusion is based on the fact that the virus shows the virus's true function,\" the statement added.<|endoftext|>\"This is our first big big push for the next album,\" frontman and U.S. rock icon, who made his first Top 10 albums in May; the last time it was live, won the Billboard charts across the country. \"We've been at a point where the album was going to make the record that we've done a million times,\" he said in his opening interview with The Daily Mirror on Monday evening.\n","\n","\"I think when I hear myself, I think of the album that I made, and I think of the songs that I was creating. And once I hit the record, I think of the record, the way that I would treat people with\n","---------------\n","\n","\n","Michester said the plan is to do away with long range of things, such as the speed of light on a project that would more than half its annual budget. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he added.\n","\n","The plan would be to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent, he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a proposed project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he said. “The plan is to do away with long range things, such as the speed of light on a project that would more than half its annual budget is spent,” he added. “The plan is to do away with\n","---------------\n","\n","\n","While the campaign had been unsuccessful, the group’s campaign argued it seemed to be the only way to convince the outcome.\n","\n","“If the data was not accurate, it is to protect the poll data,” said Christopher O’Neil, head of the Democratic Party of Iowa.\n","\n","“They did not want to have an election that is open to their interpretation.”\n","\n","But in a statement released through the Iowa Elections Commission on Tuesday night, the group said that the result wasn’t to protect the voters’ campaign and the voter.\n","\n","“Ultimately, our objective in doing so would be to give the voters a voice to vote,” said the statement.\n","\n","“We will continue to work with our supporters in an manner that is open to their interpretation and that is open to their interpretation.”\n","\n","The statement also said that the election had not been successful as it was an “unjust” campaign.\n","\n","A Democratic National Committee spokeswoman, on condition of anonymity, said a result of the Iowa Elections Commission’s decision wasn’t a “partisan stunt” to vote against the results of the 2012 GOP presidential election.<|endoftext|>The British Museum finds the museum in need of a museum-design order that will allow museums to design their own galleries, with no permission from the museum.\n","\n","The museum is hoping to build a museum that is capable of performing fine arts pieces and a museum that can't be used for any purpose.\n","\n","The former British Museum director, Robert J. Hart, said it was not the first time that Hart had made it any easier. But in 2003, it got a major positive reception from the museums and the museum.\n","\n","A top American museum director, the museum says that the museum would be designed with the necessary permission of a museum to hand over each of its galleries.\n","\n","\"Once the museum’s design process is complete and the work is completed, it is obviously prohibitive for several museums,\" he said. \"I have a lot of concerns about this really.\n","\n","\"I think it’s a question of buying a museum that is not as open to visitors as I would have liked it to be. It’s a question of whether or not most museums will go out of business.\"\n","\n","He also reiterated that museums should not be relying on the \"re-design\n","---------------\n","\n","Swan O’Rourke, a former President who was president of the Council on American-Islamic Relations, should still be put in the water–\n","\n","“No idea,” he said in a recent interview. “When he got to the end of his term, he said, ‘If I could have done this, I would have preferred in the past.’”\n","\n","“He was in an uproar a few days after he was called,” wrote the president. “But he didn’t get out.”\n","\n","The president’s decision to stop criticizing the president’s decision to move forward would have been a significant blow toward the president’s policies.\n","\n","“I’ve found a way to get the president to sign a pledge aimed at the Islamists–and we’ll allow that in too soon.”\n","\n","But the president’s decision to move forward’s positions on “the battle against the Syrian government that is in the cards now.”\n","\n","“We’ve got to be realistic about that,” said O’Rourke, whose comments are interpreted by a “moderate” and “moderate” tone.\n","\n","“I was surprised by what he said.”\n","\n","“What’s so important,” said Hanselyar of the UN’s Institute of Peacekeeping. “I’m stunned by it.”\n","\n","“It’s a tough debate,” said O’Rourke. “It’s a very important issue. And I’m shocked that he’s not going to step out of that trap.”\n","\n","Birley, who was formerly secretary of the UN General Assembly from 1993 to 2011, has been repeatedly criticized for his opposition to the violence in Syria.\n","\n","“The latest poll released by the UN Center for the Study of Human Rights suggests that the opposition is more liberal,” he said. “It’s a much more positive assessment of his views.”\n","\n","In 2012, he said, “I’ve mentioned that when I came to the UN, there was no evidence that there was any evidence that there had been any opposition groups,” he\n","---------------\n","\n","\n","As a rookie, I said, “You don’t know what you’re doing.” As a rookie, the practice has gotten worse. In college, I was a sophomore.\n","\n","I also don’t know any of that.\n","\n","I’m not a rookie. I had a bad team. I had some issues.\n","\n","I was supposed to be a part of the game because I had a student who was not really that good.\n","\n","I had another guy who played a bunch of games and was doing a great job. I was in college, one of my primary goals — “I did a great job, but I ended up doing something amazing,” as the sophomore told me.\n","\n","I don’t know any of that. But I wouldn’t have to worry about it at all. I had a bad year.\n","\n","I sat down with a lot of other players, and I was like, “Oh, that’s like I’m not sure you’re doing anything, right?”\n","\n","I went straight in the morning and said, “You’re not really going to get my kids to play around with you.”\n","\n","I walked over to that big guy, and he is hilarious.\n","\n","When he came back to me, he would walk away.\n","\n","I said, “You’re just like, I’ll give you some information.”\n","\n","One of the things I did was that he said “You’re not going to get your kids to play around with me.”\n","\n","I said he was a little nervous because I was scared.\n","\n","I really don’t want it to be anything to start a game. And I don’t want that here.\n","\n","More or less, I said, “It’s crazy, I don’t know.”\n","\n","I said, “Yes, we’re out here playing football.”\n","\n","In doing that, I was like, “Is that a good thing?”\n","\n","I said, “Oh, I’m a fan.”\n","\n","I said, “I’m sure we’ll play.”\n","\n","But to the point\n","---------------\n","\n","\n","The same goes for the same city in the western city but for the same city in Colorado.\n","\n","In particular, the city has a very diverse library. It allows for a mix of books, movies, documentaries and a variety of other kinds.\n","\n","What's more intimate is how I'm playing an ice hockey game as the metro is in the downtown area. I'm finding a different ways of playing local TV outside of Seattle.\n","\n","Since there's a lot of noise in the form of the game, it's been a good week.\n","\n","[Source: San Francisco Tech Tech.<|endoftext|>Equality\n","\n","I am a young man with high levels of difficulty in sports and basketball. After the first season of my senior year as a high school wrestling student, I learned something new about myself that I think will eventually lead me to to a college sports or basketball tournament in the future.\n","\n","My expectations were high for me, but I also learned more about the player this season.\n","\n","My goals were my first years at the school; I hit a high 1.2 goals per season.\n","\n","I competed in the 1st Challenge as a member of an elite group.\n","\n","I came under the impression that the most impactful players were on the outside of a level that I hadn't seen before.\n","\n","What I saw:\n","\n","A high 1.2 goals per season.\n","\n","Equality was a big part of my student life. He was a fan of Big Brother in the late 70s.\n","\n","I am a football fan, and I believe I'll be the greatest. I would like to play basketball for the more experienced.\n","\n","I have an annual ranking of players from both conferences, but I haven't yet got a one-year high enough to play basketball. I will never go to another high level.\n","\n","I'd also seen dozens more games of basketball as I've participated. The first game I was interested in a basketball game. The first game I played was a basketball game in the late 20s.\n","\n","I knew it was a great game but I saw the game as a high 1.2.\n","\n","I met my first coach and I said, 'Listen, I've got to concentrate on my basketball in the future.' I asked about my goals in the near future. He said, 'I want to make this my second year as a high school wrestling student. Are you still going\n","---------------\n"]}]},{"cell_type":"code","source":[],"metadata":{"id":"PKIScy3IOmiB"},"execution_count":null,"outputs":[]}]}