$ python replay_buffer.py --workers 4

[boot] sandbox worker-3 attached, pid 4172

[info] step 1180 loss 2.3141 780/s

[info] step 1200 loss 2.3024 833/s

[info] step 1220 loss 2.2907 798/s

[info] step 1240 loss 2.2790 851/s

[info] step 1260 loss 2.2673 816/s

[info] step 1280 loss 2.2556 781/s

[info] checkpoint shard 12/64 written

[info] step 1300 loss 2.2439 834/s

[info] step 1320 loss 2.2322 799/s

[info] step 1340 loss 2.2205 852/s

[info] step 1360 loss 2.2088 817/s

[info] step 1380 loss 2.1971 782/s

[info] step 1400 loss 2.1854 835/s

[info] step 1420 loss 2.1737 800/s

[warn] queue depth 96, consumer lagging

[info] step 1440 loss 2.4188 612/s

[warn] gc pause 140 ms, heap 71%

[info] step 1460 loss 3.0912 404/s

[warn] queue depth 288, dropping batches

malloc: arena 2 grew to 256 MB

[info] step 1480 loss 7.4451 212/s

[warn] worker thread 9 forked, unjoined

[warn] queue depth 516, dropping batches

malloc: arena 3 grew to 640 MB

[warn] worker thread 12 forked, unjoined

[warn] queue depth 804, dropping batches

malloc: arena 4 grew to 1024 MB

[warn] worker thread 15 forked, unjoined

[warn] queue depth 1092, dropping batches

malloc: arena 5 grew to 1408 MB

[err] EAGAIN spawning thread 31

[warn] worker thread 33 forked, unjoined

malloc: rss 1.66 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1500 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 37

[warn] worker thread 39 forked, unjoined

malloc: rss 1.76 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1502 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 43

[warn] worker thread 45 forked, unjoined

malloc: rss 1.87 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1504 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 49

[warn] worker thread 51 forked, unjoined

malloc: rss 1.98 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1506 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 55

[warn] worker thread 57 forked, unjoined

malloc: rss 2.09 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1508 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 61

[warn] worker thread 63 forked, unjoined

malloc: rss 2.20 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1510 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 67

[warn] worker thread 69 forked, unjoined

malloc: rss 2.30 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1512 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 73

[warn] worker thread 75 forked, unjoined

malloc: rss 2.41 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1514 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 79

[warn] worker thread 81 forked, unjoined

malloc: rss 2.52 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1516 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 85

[warn] worker thread 87 forked, unjoined

malloc: rss 2.63 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1518 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 91

[warn] worker thread 93 forked, unjoined

malloc: rss 2.74 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1520 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 97

[warn] worker thread 99 forked, unjoined

malloc: rss 2.84 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1522 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 103

[warn] worker thread 105 forked, unjoined

malloc: rss 2.95 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1524 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 109

[warn] worker thread 111 forked, unjoined

malloc: rss 3.06 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1526 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 115

[warn] worker thread 117 forked, unjoined

malloc: rss 3.17 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1528 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 121

[warn] worker thread 123 forked, unjoined

malloc: rss 3.28 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1530 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 127

[warn] worker thread 129 forked, unjoined

malloc: rss 3.38 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1532 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 133

[warn] worker thread 135 forked, unjoined

malloc: rss 3.49 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1534 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 139

[warn] worker thread 141 forked, unjoined

malloc: rss 3.60 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1536 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 145

[warn] worker thread 147 forked, unjoined

malloc: rss 3.71 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1538 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 151

[warn] worker thread 153 forked, unjoined

malloc: rss 3.82 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1540 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 157

[warn] worker thread 159 forked, unjoined

malloc: rss 3.92 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1542 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 163

[warn] worker thread 165 forked, unjoined

malloc: rss 4.03 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1544 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 169

[warn] worker thread 171 forked, unjoined

malloc: rss 4.14 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1546 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 175

[warn] worker thread 177 forked, unjoined

malloc: rss 4.25 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1548 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 181

[warn] worker thread 183 forked, unjoined

malloc: rss 4.36 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1550 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 187

[warn] worker thread 189 forked, unjoined

malloc: rss 4.46 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1552 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 193

[warn] worker thread 195 forked, unjoined

malloc: rss 4.57 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1554 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 199

[warn] worker thread 201 forked, unjoined

malloc: rss 4.68 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1556 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 205

[warn] worker thread 207 forked, unjoined

malloc: rss 4.79 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1558 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 211

[warn] worker thread 213 forked, unjoined

malloc: rss 4.90 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1560 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 217

[warn] worker thread 219 forked, unjoined

malloc: rss 5.00 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1562 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 223

[warn] worker thread 225 forked, unjoined

malloc: rss 5.11 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1564 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 229

[warn] worker thread 231 forked, unjoined

malloc: rss 5.22 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1566 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 235

[warn] worker thread 237 forked, unjoined

malloc: rss 5.33 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1568 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 241

[warn] worker thread 243 forked, unjoined

malloc: rss 5.44 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1570 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 247

[warn] worker thread 249 forked, unjoined

malloc: rss 5.54 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1572 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 253

[warn] worker thread 255 forked, unjoined

malloc: rss 5.65 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1574 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 259

[warn] worker thread 261 forked, unjoined

malloc: rss 5.76 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1576 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 265

[warn] worker thread 267 forked, unjoined

malloc: rss 5.87 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1578 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 271

[warn] worker thread 273 forked, unjoined

malloc: rss 5.98 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1580 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 277

[warn] worker thread 279 forked, unjoined

malloc: rss 6.08 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1582 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 283

[warn] worker thread 285 forked, unjoined

malloc: rss 6.19 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1584 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 289

[warn] worker thread 291 forked, unjoined

malloc: rss 6.30 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1586 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 295

[warn] worker thread 297 forked, unjoined

malloc: rss 6.41 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1588 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 301

[warn] worker thread 303 forked, unjoined

malloc: rss 6.52 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1590 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 307

[warn] worker thread 309 forked, unjoined

malloc: rss 6.62 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1592 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 313

[warn] worker thread 315 forked, unjoined

malloc: rss 6.73 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1594 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 319

[warn] worker thread 321 forked, unjoined

malloc: rss 6.84 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1596 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 325

[warn] worker thread 327 forked, unjoined

malloc: rss 6.95 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1598 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 331

[warn] worker thread 333 forked, unjoined

malloc: rss 7.06 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1600 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 337

[warn] worker thread 339 forked, unjoined

malloc: rss 7.16 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1602 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 343

[warn] worker thread 345 forked, unjoined

malloc: rss 7.27 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1604 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 349

[warn] worker thread 351 forked, unjoined

malloc: rss 7.38 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1606 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 355

[warn] worker thread 357 forked, unjoined

malloc: rss 7.49 GB, gc pause 250 ms

[err] cannot allocate memory, retry 1/3

[info] step 1608 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 361

[warn] worker thread 363 forked, unjoined

malloc: rss 7.60 GB, gc pause 460 ms

[err] cannot allocate memory, retry 1/3

[info] step 1610 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 367

[warn] worker thread 369 forked, unjoined

malloc: rss 7.70 GB, gc pause 355 ms

[err] cannot allocate memory, retry 1/3

[info] step 1612 loss nan 0/s

[err] heartbeat missed, supervisor unreachable

[err] EAGAIN spawning thread 373

[warn] worker thread 375 forked, unjoined

malloc: rss 7.81 GB, gc pause 250 ms

[err] oom: kill process 4172 or sacrifice child