Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.........................XX.X.......] duration: 2.952511303s FAIL (2025-07-27 01:00:59 UTC) orbiter warning: context deadline exceeded Duration: 2.952537252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.XXXX.XX......XXXXX...XX.X...X............X...........] duration: 2.275610716s FAIL (2025-07-27 01:00:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.275650217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [....X..X..XX.XX.XXX..XX..X.X.X.XX....................X......] duration: 3.104448424s FAIL (2025-07-27 01:00:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.104481129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 920.467857ms FAIL (2025-07-27 01:00:32 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 920.526188ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X......] duration: 2.303199794s FAIL (2025-07-27 01:00:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.303236535s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.....X..X..XX.XX.XXX..XX..X.X.X.XX....................X.....] duration: 5.608206207s FAIL (2025-07-27 01:00:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.608237764s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X......XXXX..XX...X.....X.X....X...............X...........] duration: 2.628479598s FAIL (2025-07-27 01:00:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.628503581s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................X.........................XX.X......] duration: 3.03476444s FAIL (2025-07-27 01:00:10 UTC) orbiter warning: context deadline exceeded Duration: 3.034793451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 883.406704ms FAIL (2025-07-27 01:00:07 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 883.512379ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X.....] duration: 2.270502755s FAIL (2025-07-27 00:59:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.270539211s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.XXXX.XX......XXXXX...XX.X...X............X..........] duration: 2.401983556s FAIL (2025-07-27 00:59:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402018017s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......XXXX..XX...X.....X.X....X...............X..........] duration: 2.597281611s FAIL (2025-07-27 00:59:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.597305724s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 895.845733ms FAIL (2025-07-27 00:59:14 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 895.890026ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X....] duration: 16.899684856s FAIL (2025-07-27 00:59:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.899715692s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [......X..X..XX.XX.XXX..XX..X.X.X.XX....................X....] duration: 5.274738441s FAIL (2025-07-27 00:59:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.274881813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.........................XX.X.....] duration: 5.64796676s FAIL (2025-07-27 00:59:04 UTC) orbiter warning: context deadline exceeded Duration: 5.648236863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X......XXXX..XX...X.....X.X....X...............X.........] duration: 4.143627102s FAIL (2025-07-27 00:58:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.143740489s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X.XXXX.XX......XXXXX...XX.X...X............X.........] duration: 2.405860356s FAIL (2025-07-27 00:58:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405893777s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X......XXXX..XX...X.....X.X....X...............X........] duration: 2.824405839s FAIL (2025-07-27 00:58:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.8244307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X......X..X..XX.XX.XXX..XX..X.X.X.XX....................X...] duration: 3.016965868s FAIL (2025-07-27 00:58:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.017005398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.........................XX.X....] duration: 2.884858981s FAIL (2025-07-27 00:58:18 UTC) orbiter warning: context deadline exceeded Duration: 2.88489485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X.XXXX.XX......XXXXX...XX.X...X............X........] duration: 2.412522865s FAIL (2025-07-27 00:58:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41255801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.X................................................X........] duration: 846.093025ms FAIL (2025-07-27 00:58:06 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 846.145503ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [X....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X...] duration: 2.279580256s FAIL (2025-07-27 00:57:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.279611732s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XX......X..X..XX.XX.XXX..XX..X.X.X.XX....................X..] duration: 3.384881895s FAIL (2025-07-27 00:57:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.384917862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X.XXXX.XX......XXXXX...XX.X...X............X.......] duration: 2.340272554s FAIL (2025-07-27 00:57:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340291755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.........................XX.X...] duration: 2.818159967s FAIL (2025-07-27 00:57:22 UTC) orbiter warning: context deadline exceeded Duration: 2.818189233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....X......XXXX..XX...X.....X.X....X...............X.......] duration: 2.603487298s FAIL (2025-07-27 00:57:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.603519906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.X................................................X.......] duration: 851.873283ms FAIL (2025-07-27 00:57:13 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 851.903245ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........X.XXXX.XX......XXXXX...XX.X...X............X......] duration: 2.313688191s FAIL (2025-07-27 00:56:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.313731549s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................X.........................XX.X..] duration: 2.940858282s FAIL (2025-07-27 00:56:52 UTC) orbiter warning: context deadline exceeded Duration: 2.940906073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.XX......X..X..XX.XX.XXX..XX..X.X.X.XX....................X.] duration: 13.945369866s FAIL (2025-07-27 00:56:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.945394282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X..] duration: 4.47791863s FAIL (2025-07-27 00:56:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 4.477945036s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..XX......X..X..XX.XX.XXX..XX..X.X.X.XX....................X] duration: 1m0.108940954s FAIL (2025-07-27 00:56:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.10896733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X.] duration: 17.242315294s FAIL (2025-07-27 00:56:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 17.242344924s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.........................XX.X.] duration: 27.321013933s FAIL (2025-07-27 00:56:30 UTC) orbiter warning: context deadline exceeded Duration: 27.321029643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.X................................................X......] duration: 825.848018ms FAIL (2025-07-27 00:56:22 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 825.879069ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX....X......XXXX..XX...X.....X.X....X...............X......] duration: 2.694556633s FAIL (2025-07-27 00:56:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.694585129s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.........................XX.X] duration: 1m0.001641798s FAIL (2025-07-27 00:56:03 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001669638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........X.XXXX.XX......XXXXX...XX.X...X............X.....] duration: 12.244202052s FAIL (2025-07-27 00:55:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.244251132s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX....X......XXXX..XX...X.....X.X....X...............X.....] duration: 2.785229318s FAIL (2025-07-27 00:55:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.785259907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...X] duration: 30.113058497s FAIL (2025-07-27 00:55:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.113089051s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...........X.XXXX.XX......XXXXX...XX.X...X............X....] duration: 4.281161367s FAIL (2025-07-27 00:55:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.281216245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X.X................................................X.....] duration: 1.150980067s FAIL (2025-07-27 00:55:14 UTC) orbiter warning: digest mismatch: after fb11b40981e820a4517427aa2754a36fa2b862081373c3264bb1b1bb8591289d != before 935913abd05c74183b7f20b168f77786f9bb859af7a95d859172d883776a89e8 Duration: 1.151012128s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X...........X.XXXX.XX......XXXXX...XX.X...X............X...] duration: 3.690151712s FAIL (2025-07-27 00:55:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.690258888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XX......X..X..XX.XX.XXX..XX..X.X.X.XX....................] duration: 28.647550042s FAIL (2025-07-27 00:55:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.64757263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX....X......XXXX..XX...X.....X.X....X...............X....] duration: 2.588827529s FAIL (2025-07-27 00:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.589094555s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [...XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.........XX...] duration: 4.35370911s FAIL (2025-07-27 00:54:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.353745942s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XX......X..X..XX.XX.XXX..XX..X.X.X.XX...................] duration: 24.385139615s FAIL (2025-07-27 00:54:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.385163608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X...........X.XXXX.XX......XXXXX...XX.X...X............X..] duration: 2.417271897s FAIL (2025-07-27 00:54:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41730458s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)