Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..................X.X.....XX......X.....................] duration: 5.796672779s FAIL (2025-09-03 21:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.796708079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..................X.X.....XX......X....................] duration: 3.100971683s FAIL (2025-09-03 21:02:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.101017783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.637229053s FAIL (2025-09-03 21:02:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.637271329s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........XX...........................XX....................] duration: 3.2828812s FAIL (2025-09-03 21:02:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.282993591s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.59661013s FAIL (2025-09-03 21:01:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.596647022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.634162959s FAIL (2025-09-03 21:01:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.634198062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 2.538749148s FAIL (2025-09-03 21:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.538782354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X..................X.X.....XX......X...................] duration: 3.288587766s FAIL (2025-09-03 21:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.288625962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........XX...........................XX...................] duration: 2.945167717s FAIL (2025-09-03 21:01:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.945203497s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X..................X.X.....XX......X..................] duration: 2.662817049s FAIL (2025-09-03 21:00:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.662854527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration: 2.626354904s FAIL (2025-09-03 21:00:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.62642574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X..................X.X.....XX......X.................] duration: 2.708274454s FAIL (2025-09-03 21:00:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.708314009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X........XX...........................XX..................] duration: 2.835537748s FAIL (2025-09-03 21:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.835568828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X....X....................................................] duration: 2.537863913s FAIL (2025-09-03 20:59:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.537960397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.X........XX...........................XX.................] duration: 2.722411622s FAIL (2025-09-03 20:59:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.72243619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X..................X.X.....XX......X................] duration: 2.927354953s FAIL (2025-09-03 20:59:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.9274115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X..................X.X.....XX......X...............] duration: 2.575148875s FAIL (2025-09-03 20:58:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.575179895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX.X........XX...........................XX................] duration: 2.83527433s FAIL (2025-09-03 20:58:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.835292115s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X..................X.X.....XX......X..............] duration: 2.800035168s FAIL (2025-09-03 20:58:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.800072902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X....X...................................................] duration: 2.561456421s FAIL (2025-09-03 20:58:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.561487191s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XX.X........XX...........................XX...............] duration: 2.768954813s FAIL (2025-09-03 20:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768986594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X..................X.X.....XX......X.............] duration: 3.520193307s FAIL (2025-09-03 20:58:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.520226313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 2.416153759s FAIL (2025-09-03 20:57:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.416184287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X..................X.X.....XX......X............] duration: 2.992610616s FAIL (2025-09-03 20:57:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.992639992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 2.39251931s FAIL (2025-09-03 20:57:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.392553829s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.XX.X........XX...........................XX..............] duration: 3.077849529s FAIL (2025-09-03 20:57:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.077876199s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X..................X.X.....XX......X...........] duration: 2.966551875s FAIL (2025-09-03 20:56:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.966585058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.XX.X........XX...........................XX.............] duration: 2.586143741s FAIL (2025-09-03 20:56:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.586342013s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X..................X.X.....XX......X..........] duration: 2.924280184s FAIL (2025-09-03 20:56:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.924309205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 2.700853236s FAIL (2025-09-03 20:56:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.700882227s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XXX.XX.X........XX...........................XX............] duration: 2.777647918s FAIL (2025-09-03 20:55:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.777672457s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.XXX.XX.X........XX...........................XX...........] duration: 2.78376331s FAIL (2025-09-03 20:55:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.78381382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......X....X...............................................] duration: 2.729132292s FAIL (2025-09-03 20:55:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.729178139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X..................X.X.....XX......X.........] duration: 2.741989641s FAIL (2025-09-03 20:55:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.742039726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XXX.XX.X........XX...........................XX..........] duration: 2.944169296s FAIL (2025-09-03 20:55:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.944199788s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X......X....X..............................................] duration: 2.407249617s FAIL (2025-09-03 20:54:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.407282125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X..................X.X.....XX......X........] duration: 2.822170989s FAIL (2025-09-03 20:54:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.822287009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.XXX.XX.X........XX...........................XX.........] duration: 2.577725264s FAIL (2025-09-03 20:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.577768372s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......X....X.............................................] duration: 2.575366067s FAIL (2025-09-03 20:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.575395604s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX.XXX.XX.X........XX...........................XX........] duration: 2.771865321s FAIL (2025-09-03 20:54:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.771976329s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X..................X.X.....XX......X.......] duration: 2.950817174s FAIL (2025-09-03 20:53:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.950848706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX.XXX.XX.X........XX...........................XX.......] duration: 2.806240345s FAIL (2025-09-03 20:53:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806280188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......X....X............................................] duration: 2.55954981s FAIL (2025-09-03 20:53:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.55960277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X..................X.X.....XX......X......] duration: 4.168337263s FAIL (2025-09-03 20:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.168373713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X......X....X...........................................] duration: 2.493485245s FAIL (2025-09-03 20:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.493523412s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XXX.XXX.XX.X........XX...........................XX......] duration: 3.517099829s FAIL (2025-09-03 20:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517140094s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X..................X.X.....XX......X.....] duration: 2.950221116s FAIL (2025-09-03 20:52:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.95025211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X......X....X..........................................] duration: 2.30572419s FAIL (2025-09-03 20:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.305757277s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XXX.XXX.XX.X........XX...........................XX.....] duration: 3.350640881s FAIL (2025-09-03 20:51:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.350670244s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X......X....X.........................................] duration: 2.454374643s FAIL (2025-09-03 20:51:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.454413192s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)